The challenge of software estimation is rooted in software engineering. As much as software wants to be called engineering, its current state of art is still craftsmanship, more of development than engineering.
Hardware engineering is possible as physics is the boundary and since has become categorically managed by human's mind of structure and logic. A jumbo jet can be decomposed to millions parts each confined by own specification therefore once design is done the production timeline is easy to estimated.
Software production to this day is still struggling between modular development and any attempt to standardize on a way to create software is overwhelmed by thousands of new frameworks or libraries even engineering methodologies every day. The reason software has such "joy" is because not like material or aerodynamic physics, software and way of making software are frequently derailed by its foundation, computer chips new advancement. Literally, every few years whoever can exploit the new capabilities of computer hardware has a chance to become next Steven Jobs.
Granted, most of corporate enterprise software projects don't have to aim high and certainly can be managed like any consumer production manufacture, so long as people are willing to settle on certain software framework, engineering approach, and treat pre-existed software component like real "physical" component meaning quit thinking the possibility of customize anything, simply focus on assembling. This is the engineering propaganda from school of IBM, Microsoft and SAP et al have been trying for decades but reality is no one but the labor cost sensitive offshore software firm would be interested to this. Everyone in corporate world pretend they're working in a creative campus and fancy about the software they make is the core competitive edge of their employer. The price to pay is uncertainty, but the question to ask is if their employer is in the business is in software business. The hard part is even this question is full of uncertainty, who'd imagine Walmart will be compete on the software it runs based on its past business model, well that might change.
The art to be able to estimate accurately (heck, even Boeing failed 787 estimate) is to turn craftsmanship to production assembling line. Stop thinking creatively, focus on manage a collection of proven engineering approaches of "manufacture" features meeting requirement. Everyone in software industry should realize there needs a distinct separation between the creative making of software world and the basic business operation needs. The line is not static though, as Google's engineers constantly made IBM's scalable/reliable enterprise software suite a joke. Still, not everyone is google and effectively manage the evolution of this distinction is not easy but necessary, because while google is looking for uncertainty, most of corporate enterprise hate uncertainty.
Once accepted the confined world, it will be possible to maintain a model of software composition by different combination of modular components, pre-existed or further decompose. The process can be fully automated and emulated as long as the time factor is added to each component. The upside is this can be estimate/managed down to minute level yet the downside is that the software created would be boring lack lust, but built with complexity to support operation reliably which is enterprise software all about anyway. Most importantly, once the estimate and cost become predictable, managing software system evolution will become a much easier and lucrative job. Imaging how joyful will it become if you go to your CFO asking for next round of upgrade (or even revamp) due to the "demand" of some fancy new capability the business users have experienced from their personal consumer digital gadget, while your commitment to the money and time has a solid track record.
The key to resolution to the challenge of corporate software engineering, including the majority of creative product builder in start ups, is the lack of an effective and efficient way to manage the software evolution and the chaos it brought. Sourcing to less cost labor or adding more bodies to the equation will not make it better, except short term psychological benefit.
In sum, software industry desperately needs automation. Ironically, software starts as the automation king to everything else, but itself has been largely remained in the hand of artist. The good news is cloud and DevOp are closing fast to bring an end to the nightmare of software engineering, at least an interim until the day software is no longer made by human.
If you can accurately model and estimate a software project, then you can automate it, at which point the project is already done and it is time for another completely different project.
Hardware engineering is possible as physics is the boundary and since has become categorically managed by human's mind of structure and logic. A jumbo jet can be decomposed to millions parts each confined by own specification therefore once design is done the production timeline is easy to estimated.
Software production to this day is still struggling between modular development and any attempt to standardize on a way to create software is overwhelmed by thousands of new frameworks or libraries even engineering methodologies every day. The reason software has such "joy" is because not like material or aerodynamic physics, software and way of making software are frequently derailed by its foundation, computer chips new advancement. Literally, every few years whoever can exploit the new capabilities of computer hardware has a chance to become next Steven Jobs.
Granted, most of corporate enterprise software projects don't have to aim high and certainly can be managed like any consumer production manufacture, so long as people are willing to settle on certain software framework, engineering approach, and treat pre-existed software component like real "physical" component meaning quit thinking the possibility of customize anything, simply focus on assembling. This is the engineering propaganda from school of IBM, Microsoft and SAP et al have been trying for decades but reality is no one but the labor cost sensitive offshore software firm would be interested to this. Everyone in corporate world pretend they're working in a creative campus and fancy about the software they make is the core competitive edge of their employer. The price to pay is uncertainty, but the question to ask is if their employer is in the business is in software business. The hard part is even this question is full of uncertainty, who'd imagine Walmart will be compete on the software it runs based on its past business model, well that might change.
The art to be able to estimate accurately (heck, even Boeing failed 787 estimate) is to turn craftsmanship to production assembling line. Stop thinking creatively, focus on manage a collection of proven engineering approaches of "manufacture" features meeting requirement. Everyone in software industry should realize there needs a distinct separation between the creative making of software world and the basic business operation needs. The line is not static though, as Google's engineers constantly made IBM's scalable/reliable enterprise software suite a joke. Still, not everyone is google and effectively manage the evolution of this distinction is not easy but necessary, because while google is looking for uncertainty, most of corporate enterprise hate uncertainty.
Once accepted the confined world, it will be possible to maintain a model of software composition by different combination of modular components, pre-existed or further decompose. The process can be fully automated and emulated as long as the time factor is added to each component. The upside is this can be estimate/managed down to minute level yet the downside is that the software created would be boring lack lust, but built with complexity to support operation reliably which is enterprise software all about anyway. Most importantly, once the estimate and cost become predictable, managing software system evolution will become a much easier and lucrative job. Imaging how joyful will it become if you go to your CFO asking for next round of upgrade (or even revamp) due to the "demand" of some fancy new capability the business users have experienced from their personal consumer digital gadget, while your commitment to the money and time has a solid track record.
The key to resolution to the challenge of corporate software engineering, including the majority of creative product builder in start ups, is the lack of an effective and efficient way to manage the software evolution and the chaos it brought. Sourcing to less cost labor or adding more bodies to the equation will not make it better, except short term psychological benefit.
In sum, software industry desperately needs automation. Ironically, software starts as the automation king to everything else, but itself has been largely remained in the hand of artist. The good news is cloud and DevOp are closing fast to bring an end to the nightmare of software engineering, at least an interim until the day software is no longer made by human.