That's the truth, except, I can't tell clients that "I'll know how long it will take when I'm halfway done." The best I can do is look at prior work and multiply by the LarryDarrell Constant of 1.5.
I think one of the top skills for management / senior dev is to be able to translate between software estimate uncertainty and the business reality surrounding the schedule.
For example, if you ask, "How long will this project take?"
You could just say, "I don't know. At least two weeks, but it might take as long as six months." This may be accurate but it's not helpful.
Or you could say something like, "In a half day I could put together a preliminary design, and identify the most uncertain parts of the project. From there we could figure out what is needed to tighten the schedule or adjust the requirements."
And I think that one of the more common problems with managers in larger orgs is that they can't tell the difference between somewhat arbitrary deadlines (we want this for Q3 to look good in our org, but we can break this project into smaller milestones and push it back if necessary) and deadlines that reflect the reality of business (we want this for Q3 because this will have cascading effects on other parts of our business, and if there are problems we need additional resources or an adjusted scope).
Any time there is a deadline, the manager should have a clear understanding of the consequences of missing that deadline, and be able to communicate that to the engineers. The efforts to come up with a schedule and milestones should be collaborative. Too often there are mandates that have unclear consequences for missing them, and when schedules slip you have no idea what you are supposed to prioritize.
Funny thing. If you do this organizationally, then you eventually end up making larger and larger estimates. IMO it is better for management to understand the inherent risks associated with engineering estimation and build the padding on their end.
I usually do 2x my expectation. I keep track of my prior estimates and compare against my real time. I normally miss by about 30% - 50%, this leaves me a fair margin. It’s also been improving with time, which means I’m getting better at assessing my skills and problems.
For me, I tend to work quicker than other contractors / employees, so I still often beat out the other quotes (time or money). That’s not always the case for others, so they margin of safety may not be there. In which case, multiply by a smaller constant.
I kind of lean towards double the schedule and cut the scope in half and hope that maybe we can deliver a little extra credit work.
It's important to note that padding schedules is not just done in software, it's a common practices in all sorts of engineering. It's not bad that we do this, it's bad that management often doesn't understand that we do this and doesn't understand why. The understanding why part is actually really important, because that means management can be expected to do their part in managing the uncertainty or at the very least they can be expected to be careful about not making the situation worse.
