A general question to anyone in the industry: how come after so much experience with over budget/over timeline projects, the industry still vastly underestimates those variables?
A couple of reasons:
1. Just like with software, you make assumptions that may be wrong because you don't have enough info up front. Whenever you remodel a house, there is always a chance that you're going to open the wall and find that the wiring isn't up to code and needs to be fixed. You always know it's a possibility and can make some guesses about how likely it is based on things like the house's age, but ultimately you're probably not going to include it in the estimate. After all, there's a ton of stuff like this that can come up, and if you included all of them in the estimate, it'd always end up being way too high.
2. Competitive pressure. You're bidding against other contractors. If you bid $1 million and the other guy bids $800k, you're going to lose. When that other guy comes with the bad news that it's going to run $200k over budget, the client grudgingly accepts because the cost would ultimately be higher if they switched contractors at that point.
I find comment 2. interesting for a couple reasons.
a) What it says about lowest unique bid auctions [1] and reverse auctions [2] with multiple bidders and multiple auctioneers. That the auctioneers will often (?) take the lowest bid, even if it represents "unrealistic" based on the criteria of the auction (this seems especially relevant in contract procurement).
Especially though, that even in a market, with relatively "transparent" information (concrete costs this much, lumber costs this much, ect...) there's still not much consumer transparency on what "reasonable" prices for construction represent. And consumers or "auctioneers" are likely to just take the lowest number, like a lowest unique bid auction. Whether its lack of information, strong preference for lowest price, can't be bothered to check reasonableness, purposeful cost obfuscation, or some other issue.
b) That people often fall for sunk cost issues. Especially on big purchases. Gov't might drop $100k contract without much notice. Except a $10B contract? Just keep throwing good money after bad. Consumer might switch soap brands on a tiny price fluctuation. Except a $1M house? Just keep throwing good money after bad.
Australia's had a rash of collapses lately, and it's felt this way.[3] 2000 companies in two years, and reading along, lots of companies that basically said "Give us another $200k as a payment right now." ("Even though we know we won't be able to finish your house, because our financials are horrible.") Except buyers felt like they had to, because they already committed to an expensive project.
In this case, it's not necessarily sunk cost. There are switching costs associated with bringing in a new contractor, so it is likely cheapest to continue with the current one, even if they didn't bid it right initially.
Switching costs are massive in construction. The permits and loans involved are often tied directly to the contractor and sometimes their subs for licensed trades. Many contractors wouldn’t want to come in and take over at a late stage because they can’t really vouch for what’s been done so it will be seen as risk.
It’s not just that there’s little incentive, it’s that just as in software getting the estimation right is an almost impossible job.
Hence if you go the other way and include severe penalties you’ll only get significantly padded estimates because the contractor now has to bear significantly more risk.
There's also a huge moral hazard in these negotiations due to the different traditional ways ( arising from the problem that all parties have perfect knowledge of the agreed upon price, but very imperfect knowledge of the putatively agreed upon product) that we view responsibilities of the person spending the money versus the person delivering the product. and it is doubly hazardous in that it disproportionately harms the honest, thorough vendor by artificially equating their product to the dishonest, corner-cutting vendor.
The buyer has an advantage in that they can negotiate the vendor down through competitive bidding techniques, and by doing so transfer the fault for insufficient delivery entirely on to the bidder instead sharing the fault themselves. If I work two contractors against one another to drive one into the position where they are now no longer safely comfortable they can deliver the product they promise for the agreed upon price, (however I am now safely in my own comfort zone knowing that I've got the lowest possible price extracted from them), well if they fail to deliver what is requested it's now their fault for promising they can do it, not mine for pressuring them into a risky position.
Surely that's only half the story. A competent customer has got to know low balling now just defers "unforeseen" costs later. If I've done 100 projects surely it's occurred to somebody to look at bid v. actual to gauge discrepancy?
Like all estimates it depends if you must do it for a fixed price or if its just an estimate.
If its just an estimate, you underbid everyone else, win the contract, start building, run out of money past the point of no return, then get more funding.
Like anything, look at the incentives. There are simply very strong financial incentives to generate ridiculously unrealistic low bids.
As for people talking about "why is the software industry like that?" I have great personal anecdote that to me explains why. Years ago I was tasked with a small team (~5 people) on a software project. We all got together and spent a good solid 3 days on planning and estimation, and our estimate was something like 5 months. Sales team freaked out, they said they needed it next quarter, so we had 3 months. So we looked at our original estimates and tried to find ways where we could cut scope, and came back with a "well, if we cut out literally anything remotely non-critical, and pray a little, I guess we can say 3 months." When all was said and done, the project took nearly exactly 5 months, just with a ton more stress and worse quality.
So these days, I'm more wise to the ways of the world. If someone says "we need you to find a way to cut the schedule by 25%", I basically say "OK, we can cut scope, but in my experience it always takes the time of the original estimate anyway." So then when things eventually go over budget, I don't stress at all - I leave work at a normal time, and just say to myself "everyone knew how long this was going to take, we just wanted to pretend it wouldn't take that long."
A big reason software is chronically over budget is that people simply want to pretend it will take a shorter time than it actually does, but they also know there is some real value to everyone engaging in that "mass pretending" in the first place.
"When all was said and done, the project took nearly exactly 5 months, just with a ton more stress and worse quality"
When in the heck did corporate America stop having men/women in the adult sense? I work in software/finance and it's disgusting the number of people who know better but don't have the backbone to stand their ground. People are too afraid of the mysterious ways of "blowback" by corporate whiners and politicians.
Because most of the gravy is in getting those things wrong. The bank or the government will find more money to keep the work going.
I worked with a GC who bragged about tripling the budget of a highschool build. If you are doing large projects the only way you're going to win is to be way off on bid day and to be good friends with the people who figure out the financing.
Yeah this. One of my friends is a PM for mostly government contracts. He also gets serious commissions for any mid project add ons so they're incentivized to low ball and win contracts then get customers to pay more later.
All the takes are correct, but they don't really explain the root of the problem imo.
We can recognize all these problems with budgets - that sales is incentivized to sell under budget, and submit change orders or project renewals later to get more money. Yet why does this happen? Can a private company really compel a government to change a contract? If the government really wanted to, they could take steps to limit the budgets, sue, nationalize a business, etc... This brings concerns that a government could wield it's power to exploit private companies, but I don't think people would be that opposed to the idea of a government trying to save taxpayer money.
The core of the issue is actually cultural. In business circles, closing a sale is considered a "skill". Lying, cheating, bribery - these are all culturally laundered as a method of sales tactics, and businesses laud salesman that can close above all else. I would argue that this is a cornerstone of business culture, at least in the US.
I think this is the core of why bids are so far off, and even a core issue with the world economics. A large portion of business occurs in the "enterprise" space behind closed doors, and the sales process is very obfuscated. It is the opposite of a free transparent market. These deals occur behind closed doors, net salesman a bundle in commission, and fall apart when they have to be implemented.
Yeah don't underestimate the physical or material limitations.
Cold, rain, or even local kids playing with not-yet-dry concrete can cause all sorts of issues.
Shortages of steel or wood, or a small accident that breaks something, can often set work back weeks. When I did welding I remember waiting around for for hours for more rebar to show up -- and you bet your ass we billed for that.
On one job some goober fell through an unfinished staircase due to it being mostly (but not totally) completed; it halted work and were down a skilled worker. IIRC dude cracked a few ribs and had shoulder problems, not sure if he got workers comp or disability -- but should have.
One a different gig someone kicked over all of the porta-pottys and turned on all of the outdoor faucets, creating a huge mess of poo and mud. Later on that job site someone stole a lot of copper, and I think at least 2 pallets of cinderblocks, as well as a non-trivial amount of fence.
Meanwhile, I've got an IT job now, and when it's cold and rainy I can sit at home and jam out lines of code as long as my wifi and HVAC mostly works.
The scary take from all the answers is that there seems to be essentially NO incentive to get the estimate right in this industry. All the incentives are toward underestimating under cover of the slimest excuse possible - just for appearances sake.
For example, you don't know what's in the old wall (but you have a pretty good idea of the amount and odds but you don't mention that) and so you don't include it (because the other guy won't and you have to stay competitive). Instead of including in proportion to the odds. You then later go to the client with "uh oh, we need to fix that. It happens. What are you gonna do?" And the client accepts it. This is mind boggling. If that's not the signs of a thoroughly broken industry, I don't know what is.
Before someone mentions that it's the same for software, not it's not always like that. With experience we got very good at estimating each in a series of comparable projects. It's very possible to fail to estimate large one-off projects. It's very possible to get it right with series. And house building is a very repetitive business. They are all different, but only so different and they are all built of human-sized rooms.
But putting aside the various incentives and psychological reasons why people tend to underestimate, I think there is a mathematical reason too.
Most complex processes with a large number of varied tasks tend to follow a log-normal distribution or something similar. And a property of this law is that when you go faster than expected, you don't go faster by much, but if you are late, you are late by a lot.
So, imagine the guy who does the estimate is completely unbiased and competent and tells you how long it usually takes (the mode), then there is more than a 50% chance that it will take longer (the median is greater than the mode), and on average, it will be even worse (the average is greater than the median).
The people who make the timeline have to make it seem like the completion is sooner than others, deals are made on that assumption.
But when building you have so many different workers, and if the foundation can't be laid because of bad weather then you can't have the plumber come in the week after. This plumber then can't reschedule to the desired date so it gets pushed back.
You make a budget but then one of suppliers drops out, so you find a new one that can do the job on such short notice but you pay the price for that.
Also You always make the budget as tight as possible so that it is very attractive, but there are clauses so you know you can go over budget in certain ways.
> the industry still vastly underestimates those variables?
Do they usually? I don't know, but curious.
My only experience with the construction industry was the construction of my house and those of a few friends where I was able to follow along.
The cost was exactly as estimated a year+ earlier and the timeline was pretty much spot on except for weather-induced delays (can't blame them for mother nature).
The people I've seen complain about cost/delay are also those who changed their minds in the middle about details, so of course it caused delays and cost.
We have exactly the same issue in software development, don't we? It's been decades and we're still bad at it. We even have a cutesy name for it: Hofstadter's law. A project always takes longer than expected, even when Hofstadter's law is taken into account.
Of course, but why are we bad at it after so many iterations? I think a comment above mentioned that there is actually incentive to get estimate to skew lower and worry about the extra costs later. What does a world look like where estimation is correct within a reasonable degree and people don't race to the bottom to get the lowest bid so that they can inevitably put out a crap product?
The blessing and curse of digital delivery of software is that retooling is comparatively easy to physical products and distribution cost is pretty negligible.
Getting it right up front matters a lot less for most software than it does something like a product where you invested a bunch of money in tool and die, stood up an assembly line, have warehouses full of product to deal with of you release a new version that obsoletes them, and so on.
If software is incomplete or wrong or bad, the real costs are mostly just labor to fix it. If physical products are bad, the very act of doing the fix has costs beyond the labor of the design work.
(I’m certainly not endorsing sloppy planning in software, fwiw, this is just how the incentives play out much of the time)
There's also the issue of requirements changing many times over, based on new requirements are added or limitations that are discovered only after making some progress in the project.
Yah you'd think with all the management know how in the US we'd be better not worse. US medicine and US education are the other two sectors where things suck too. Not coincidentally these three areas are what make it hard for a lot of people to get ahead. Costs in these areas (to purchase a house, deal with med costs with no or sucky insurance, or pay for education) is stupid high.
The construction business is one of the few where contractors are asked to provide a price to build something they really have no idea what its going to cost. They have an "estimate" but so many things are unknown.
Compare that to auto manufacturing. Ford knows exactly what an F150 costs. They can then price for known profit within a range.
Manufacturing millions of roughly the same thing is a lot different than constructing millions of roughly one offs.
Site prep has unknown unknowns, as each site is different. Maybe there's an unexpected geologic condition, maybe the utilities won't connect without a fight, maybe the jurisdiction issues a building moratorium at the last minute. Most construction jobs won't get guaranteed pricing on materials, so any delay can change those prices, and then the client may prefer to change materials and then you may need different labor, etc.
Remodelling is full of even more unknown unknowns. Residential construction doesn't tend to have as-built drawings, and the design drawings, if available, may not have much connection to the structure as built. When you open up the walls and find surprises, that's going to increase costs.
Construction quotes are generally for no delay, no suprises, no changes. Good quotes will also include estimates in case of likely surprises, and how delay costs will be allocated and change costs and procedures.
Most jobs take bids and so many people hire the lowest bidder. Because of this, a lot of contractors price their bids as a best case scenario... Or they bid an exactly what was asked and nickel and dime for everything extra.
The scope is mutable. What was asked in the quote, is different from what it became when planing started and is different from what it endup 6 months down the road.
I've had a lot of experience on one of the rungs of the commercial construction ladder. There are multiple reasons (of course), but there's an interesting correlation (in my experience) with the advent of higher technology.
CAD software, and now the various BIM tools, make it more likely rather than less likely to go into cost/timeline overruns.
Way back in the olden times, drawing plans was extremely labor intensive. Draftsmen using pencil and pen are many, many times slower than drawing on the computer. Because of this strict and unforgiving hard limit, a lot more time was spent in the planning stage. Architect and owner would work for a long time to get all of the details and requirements together before anybody touched a sheet of mylar. Any changes needed to be minor, because erasing and redrawing was expensive and difficult.
Because of the manual drafting restriction, details depended very much on using knowledgeable and experienced tradesmen. This also applied to specifications. Specifications were a lot more general, and were heavily reused between projects, because retyping 200 pages on an IBM Selectric is also time consuming and expensive.
This selects for a high level of competency across the board.
Now, because it's so easy to make massive changes at the last minute, everybody is pushing things down the road. "We'll catch it in addendums," is the rule, not the exception. And one of the biggest problems is the interface between architect and owner. The owner has ALWAYS been slow to get their people together and make decisions. So you end up with a project out to bid, and they're just now making calls on AV equipment and the like.
And the professionals, the architects and engineers, tend to put more on the drawings, and more in the specs. And because of this, a lot will get missed, or overlooked, or ignored. The general contractor sprays their sub-contractors with the drawings and specs, and the attending billion addendums, and so things get lost.
Plus, nowadays, J. Random Guy with a truck and good patter is now a licensed plumbing sub-contractor, who gets the job because he underbid. So he is now deeply incentivized to interpret the drawings and specs in a way that what should have been caught in the bidding phase is now considered a change order, which the owner has to pay for.
TL;DR: the ability to work fast in the design phase does not necessarily translate to better results in the construction phase.
(This works for software as well. We're still using software that was hand-chiseled into clay hard drives by the Great Beards Of Yore, while software that was programmed with amazing syntax-highlighting IDEs with built-in AI is riddled with obscure bugs.)
This sounds much like the same malady that has befallen movie production, and to some extent illustration and music.
The kind of artistic result achieved with true "do everything in triplicate" drafting and capturing one good performance, has fallen by the wayside when everything changes at the last moment. Movie budgets have become dominated with post-production, even when they make the effort to record in-camera scenes and effects.
And what I've noticed about illustration is the rise of a cargo cult of "the blind leading the blind" online akin to programmers who have only learned Bob Martin, driven by young people want to learn drawing digital-first, for some legitimate reasons, but also because of the gadgetry and because they want to cling to the undo button and use layers. But it's the same thing: most of the knowledge of drawing, like with every other trade, is tacit, motor-skills heavy and needs to be trained into muscle memory. Being very bad at getting the line you want and mashing undo to force your way through just means every project is frustrating.
And once you do cross that threshold and train it, having the undo and layers are nice additions, but predispose you to not do as much planning. Thus, much of digital illustration tends to have a formulaic quality to it: the artist could configure things all sorts of ways, but the overwhelming tendency is to drive towards excessive detail and excessive rendering, because the medium is an enabler for that and doesn't require you to think it through.
