Hacker News new | past | comments | ask | show | jobs | submit login
Killing the Crunch Mode Antipattern (2014) (chadfowler.com)
108 points by xvirk on May 16, 2015 | hide | past | favorite | 52 comments



Union, Yes!

The film industry has most of the problems of software development and game programming. But because Hollywood is heavily unionized and overtime is paid at 1.5 to 2x normal, film scheduling and budgeting is a well-developed discipline.

Film production has something known as a "completion bond". This is an insurance policy that guarantees to the investors that the film will be completed. The insurance company has the right to fire the director and anybody else and take over the production if it goes significantly over budget. The result may not be great, but you will get a movie.

"Bad Girls", a 1994 Western [1] is an example of a movie where the insurance company stepped in and took over. They put in a new director, who spent a day looking over the production and talked to everyone. She then sent the camera and lighting crew back to Hollywood, told the set builders to finish the two sets they nearly had finished and forget the others, put the stunt director on teaching the lead actresses some horse riding tricks, told the costume directors to come up with some bad-girl riding outfits, and then went off with the writers to hammer together a script that used those components. In a few weeks, they had a movie. Not a great one, but one that made back $15 million, which is a lot better than zero.

That's how you beat the crunch problem. Underestimation in the film industry applies great pain to management, not the employees.

[1] http://en.wikipedia.org/wiki/Bad_Girls_%281994_film%29


Sometimes, the entire crew is much happier and better off pulling a "crunch mode" if that's what it takes to achieve something of value. More often than not, outstanding art "runs over budget" and requires sacrifices (Citizen Kane and Brazil come to mind). "Bad Girls" may have been in a success in mitigating risks/costs, but in the grand scheme of things, it's mediocre junk nobody will care to remember.


Actual takeover by a completion bond company is very rare. If the financing party is convinced by results that it's worth putting in more money, they have that option. That's quite common when the production is going well and there are good results to show. But the director does not have the option of squeezing it out of the employees.


> The insurance company has the right to fire the director and anybody else and take over the production if it goes significantly over budget.

Sounds like market-based regulation. It would be interesting to see an example contract with the insurance clauses and list of conditions which trigger a change of direction/control. Are there parallel conditions in VC term sheets? Are there periodic reviews to determine whether the project is on track, or does the insurance company only get called in when the project approaches its budget? Any creative industry would want to avoid micro-management by insurance risk managers.


" It would be interesting to see an example contract with the insurance clauses and list of conditions which trigger a change of direction/control."

Here's one.[1] There are three parties, the producer, the financier (the party putting up the money for the film), and the guarantor (the insurance company). The contract calls for a "strike price" (the maximum film budget) and a delivery date. If the producer and director overrun those, the financier has the option to invoke the completion bond. The completion bond company can then 1) take over, or 2) abandon the picture and pay the amount of the bond to the financier. If the completion bond company takes over, they may have to put money into the production, in which case they get all gross receipts until they are paid back.

A well known director with a track record might get terms like that. Further down the food chain, the terms get tougher.[2] The bonding company watches progress and spending closely, and if the schedule starts to slip, sends observers in to monitor the production, and may take over before the production runs through all the money.

There's an up-front phase before production, during which the guarantor reviews the production budget, script, and shooting schedule for realism. This is where the real action happens. The guarantor wants to prevent cost and schedule overruns, not take over production. Completion bond companies are good at cost estimation; they see the full cost data for hundreds of films and know what everything costs. If the producer's estimate is, in the opinion of the guarantor, likely to result in an overrun, there are negotiations between the three parties. Either more money or more schedule has to go in, or some expensive or slow scenes have to come out.

This third-party check, by an independent party who's on the hook if there's an overrun, forces realistic budget and schedule estimates.

A completion bond costs 2% to 3% of the total budget.

If software development (especially game development) had that, there would be far fewer "crunches". Remember, in a unionized industry, crunches and 1.5x to 2x overtime pay increase the cost per unit of work.

[1] http://www.filmfinances.com/pdf/Completion%20Guaranty.pdf [2] http://www.eqgroup.com/completion_bond.htm


> There's an up-front phase before production, during which the guarantor reviews the production budget, script, and shooting schedule for realism. This is where the real action happens.

Thanks for the insight and sample contract. Many large software projects could benefit from formal benchmarking of estimates against a history of success/failure. Challenges would include cultural pushback, normalization of estimates across different subject domains and corporate concern about trade secrets and IP.

>The bonding company watches progress and spending closely, and if the schedule starts to slip, sends observers in to monitor the production,

These observers/PMs must be highly competent and well compensated, in order to sustain relationships with multiple creative teams while adding value to an enforcement role. Analogous roles in software development might be VC technical due diligence or PM supplied by a prime contractor on a large project with various sub-contractors.


>>> The insurance company has the right to fire the director and anybody else and take over the production if it goes significantly over budget.

Okay, but how would this work in the software field? If a product is significantly over budget, that usually means the software has a ton of unfinished features and bugs. You can't just bring an outside project manager or whatever and expect them to set things right in a few weeks. There's a lot that goes into making movies to be sure, but making software is arguably much harder.


The bond company can either finish the job or pay the bond amount. So, if a project is too buggy, it'll be abandoned.

Also, there managers who can take over a failing software project and right it. But since software is so unruly, any sensible bond company would set the budget threehold early enough to prevent those problems.


I was on a continuous death march project at my last job. Constant crisis mode got old real fast. When I was asked for a time estimate, it would generally go like this:

Lead: "How long do you think this will take you?"

Me: "Two months"

Lead: "I already promised the customer we would have it to them in 3 weeks."

Actually meeting a deadline happened so rarely I'm surprised people weren't constantly getting fired (no one got fired that I know of. Well, except our CFO for some undisclosed malfeasance). In reality, people just got burned out and quit. One especially smart guy left after only a month on the project.

After my first few failed attempts to meet an impossible deadline, I just gave up trying. I worked a straight 8-hours and left at the end of the day, waving at the missed deadlines as they sailed by. Still couldn't get fired. Damn, guess I'll have to quit. Got a new job for a small pay cut (though in a lower COL city).

I don't mind the very occasional crunch. Stuff happens. But my tolerance level for "constant crisis mode" BS is just about nil nowadays. I must be getting old.


I also worked in similar environments where sales people already promised unrealistic deadlines to customers. But you have to know the 3 weeks is not your deadline, you said 2 months, it's the lead's deadline, he is just trying to shove the hot potato on your lap.

There's a great quote for this: "Bad planning on your part does not constitute an emergency on my part".


> Actually meeting a deadline happened so rarely I'm surprised people weren't constantly getting fired

There's the problem. Developer performance was being measured according to a framework in which it was likely to fall short. This is because for many problems software development is inherently exploratory and difficult to spec out up front in sufficient detail to know in advance where the surprises are. If one is worried about being fired for not meeting deadlines, one should probably quit as soon as possible and find better work.


I've been through a few 'crunch modes' before and while there was a general feeling of "wow, we accomplished that" afterwards. Looking back, however, I now realize how profound of a hangover it produced in me and some of my teammates, and the hard truth is that it permanently takes a toll (e.g. sluggishness, lost trust, reduced motivation) that you may never get back wherever you are, at least in my experience. The fallout, it seems, truly underscores the 'nuclear option' moniker.


It is an amazing feeling to ship a product when it should have been a failure. It is unfortunate that I worked alone and didn't have a team to share the experience with. I felt I conquered something insurmountable, yet no one saw my commitment to the company, the creativity and hard work that went into it.

Management never grasped how poorly things were handled on their end. I grew to resent the company and its leaders, and had no choice but to move on.


You can't kill it. It's not even specific to software. As long as humans form hierarchies in which real productivity is dissociated from apparent productivity and "shit rolls downhill," there will be managers who believe their job is to squeeze as much out of lazy workers as possible, and that visible metrics like hours are a reasonable proxy for how much value the worker gives to the company, and that blaming others and theatrically cracking the whip is a good thing to do when goals are missed. And there will always be workers who want to pander to these managers by putting in many hours regardless of whether that is productive or healthy. If anything, startups are even more vulnerable to this problem, framed as a matter of proving that you are committed and "passionate."


Sure you can: overtime pay. Once management is punished rather than rewarded for making scheduling "mistakes," the mistakes will stop very quickly.


Employers counter with 'all-in' contracts.


The key item in the article is "loss of accountability".

IMO, that's the whole point of constant crunch mode. Everyone is a hero, which makes it more difficult to call out dumb decisions. In the places that I've worked at that abused crunch time, the management was over its head completely and was unable to get responsibility delegated to appropriate places. So the VP or even C-level execs were making decisions about LUNs (that they were unlikely to be qualified to make)

It's easier to congratulate the heroes than to actually manage.

In the worst environment, senior leadership patted themselves on the back for having as many as two dozen people on incident response calls in 15 minutes. Problems didn't get fixed, but by golly we spread the suffering around!


> It's easier to congratulate the heroes than to actually manage.

In the Capability Maturity Model 'Crunch Mode' is Level 1 - Initial (Chaotic).


The way to defeat crunch mode is data. If your velocity is 10 and you have 120 points left you probably aren't launching in 2 months. Engineers seem to shy away from metrics to describe their work because it's hard to quantify but it tends to work against them because they get trapped into unrealistic and non-data driven deadlines. Either the date has data backing it up or it's an arbitrary number.


This is exactly it. In my first software job, I found it very difficult to communicate the status of a project because we had no project management. I naively thought I was only a few brilliant coding sessions away from getting on top of things, but that slipped into a few more months of crunch mode.

Also, learning to say "no" is vital to preventing crunch mode. Wanting to please my superiors, I would typically say "yes, I'll do my best" or "maybe" when things were dire but never "no, that's not possible".


I'm not convinced data would defeat crunch mode. In your example, if a pointy-haired boss or incompetent project manager saw that the group's velocity is 10 and there's 120 points left, they would simply declare that the velocity must increase to 20 and demand everyone work extra to make it happen. Because, you see, they have already promised their boss and the client that the project will be ready in one month. Data simply empowers them to push the team into crunch mode.


Velocity only works if your project is large enough to have stable measures of velocity, but, really, it takes multiple project executions in a stable environment to get it right. That's not to say it can't be done, just that, under the conditions where it does work, you probably have smooth-running projects anyway.


The author is not necessarily wrong, however he provides absolutely no data short of personal antecedents to back up his claims about the inefficiency of "Crunch Mode." I'm sure punctuated periods of stress have been extensively studied, and I wonder if these studies align with what the author is saying. It seems to me that being able to cope and be more productive in sudden high stress situations would be evolutionarily advantageous. "Crunch Mode" is not unique to the software industry, as a trip to a college dorm during exam season will clearly point out.


It has been studied (first in 1909 by Henry Ford!). As I understand it the results agree with the original post.

In short, you can use crunch time strategically to meet deadlines but you always need a recovery period afterwards. If you work 60 hour weeks, in about 4 weeks your productivity will drop lower than it was when you were working 40 hour weeks - despite putting in 50% more hours. Also many people in this burnout zone will self report that their productivity is higher than it was (and they're wrong). If you've been crunching for a month straight, you're working ineffectively and you're too tired to tell.

Here's a presentation on the topic from Dan Cook, with links to papers: http://lostgarden.com/Rules%20of%20Productivity.pdf

And here's a fantastic write-up of a recent quantitative study in the games industry. They looked at how the success of video games correlates with crunch time and overwork. I suspect that these results would also hold true amongst startups: http://www.gamasutra.com/blogs/PaulTozour/20150120/234443/Th...


Crunch / cram in college is often a symptom of poor time management by students.

Crunch time in business is a symptom of poor time management by management.

True, the author does not provide empirical data, but anyone who has worked through a few projects culminating in a rush is familiar with the effects on person and quality.


Personal anecdote isn't evidence, but I trust it more than handwaving appeals to evo psych. You do realize too that humans at college age don't have fully developed frontal lobes, otherwise known as the seat of planning and discipline?


Or perhaps I'm just attempting to justify my bad habits.


I once had an interesting conversation with a director of engineering - he asked for an estimate on how long a project would take given the resources. I gave an optimistic estimate of 3 months, and expressed that it was optimistic - he was dumbfounded, and told me that the company didn't have that sort of time.

I ended up being quite prescient - the project took 3 1/2 months, including a bad crunch near the end of it. Fortunately I was placed in a different team before that crunch, I probably would have quit fast after surviving an insane crunch a few months beforehand where I wrote ~50% of the frontend code due to a series of unfortunate circumstances.

Crunch mode makes developers want to quit and distrust managers/executives - it's a breach of trust.


Other reasons to go into crunch mode, not mention by Chad Fowler:

1) You think the work your team doing is already of such low quality that it can't get worse, and increased "quantity" of low quality work is better than the usual.

2) You've created a crisis and feel you can convince your team that this is just the natural state of capitalism, or that by accepting salary they are implicitly accepting the debt and promises that you made to your VC firm.

3) Feature lists and time to market trump actual performance. You're trying to get something that can be demoed and sold without regard to how well it actually works. Maybe your product is for a niche industry and you can make a convincing business case on numbers alone, or maybe the people who do the buying are not the people who will use the software.

4) Your product is on shifty legal footing. Having a saleable product will bring you legal support, but until then you are a sitting duck for a lawsuit or regulatory action.

5) Your team is capable of better work, but you believe that the only way to extract it is to create a crisis.


Great article. There are even more negatives than it points out. After such sessions, it's likely that employees will underperform and try to get away with it as much as possible. When employees see that those weeks of working around the clock served absolutely no purpose, it's likely they will develop a resentment and try to take that out on the company in whatever ways they can. Taking extra time off or putting in low quality or bare minimum work is not unheard of from disgruntled employees who have just been forced to work for free.


I am firmly in the other camp when it comes to the common theme on here that anything above a 40 hour workweek is a sign of poor management of time and people. It's one thing to be forced to do those work hours against one's will, and I'll grant that doing so is bad business practice. However, when you are working in a startup environment, especially if you're burning through other people's money, there should be a fire lit under your ass.

A few weeks ago, my startup[1] did a crunch mode week before releasing our beta to the first set of users. We had promised access to the application on a certain day, and I am absolutely never going to be convinced that missing a deadline that you've promised someone else is an okay thing to do, as the article suggests as a form of self-punishment. In doing so, I did a few things in order to take care of my one employee. First, I said that lunch and dinner would be covered by the company, since we were both staying there until the nighttime hours. Second, I made a Costco run and got us anything would could need to snack on or drink during the week (not very expensive when you're buying for two people). Third, immediately following our successful launch on a Monday, he had Tuesday and Friday off. It's very important that when you're not on crunch that it feels like you're not on crunch, and you "make up" for the time that you burned through.

I am always of the opinion that if you're not willing to put in a ton of hours, there's someone just as smart and focused as you that is. I think my perspective has been forever changed, though, because I've been on deployments where we worked 12 hours a day for 9 months straight with no holidays, doing knowledge work I might add. Becoming accustomed to that makes everything that comes after it seem incredibly easy.

Both my employee and I feel incredibly lucky to be working on something that we both really believe has a chance to succeed greatly, and it is so much a part of our lives that when we're not at work we're thinking of more ways we can improve our product. When you're that deep into an idea, be it a business, hobby, love interest, etc., the amount of time you spend on it always seems like not enough. I think people bristling about long work hours are doing so because they haven't found something that makes them feel this way.

[1] http://pleenq.com


> I think people bristling about long work hours are doing so because they haven't found something that makes them feel this way.

Or they are not in a position where they stand to profit in proportion to the extra work. Extra time with out extra compensation means the employee's individual ROI drops, and that is not something a smart person should accept on an ongoing basis.


"I think people bristling about long work hours are doing so because they haven't found something that makes them feel this way."

Or maybe they just don't like being forced to work for free. What you're saying is that employees should be glad that they get a chance to work overtime for free. They should be glad to be victims in other words. That's just insane.


Well, he -did- say "I am always of the opinion that if you're not willing to put in a ton of hours, there's someone just as smart and focused at you that is."

Mind you, another wise man once said "there's a sucker born every minute", and psychologists have identified Stockholm Syndrome as being a thing, so he could be right.


It's likely there is someone just as smart and focused as me that is willing to put in the long hours. The problem is finding that person, and once you find that person and you work them to the bone for a couple of weeks, they will not be as smart or as focused no matter where they started out at. They will continue to deteriorate till their output is of lower quality than average, even if they were brilliant at first. This is the point most manager, founders, owners, and executives that support working their employees overtime intentionally miss. There is no way a human being can keep up quality work being sleep deprived and working around the clock. Even a little sleep deprivation and a bit of extra overtime will take its toll after a few weeks.

That will happen even if you pay employees overtime, which most companies and proponents of overtime are vehemently against because the real reason they demand it is because they want to exploit free labor along the lines of temporary slavery. No wonder there is so much shitty code out there, especially at startups and other companies that love to overwork their employees without paying them. Then they call it "technical debt," as if there was anything technical about depriving employees of sleep, health, and their life outside the company. And finally, to top it all off, they want us, the employees, to enjoy being worked overtime and having our health ruined and our time stolen. The delusions of management and executives in this area is incredible.

But I don't think it's fair to call employees willing to subject themselves to this suckers, only the ones who do so knowingly and willingly. Many people simply don't have other choices or have other obligations (family, mortgage) that make changing jobs difficult. Others, especially young people, just don't realize what they're doing to themselves until they burn out or are too caught up in the social "coolness" of it.


> However, when you are working in a startup environment, especially if you're burning through other people's money, there should be a fire lit under your ass.

"Fire lit under your ass" doesn't mean "work 16 hours a day." If you want to succeed, you should be focused on producing and maximizing the right outputs, not measuring time worked, which is simply an input.

> I am always of the opinion that if you're not willing to put in a ton of hours, there's someone just as smart and focused at you that is.

That might be true, but can you recruit and retain those smart and focused people? In today's market, the talented and experienced have options. Unless you're offering top-notch salary and benefits, chances are you're not going to be able to retain and recruit the best people while demanding they live in a constant crunch mode. They don't have to subject themselves to this to earn a good living.

> Both my employee and I feel incredibly lucky to be working on something that we both really believe has a chance to succeed greatly, and it is so much a part of our lives that when we're not at work we're thinking of more ways we can improve our product. When you're that deep into an idea, be it a business, hobby, love interest, etc., the amount of time you spend on it always seems like not enough. I think people bristling about long work hours are doing so because they haven't found something that makes them feel this way.

Is it not possible that people bristling about long work hours are doing so because they have other interests in life besides work?

While your earliest employees should drink a little bit of kool aid, as you scale a company you can't reasonably expect your employees to be as invested as you are. And you should be mindful of the fact that the people who provide the level of dedication you seem to be looking for are often the ones who burn out and become the most disillusioned and resentful when you hit the inevitable bumps in the road.


> > I am always of the opinion that if you're not willing to put in a ton of hours, there's someone just as smart and focused at you that is.

> That might be true, but can you recruit and retain those smart and focused people?

There's also the problem that after a month or two of 60-hour weeks, that person will be less productive than someone who works 40-50 every week and saves emergency mode for the rare genuine emergency. So even if you can attract and retain that person, you'll be overpaying for shoddy work. It's a wonderful economic outcome, in which both parties are suckers.


To point out, "A few weeks ago, my startup did a crunch mode week before releasing our beta to the first set of users."

You did -one week- of crunch. That's not what the article is speaking to. You're talking that "if you're not willing to put in a ton of hours", "my employee and I feel incredibly lucky", etc etc etc, but you only pulled a -week- of crunch. This isn't talking about where your down time outside of work sometimes incorporates thoughts of "what could we do better at work?", nor is it talking about "We're so close; if we just work a little longer this week we'll hit this deadline, learn from it, and can take a break right after", it's talking about the repeated pattern in some places of crunch mode all the time, > 40 hours a week for multiple weeks, which research has shown will lead to negative performance.


It's not me who wants lunch, dinner and snacks. It's my kids! Want me to pull an 80h week? I'd be delighted, just pick my kids up at daycare at 4.30, make sure they wash their hands after dinner and be ready for a bit of mayhem when it's time for bed.

Joking aside, I have only sold 40h of my week, not because I wouldn't be happy to sell you more hours but because the rest are already sold.


This attitude is incredibly short sighted.

No one is suggesting that you just miss deadlines. The suggestion is that you get better at setting deadlines.

You may be living and breathing your exciting project but, really, how many lives were saved by setting such an aggressive deadline? How many businesses did you save from bankruptcy? If your own, then you might want to think harder about saner financial planning as well.

Very few projects need to rush to the finish line to be successful.

You live and breathe an exciting project. I get it. But how is the quality of your thought process affected by pushing yourself to the limit everyday? People need space to think. People need space to see outside of their obsession. Not allowing for that misses opportunities or, worse, puts you in a position to be surprised by something outside of that obsession.

The rest of us aren't stupid. Many of us are seasoned. We realize that one must be wary of obsession because keeping your mind in one mode for too long makes it very hard to move from that mode or even eventually realize that other modes exist. And, make no mistake, eventually reality will force a change. The question is, will you be cultivating the flexibility to deal well with it or will you be stuck in one mode?

You're running a marathon, not a sprint. Let your demons drive you but make sure you can take the reins back when you need to.


your first mistake was to promise arbitrary date for delivery without any data to support that; that is what a common folk call wishful thinking. It never works. Being passionate about something doesn't mean you have to kill yourself working. It sounds to me that you are fairly young and IMNSHO if you are smart you should know not to put ton of hours and have things done. Quantity of hours put into a product doesn't correlate with a quality


Out of pure curiousity,

1) how many extra hours did your employee work (i.e. over an 8 hour work day)?

2) if extra hours > 16, how do you feel about that? (FYI as an employee I'm pretty eye-rolly if I get a day off as a "favor" after burning >=8 hours in "over time" -- the = is important!)


here's elon musk on the matter:

http://www.simplethingcalledlife.com/2015/elon-musk-usc-succ...

> And if you do the simple math, say that someone else is working 50 hours and you’re working 100, you’ll get twice as much done in the course of a year as the other company.” – Elon Musk

Elon is a typical SV guy right?


Elon is probably quite wrong if he actually expects a consistent linear relationship between hours in and output.

It does seem as if he's very smart and able to lead significant enterprises to success, however.

There might be two takeaways here:

1) Even smart/successful people may believe things that are wrong.

2) The point of Musk's statement may be something other than to communicate an accurate model of output vs labor.


He also expects employees not to be there for the birth of their children (http://www.autoblog.com/2015/05/11/elon-musk-scolded-employe...) with the money quote being "you need to find out where your priorities lie.


Ah, good find. Bookmarking this for next time I need an example of a smart person believing something incredibly stupid.


Anybody on a team that always wants its employees to just go a little bit faster?

Meaning, taking comfortable estimates (on the order of 1-2 weeks) and just, by default, sliding them earlier by one or two days?

It's a subtle way for management to cope with the anxiety that is trickling downhill, without engaging in or admitting a period of Crunch Time.


Not that exactly, but I've been repeatedly exposed to this:

Management: "How long will it take?" Me: "N weeks" Management: "Ok, you have N/2 weeks"

I've been told they actually teach this at business school.


My favorite manager, as far as I can tell, communicates N*2 to the client, then we either have time to fix unexpected bugs, or maybe throw in one new feature before delivering exactly when the client expected.


Not a problem, you say "in N/2 weeks I'll deliver feature Y, which is similar to X but half as complicated".

If that gets you N/4 weeks to do Y, or still N/2 weeks to do X, get new management.


Pull a Scotty and estimate N*2 or counter with a simpler feature that you can do in N/2.

Business school is just expensive signalling.


Management: How long will it take? Me: 2N weeks.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: