Let's face it- Agile has become a codeword for let's micromanage software developers while increasing the pressure on them to deliver.
My theory is that Ken Schwaber and Jeff Sutherland came up with Scrum in order to limit management's ability to screw up software development in corporate settings. That was mixed with some observations about the need for iteration, not doing upfront work when it's wasteful (and it's not always wasteful) and visibility that came from the Agile manifesto.
However, one lesson I'm constantly re-learning is never underestimate people's ability to screw things up and I think Ken and Jeff underestimated.
The whole agile thing has been subverted and if indeed Healthcare.gov was done using "Agile" practices then this is another bit of evidence.
We, as software engineers, need to move away from these and figure out some way to qualify what makes good software engineering. You don't see non-technical hospital managers going around telling doctors where to place an incision and you don't see non-technical aerospace management telling engineers what bolt to use in a rocket engine. But for some reason "programming" has been demoted to something that just being "agile" is enough to do. Badly.
In his YouTube Scrum talk Ken Schwaber says something about if you have a poor team with bad engineering practices than Scrum will generate crap for you sprint after sprint. Somehow no one listens to that part.
Bingo. When you have project managers hounding you about "story points" and meetings of managers who are concerned about each individual's burn-down rate, it becomes a micro-managing clusterfuck.
When the estimating meetings happen without you and you're told "we think this will take you five story points" (that is, two and a half days) and then the build is broken for three weeks (because of all the people checking in code in a panic) and you're called on the carpet for being behind, then it's an atomic clusterfuck.
You're in a circle of managers and PMs.
"We're concerned about your burn-down rate."
"I'm concerned about your ability to manage a project more complex than calling the elevator to your floor."
"Are you going to be done . . . today?"
"Is the build going to fucking work . . . today, or any time in the near future?"
Then the circle jerks (none of whom has written a working line of code in five years, yet they've somehow "shipped stuff") gabble amongst themselves for a minute, look confuxed, dismiss you and call in the next developer.
> figure out some way to qualify what makes good software engineering
I doubt we'll ever be able to distill it to a formula.
People ran successful software projects decades ago, long before any methodologies were formally described. And they have failing projects now, even when they use a supposedly good formal methodology.
Instead of another alternative to waterfall, scrum, or agile, I'd propose the following ingredients for success: Organized people, a good working relationship with plenty of trust, a clear understanding of roles and responsibilities, realistic expectations, and colleagues who honor their agreements with each other. To name a few.
All of this is peopleware, not process. Unfortunately, there's no silver bullet to get this stuff right.
But what about the technical/professional side of things?
I know there's no silver bullet but those of us who have been through a few projects have a pretty good idea of what's going to work and what isn't. We can tell, well in advance, what's going to work and what isn't both in terms of the technical approach and the management process. Agree?
If that is the case how do we convert that intuition into something that can work in the real world? Is certification the answer? Specialization? How do we recognize excellence?
But e.g. how many times have you told a superior something like "This won't work because of math" and then been simply ignored? Until, of course, the system/project goes down in flames.
"In any bureaucracy, the people devoted to the benefit of the bureaucracy itself always get in control and those dedicated to the goals the bureaucracy is supposed to accomplish have less and less influence, and sometimes are eliminated entirely."
And this amplified restatement:
"...in any bureaucratic organization there will be two kinds of people: those who work to further the actual goals of the organization, and those who work for the organization itself. Examples in education would be teachers who work and sacrifice to teach children, vs. union representatives who work to protect any teacher including the most incompetent. The Iron Law states that in all cases, the second type of person will always gain control of the organization, and will always write the rules under which the organization functions."
Lawyers frequently don't get that. Doctors? Not so sure anymore, the same masters of the universe have been revising the way they have to do medicine (e.g. the Obamacare foretaste in the stimulus bill requiring "meaningful use" of Electronic Medical Records). Scratch that, Obama's utter contempt for doctors was strongly on display while he was barnstorming the nation, e.g.:
"Right now, doctors a lot of times are forced to make decisions based on the fee payment schedule that’s out there. So if they’re looking and you come in and you’ve got a bad sore throat or your child has a bad sore throat or has repeated sore throats, the doctor may look at the reimbursement system and say to himself, 'You know what? I make a lot more money if I take this kid’s tonsils out.'"
Or what about the conceit that a generic MBA can manage anything?
MIT considered this the #1 problem for it to address in the '80s.
> We can tell, well in advance, what's going to work and what isn't both in terms of the technical approach and the management process. Agree?
Yes, those of with experience can usually spot this stuff early on. But that doesn't mean we're in any position to do anything about it. Most often, software projects come pre-populated with people and institutional cultures. We don't get to pick the people and design the cultures from the ground up. At least not when we're contracting.
> If that is the case how do we convert that intuition into something that can work in the real world?
If you're building a startup that sells a product, then you hire good people and foster a healthy company culture. You're not beholden to any one outside entity, so you're not forced to import any such entity's preexisting pathologies.
If you're contracting? I think you just have to accept that projects will involve destructive people and cultures at times. Sometimes they'll cause projects to fail. In that case, the best you can hope to do is to protect yourself and your reputation.
While recognizing the completely different context, I'd approach it like organized sports management/coaching. There you have disparate members with similar skillsets but different roles that as a whole contribute to one thing: winning. Trust, respect, belief in each other's abilities, and most of all, pushing HARD to accomplish a shared goal and vision. You have a captain who plays, a coach who is trusted completely, and different roles that are recognized as indispensable yet won't work without the others playing their part.
Of course not everyone wants to be pushed this hard, wants to be part of something like this, and is paid enough to take the obvious abuse. It's just something nice to fantasize about.
Exactly. These types of social attributes are necessary for a software project to succeed. The problem, of course, is that you can't devise a system which will reliably organize an arbitrary group of people* into the kind of effective team you describe.
* As an example of an "arbitrary group of people," consider the set of contracted engineers, contracted managers, government employees, and insurance company personnel that had to work together on healthcare.gov. These are people who were not previously on a team together, and who cannot be expected to be of a uniformly high quality.
What if product managers were trained to be more like sports coaches? Motivation, the right game plan, and working hand in hand with other products managers? Doesn't an organization like NASA work somewhat the same way? Different teams tackling different parts of a project and ending up with a rover on Mars?
Agreed. These development processes and methodologies are pretty strange, actually. The confidence expressed by their proponents in the face of failure gives it a very cultish vibe, and I get the sense that they're just making things up for the sake of having "an answer".
The best answer is probably to get people who are experienced in similar projects to manage your project, and has little to do with the process they choose to use.
I agree with this and many times we are emphasizing the wrong question with our processes. Are we shipping on time or are we making good products? If we are making good products then we need to put the quality of products above timeline on occasion. Agile does take away at times longer term feature quality for the needs of short term milestones. i.e. a programmer will hack in a feature to get it done that week, then deal with problems with that implementation for weeks as there is no time to change it. Yet taking another week on that part would have saved a ton of time.
A good product a day late is still a good product. A bad product on time is a world of hurt not just for the programmers, for sales, the users, getting next projects etc. In the end it is good project management and programmers/contractors that can build a solid product, not just deliver on weekly Agile tasks. In the end the quality and product result from where people place their demands. If it is time as the main demand, over shipping a good product, then we know what happens.
It might also be that when the deadline is deemed to be more important than the quality of the product, (which, in real life happens sometimes) the tradeoffs are not made clear to all the stakeholders. Quantifying and communicating that line in a project where it goes from, "leave it out, it's not that big of a deal", to "if we don't do this properly/fix this it will be a huge cluster#&$%" is the definition of a good process/manager, I think.
I'm not so convinced it's been subverted. Scrum — even (or maybe especially) in pure theoretical true-Scotsman form — is just micromanagement collectivized.
My theory is that Ken Schwaber and Jeff Sutherland came up with Scrum in order to limit management's ability to screw up software development in corporate settings. That was mixed with some observations about the need for iteration, not doing upfront work when it's wasteful (and it's not always wasteful) and visibility that came from the Agile manifesto.
However, one lesson I'm constantly re-learning is never underestimate people's ability to screw things up and I think Ken and Jeff underestimated.
The whole agile thing has been subverted and if indeed Healthcare.gov was done using "Agile" practices then this is another bit of evidence.
We, as software engineers, need to move away from these and figure out some way to qualify what makes good software engineering. You don't see non-technical hospital managers going around telling doctors where to place an incision and you don't see non-technical aerospace management telling engineers what bolt to use in a rocket engine. But for some reason "programming" has been demoted to something that just being "agile" is enough to do. Badly.
In his YouTube Scrum talk Ken Schwaber says something about if you have a poor team with bad engineering practices than Scrum will generate crap for you sprint after sprint. Somehow no one listens to that part.