Agile is just another in a long line of fads that try to use management "methodologies" to make programming a commodity that can be done by line-workers. It's not, and it can't be made so. Programming is a highly skilled craft.
Good code is written by good programmers. Bad code is written by bad programmers. Bad management can hobble good programmers, but no form of management can make bad programmers write good code.
(I'm not saying agile is bad, necessarily, just that it does not live up to its hype.)
Agile is just another in a long line of fads that try to use management "methodologies" to make programming a commodity that can be done by line-workers
It's sad that so many people have that perspective. I've always looked at it in the opposite way. I mean, the first part of the agile manifesto is "People over Process" and the first wave of agile evangelists were actual programmers and not fad-prone management types.
Yes, but actual programmers with an altruistic desire to improve the state of software development are never going to be as loud and noisy as various salesmen (agile consultancies, agile coaches, agile tool developers) and hence the latter group of people are going to take over the word "agile".
Just because some snake-oil salesmen are selling it, doesn't mean it's snake-oil, it just means that opportunists smell an opportunity. The altruists who are trying to improve the state of the art are still out there, if you know where to look.
I feel you got it backwards. The whole idea behind agile is to treat programming as a craft and to focus on the people, as opposed to treating development as a production line.
Some good readings on the subject: Extreme Programming Explained, Agile Software Development Ecosystems, Balancing Agility and Discipline.
My problem with this is the 4th tenet: "Responding to change over following a plan".
Some tasks require lots of planning. When you are building a complex system, you cannot plan as you go, and in some cases your first usable release cannot happen after the first 'sprint'. In this way, agile programming assumes that the programmer cannot make long term development plans, and in fact prevents him from doing so. While this eliminates some of the risk, it also prevents programmers from reasoning about their problem holistically, and can deliver inferior or non functioning solutions for projects that do not happen to fit the model well.
This refusal to allow programmers to reason holistically is part of what us critics mean when we say that agile attempts to turn programmers into line workers.
Complex projects, like building a jetliner (or the programming equivalent), cannot be "continuously integrated" with no up front planning. For projects like this, you need to decide the thrust of your engines before you start to design the airframe, or your entire project may fall apart, literally.
Agree. I think of Agile as sort of "the hill climbing of software development" (hill climbing as in the optimization algorithm). If you want to take the analogy to its logical conclusion it's really simulated annealing because in a refactoring you're allowed (with some probability) to back down from the hill in search of another slope. :)
Agree about the line worker thing too, and that I think is a fundamental problem with most methodologies. Great authors write great books. Great programmers write great software. Any methodology that pretends there is a way around that I think is bound to fail.
I don't think agile methods necessarily rule out planning. It's possible to combine Waterfall and Agile within the same project, so long as you are clear about what you are trying to achieve.
When I embark on a new project, it's routine for me to spend a couple of weeks with the customer sounding out their requirements and helping them form an initial vision of what they want the software to do. (Big companies call this "business analysis".)
Once I'm reasonably certain what sort of thing the customer wants, I start work on the first "sprint". This first sprint is often quite a bit longer than subsequent sprints, and incorporates the holistic design phase you mention. It's not uncommon for me to spend a month designing and working on this first iteration.
The goal of this first iteration is to capture the essence of the vision without implementing the details. Only once the customer has viewed the results do I reduce the time for each sprint to a week. (I believe two weeks is more common, but this period suits me and seems to fit with what my customers want too.)
A dozen or so iterations/sprints later, and I tend to be done. At least for a while, anyway... inevitably a customer will decide they want another big feature and I start over with the business analysis and the long first iteration followed by shorter incremental steps.
It really depends on the kind of project you're working on. Some modern Agile authorities (see Balancing Agility and Discipline: A Guide for the Perplexed) will tell you that you should tailor the process to your specific requirements. They would never recommend that you use XP for the development of life-critical software that has hundreds of engineers working on it.
Agile is born from rapid prototyping and makes a lot of sense for that environment. If you're doing work where one day you might be developing one thing, and the next day you might be working on a completely different product altogether, then reducing the cost of change is the only thing that makes sense. If you're not, then don't use pure agile, it doesn't make that much sense.
There's obviously an issue with one size fits all approaches to agile, and as agile matures people realize that more and more, instead of forcing you to do everything by the book. The point is, as you get more and more experience with driving software projects, you need to be smart enough to understand what will make sense and what won't.
I agree with your general direction, but I have a quibble.
Agile methods were born from in-house enterprise software projects. The ur-project for Extreme Programming was a payroll system, and SCRUM's was something similarly dull. The original SCRUM paper says, "Scrum is concerned with the management, enhancement and maintenance of an existing product[...]. Scrum is not concerned with new or re-engineered systems development efforts."
If I'm doing rapid prototyping, that's when I'm least interested in Agile techniques like pair programming, test-driven development, and refactoring. Those only really pay off in the long haul.
No serious agilist suggests that you never plan, or that all projects can deliver a useful release in a week. Indeed, Agile teams see planning as something so important that you do it frequently. What they're opposed to is blindly following an old plan in the face of new circumstances.
If you are lucky enough to work in a domain where you can reliably plan years or decades ahead, then you can still happily use Agile methods. Incremental delivery still lets you get more for your money by putting the most valuable parts of the system in production first.
But if, like most people, you have competitors, or if you learn new things when you put your product in the hands of users, then the ability to respond to change is absolutely vital. In that case, holistic reasoning is an ongoing process, not something that happened once, years ago.
Agreed. I think agile, while incorporating some good ideas (testing makes better code! talking with the customers helps you figure out what they want!), ends up in practice looking like a shyster scheme and burning a lot of businesses and programmers.
Assuming facts not in evidence error at line 1: Presumption that Agile ever ascended.
The basic problem with the Agile movement is that it took a few proven helpful practices -- almost all of which were well known and often used by experienced software developers before the buzzwords were attached, even if managers didn't realise -- and instead of promoting those, it wrote a "manifesto" so vague that it would make a political speechwriter proud, which didn't actually say anything at all about the practical steps that were useful.
Then concrete examples of "Agile" processes started to appear, and a few less proven or even discredited practices were added to the basically good ideas. Suddenly, despite valuing people over processes or some such vacuous claim, it's all about The Process again.
The predictable result was something non-technical managers thought sounded good but didn't really understand, and technical people who didn't drink the Kool-Aid looked at sceptically while asking where the evidence was that it was better than what they had been doing already.
And the only answer the Agile community has collectively agreed on in the ten years since then seems to be that people aren't doing Agile properly in most cases, but look at the successes where smart people have made it work! Of course, those smart people probably could have run a successful project starting from any other process, too, because if it didn't help they would have thrown it out in favour of things that did, just as they always had. And if most of the case studies of your supposedly superior process don't show significant benefits, you start to lose credibility after a while by claiming that they're all doing it wrong.
And now, we're doing the same thing all over again with the not-at-all-pretentiously-titled Software Craftsmanship movement, and the result will be the same: lots of people selling books and charging consultancy fees to peddle snake oil mixed with just enough actually proven ideas to be credible for a while until the results come in, by which time they will have moved on to the new hotness.
I am not normally a fan of Zed Shaw's style, but in this case, I sympathise with his point of view:
That is approximately the opposite of how it happened.
Several different groups of people invented processes that worked for them (e.g., Scrum (1995), XP (1996), FDD (1997), DSDM (1994)). They all liked what they had done, but discovered they had commonalities that were worth talking about. They got together for a long weekend and came up with the Agile Manifesto (2001).
Having lived through the transition, I promise not all the good practices were "well known and often used". Frequent releases were generally considered dangerous, and weekly iterations were seen as heretical. Unit testing was done by a small number of projects, but mainly in safety-critical areas. Test-driven development was unheard of. Continuous integration was certainly rare, as was using refactoring (supported by high test coverage) as a way to manage change.
Personally, I think the Agile movement is irredeemably doomed at this point. It's basically a question of incentives:
However, the current sad mess happened in spite of the good intentions and major discoveries of the people behind the methods that got rolled up into Agile(TM).
You're not the only one who lived through the transition. And IME the good practices that are used in Agile processes mostly are the ones that were well-known and widely-used.
People have written automated tests for decades. People have been testing at the interface level since the dawn of OO and before. Put those together and you've got what is now called unit tests.
People have been making small changes to improve the design of code without changing its behaviour forever, too. Fowler published Refactoring in 1999, but even then it was common practice among experienced programmers to work that way. I assume you're not seriously going to suggest that no-one thought of doing things like moving a method up and down a class hierarchy or splitting a long function into a higher level control function using lower level functions for the details before the book was out. (I'm not disputing that Fowler did a good job of pulling a lot of common knowledge together in a way that is helpful to those just starting out.)
I also assume you're not seriously going to dispute that long-term projects have been managed using shorter-term milestones for a very long time. That's hardly unique to software development, never mind to Agile processes.
Every project I've ever worked on professionally has seen developers naturally pair up or form small teams around the same computer or whiteboard to work through difficult problems or review each other's solutions.
People have been doing what is called continuous integration today for as long as there have been source control systems with check-in hooks.
I first had the idea of organising a big project by keeping a pool of little projects and picking out the most important one to do next when I was about 13 or 14, so I rather suspect that plenty of managers were using the same idea to prioritise long before anyone called it a backlog and worked through it in sprints.
I could go on, but I think you get my point.
Now, there are plenty of other practices required by various Agile processes that weren't (in my experience) almost universal among good programmers before Agile came along. They include:
- TDD
- Full-time pair programming (same computer, two keyboards)
- Forcing short-term milestones into periods of exactly the same length
- Frequent meetings (daily in some processes)
- Relying on organic designs that evolve through arbitrary refactoring by individual developers rather than a carefully planned architecture and smaller-scale designs that are peer reviewed by multiple people
If you want to claim those as the gifts Agile has given us, you're welcome to them. As I said before, most of the good practices were already in use, and the consultants mixed in a bunch of unproven ones later.
You're again wrong on the history of the Agile movement, and I'm disappointed that you entirely ignore the meat of the point that you're replying to.
As to your assertion that everything good was in wide use before: that's not how I recall it. For example, CI: the Joel Test, written in 2000, suggests that the vast bulk of shops couldn't even manage to do a daily build, and he didn't even mention CI as an option. Also, it's not clear to me what a 13-year-old's imaginings proves about what a majority of development shops were doing in the mid 90s. My recollection is that waterfall was the majority, with most of the rest of the shops doing code-n-fix. And the first edition of McConnell's Rapid Development, which was a mid-90s book, either doesn't cover or treats as exotic most of the Agile notions.
But honestly, I don't know that it matters. I'm not sure why, but you seem eager to prove that there was nothing good about it based on your incorrect version of the history. The good news from your perspective is that what's sold as Agile has turned almost entirely into dismissable buzzwords, and as a fad I expect it will be extinct soon. There's nothing left to fight off.
But personally, I plan to keep using and evolving what I learned from it. If you want to see what a shop like that looks like (including all those practices you mention above), feel free to drop by in SF sometime. We like visitors.
There are millions of programmers in the world, and plenty of us have been doing it professionally since before Agile was all the rage. It sounds like you have too, but are you really arguing that just because you personally didn't encounter these ideas before Agile advocacy came along, no-one else did either? Or perpetuating the false dichotomy between Agile and Waterfall? I have never worked on a project that followed a strict Waterfall model without any kind of cycle or iteration in the process, not once in my whole career. Now get off my lawn!
Seriously, though, you've cited Spolsky and McConnell, but you seem to be appealing to their authority on the basis that if the techniques we're discussing had been in common usage, they would surely have been mentioned in those authors' writing in the late '90s. I would point out that the first edition of McConnell's Code Complete was published in 1993 and didn't say much about OOP, yet OOP was already in use by many professional developers, and books about languages supporting it had been published many years earlier: the first edition of The C++ Programming Language was published in 1985, for example. Similarly, the second edition of Code Complete was published in 2004, yet contains no index entry for the word "concurrent" or "parallel" (though there is a passing reference to multithreading on page 337). So perhaps it is unwise to infer too much from any absence of references in the kinds of source you mentioned.
I have never argued that "no-one" encountered those ideas, and your repeated exaggeration of my position is frustrating.
Nobody in the Agile movement claimed novelty for individual pieces. What was different was the way the pieces went together, and the shift in values and cultural norms that they required. That wasn't purely novel either; the fact that several people invented similar processes at the same time suggests that, as with most inventions, change was in the air. But what the Agile people undeniably did was to define and promote that difference in a way that gave people something to rally around. At the time, waterfall and a waterfall-ish RUP were the big processes people were discussing, and the notion that one would focus on people over process, or working code over documentation was heretical.
Also, I'm having a hard time taking your approach to evidence seriously. Rapid Development was a major survey of current software project management and a catalog of best practices put out by a large tech publisher. Splosky was also writing clearly about current practices. Your counter-evidence is the (unnamed) projects you've (anonymously) been on. Plus that one of them wrote a different book that you can pick nits on.
And honestly, I don't get why you're arguing the nits and ignoring my main points. You clearly have some big issue with your version of Agile and it's imagined history. But I honestly don't care what process you follow as long as it doesn't suck for you, so arguing it with me isn't really helping anybody.
I'm sorry you're feeling frustrated with this discussion. I'm afraid I just don't understand what your "main points" are supposed to be, if you think I'm only picking nits with the details.
I started this subthread with some fairly black-and-white statements about the relative timing of good practices being adopted, Agile taking off, and less proven practices entering the picture:
Since then, you have accused me, repeatedly, of being wrong about the history of Agile. Is your objection that early versions of a few Agile processes already existed, albeit in relative obscurity, before the Manifesto was written, but I talked about how 'concrete examples of "Agile" processes started to appear' after the Manifesto was written?
If so, I'll happily concede that point. I don't think it really matters whether you define Agile as starting from the evolution of the general approach/abstract philosophy (which had evidently started by the time of those initial processes) or the writing of the concrete Manifesto and popularisation of the Agile moniker. Nor do I think it really matters whether you define processes like XP and Scrum as appearing as "Agile" processes when they were popularised and given that label (around the time the Manifesto also became widely known) or from when the original processes that became what we now call XP and Scrum were first used anywhere. Any way you cut it, we're talking about the same groups of people, the same general period of time, and the same end results.
My main point is simply that many of the good practices that were adopted by those early Agile processes and are often considered to fall under the Agile umbrella today did not actually originate during that period of time or with the people who worked on those early processes and the Manifesto. However, many of the Agile practices I consider unproven did originate during that period of time and with those people.
While you may wish that no-one in the Agile community claims otherwise, unfortunately that is not always true. While I was researching my post with sources, I found a comment on Martin Fowler's personal web site claiming that Kent Beck coined the term "refactoring" in the mid-'90s, for example, and numerous comments from various posters on other forums suggesting that Fowler's book was the definitive original reference. I think it is important to debunk such claims and any attempt to paint Agile and Waterfall as some sort of dichotomy. That just isn't how software development works, and being "Not Waterfall" is not an argument for switching to any given Agile process.
> I found a comment on Martin Fowler's personal web site claiming that Kent Beck coined the term "refactoring" in the mid-'90s, for example
No, I didn't. Sorry. I had read something similar in several other places, and when I did a search (since Fowler's site itself was blacked out) the resulting excerpt was truncated in a critical place that made it look like the above was true. Still, there are plenty of people who have claimed that the Agile movement originated ideas like refactoring in on-line forums and the like, so aside from setting the record straight regarding Fowler's site, my original point stands.
I didn't have anything specific in mind; I'm going by my own experience, discussions with past colleagues, and conferences over the years more than anything. That said, it's not hard to turn up examples with a few minutes looking through my bookshelf and Google Scholar.
For automated testing, just stick search terms like "automated software testing" into Scholar and set the latest year to, say, 2000. You'll turn up dozens of papers going back decades that talk about techniques similar to what we would call unit testing today.
Continuous integration is another obvious one. The likes of Grady Booch were writing about the advantages of frequent integration builds by the early '90s. The term "continuous integration" appears in Object-Oriented Analysis and Design with Applications (1993), for example.
The prior existence of refactoring techniques is kind of obvious, as people had been discussing simple transformations like moving functions and data up and down a class hierarchy since the dawn of OOP. If you want a specific citation, Design Patterns (1995) mentions the specific term "refactoring" a few times, in turn citing academic papers on the subject cowritten by one of the authors as early as 1990, around a decade before Fowler's book and several years before Beck coined the term according to some Agile advocacy.
That's all I've got time for now, but I'm sure you could find eqally authoritative sources for the other things I talked about if you genuinely want to know.
I'm not claiming TDD as something widespread before Agile. I'm claiming TDD as something unproven that the Agile consultants started mixing in later. (Please see the list in my second post to this thread.)
For the third time: you are incorrect about the history. Test-first development was created by working programmers as part of Extreme Programming before the term "Agile" even existed.
I'm not necessarily disputing that, though I note that you have changed from TDD to merely test-first development, which I don't think is the same thing.
My point is that TDD, like several of the other practices in XP that do seem to have originated with those people during that time period, is to this day not proven as any better practice than the alternatives.
Organizing groups of people to put together something decent in a highly limited amount of time and with humane treatment is extremely hard. I don't see Programming, Motherfucker addressing that in any way.
I don't see much evidence of Agile methodologies addressing that in any way, either. That's rather the point.
After this long, if Agile is as superior to the alternatives as its advocates frequently claim, there should be heaps of empirical data miles high to justify each and every practice these consultants charge so much to explain by now. So where are they?
The best I've seen any big name Agile consultant cite is a handful of small-scale experiments, mostly using CS students rather than experienced industrial practitioners, invariably not controlling for using unit tests alone without the TDD process, that show that projects using TDD have modest but inconsistent and frankly unspectacular gains over projects that don't bother unit testing at all.
I'm a scientist. I believe in the data in front of me, and my conclusions on these matters are falsifiable. I will be happy to revise my opinions if and when new evidence shows any of them to be incorrect, but that evidence had better be more than a few alleged experts telling me what they think I should believe if I'm going to change the development practices of entire teams based on it.
It would be awesome if somebody would fund those experiments. But since Agile methods are mainly about sustaining high quality over the long term for teams of experts, studies with a useful n would be stunningly expensive.
And honestly, there's no great reason to believe Agile consultants. Most of the market for software process improvement is in helping the clueless and/or terminally lazy. (And there's often even better money in not helping them, but instead just making them feel special.) Plus, a lot of the benefit from particular practices depends on the existing code, the business conditions, and the people involved.
I think the right way to deal with Agile techniques is the same way one deals with new tools or new languages: try 'em for a while and see. E.g., I tried out TDD for a while and ended up liking it a lot. As does the rest of my team, such that they all were specifically looking for test-driven shops when I hired them. But even we tend to only use it on core code; test-driving most UI just doesn't pay off.
> But since Agile methods are mainly about sustaining high quality over the long term for teams of experts
> Most of the market for software process improvement is in helping the clueless and/or terminally lazy.
So are we trying to help teams of experts or the clueless and/or terminally lazy here? I don't know many clueless, terminally lazy experts, so it's got to be one or the other...
Also, experts will evolve a workable process for any given project without needing to follow any particular Agile process to start with. That's one of the reasons they're experts.
Also also, while it's convenient to pretend that no serious studies exist because of the cost, it should still be pretty clear that Agile processes have potential, because there should be a strong correlation between the time many organisations introduce them or which teams within a large organisation have adopted them and an improvement in measurable performance for the Agile groups. Correlation doesn't prove a causal link, of course, but lack of correlation is a strong indicator that no such link exists.
> So are we trying to help teams of experts or the clueless and/or terminally lazy here?
Me? I'm trying to help teams of smart, well-intentioned people. Which is why I'm happy to share what's worked for me and encourage people to try those things if they're interested. That's what motivates the early Agile people I know.
But consultants are mostly in business to make money, and the bulk of their audience is clueless, so most of what gets sold as Agile today is tuned for them. I go into more detail on that here: http://agilefocus.com/2011/02/21/agiles-second-chasm-and-how...
> Also, experts will evolve a workable process for any given project without needing to follow any particular Agile process to start with. That's one of the reasons they're experts.
True, but that ignores the people who aren't experts yet. They're the ones I care most about. And it ignores the substantial costs of learning all the important lessons yourself; it's often better not to reinvent the wheel.
> [..] teams within a large organisation [...]
Personally, despite 10 years of looking, I have never seen teams within a large organization successfully adopt what I would consider a serious Agile approach. I have seen a lot of large organizations adopt Agile jargon and get nothing out of it, though.
The problem is that Agile methods require a shift in power relations from top-down to bottom-up. Their approaches and values are basically antithetical to a lot of American business culture and MBA dogma. And successful exploitation requires deep change on the business side as well.
I think Agile failed for the same reason that Lean Manufacturing failed here: the important part wasn't the easily observed rituals, but the cultural differences. And the vast majority of the people who are "doing Agile" (whatever that means) would not accept deep cultural change.
At the risk of stating the obvious, has it occurred to you that the reason you have never seen a teams in large organisations adopt Agile successfully is that the basic philosophy of Agile doesn't scale to large projects?
You can get away with a lot of informality and doing things on-the-fly for small projects, when everyone involved can get around the same table in a meeting room and a single expert from the customer side can make all significant decisions. When your project team is made from 200 people divided among geographically and temporally diverse teams and the customer for your business management software is a 100,000 employee global corporation trying to co-ordinate the activities of six divisions in five key areas? Not so much.
Why yes, that has occurred to me. It's possible. But I doubt it for reasons I explained above. The few successful Lean Manufacturing adoptees in the US are on the smaller side as well. That's not because it doesn't scale, as Toyota is now the world's largest car manufacturer. It's because small companies can make cultural changes more easily. See the This American Life episode on NUMMI for a poignant illustration of that.
I think the answer to this question in software will come from companies that have started small with something like XP and grown into large companies. I suspect you'll end up with something like the cellular model that bodies use to scale. That's how it seems to be going at the companies I've visited. One startup I know is at 50 or so engineers and they are doing fine with a lot of small teams working semi-independently, releasing daily, and coordinating as they see fit. Or take YouTube. They have a few hundred people but they maintain a weekly (or more often) release schedule, power works in a bottom-up fashion, product management is decentralized, and the teams are loosely joined. It works fine for them.
Please provide studies to the effect that the practices you consider "proven" are, indeed, effective and references to the fact that they were widely adopted before the agile movement.
Because, so far, you've drawn a line based on personal opinions -- actually, just your own personal opinion -- about what's effective and what's not, what was common practice and what was not, said everything effective was common and everything not common is unproven, and then, based on this edifice or your own single opinion, declared that the agile movement didn't improve anything.
I'm very impressed with how much you deduced from your own thoughts, which means you are probably omniscient. In that case, you'll have no trouble providing actual evidence backing your statements.
The basic problem with the Agile movement is that it took a few
proven helpful practices -- almost all of which were well known and
often used by experienced software developers before the buzzwords
were attached, even if managers didn't realise -- and instead of
promoting those, it wrote a "manifesto" so vague that it would make
a political speechwriter proud, which didn't actually say anything at
all about the practical steps that were useful.
Avoiding creating a list of helpful practices was by design. The signatories of the Agile Manifesto were all very seasoned developers, and like all seasoned developers they knew that a practice that helps in one case harms in another.
You can kind of see how it might go wrong from there:
* Seasoned developer makes list of helpful practices,
without providing context, i.e. "Helps in this case,
harms in this case."
* Unseasoned developer doesn't understand when and where
practice is "helpful", tries to use it in all and every
case. Sometimes it works, sometimes it fails.
* Unseasoned developer who complains: "Helpful practice X
didn't work for me" is told "You're doing it wrong."
Again, no explanation of when and where the practice is
helpful is provided.
The takeaway here is to always, always, ALWAYS describe the context when describing a solution, i.e. "This is what worked for me, in this situation."
I don't notice any hesitation from those "seasoned developers" when it comes to advocating specific practices elsewhere. According to Bob Martin, if you don't do TDD you're not even professional, for example.
Yes that's what I was implying. (I guess I wasn't clear enough.) If Bob Martin said something like that, and he couldn't list even _one_ case where TDD wasn't the best approach, I'd question whether he had fully thought out the problem.
Warning sign 0: When it became Agile-with-a-capital-A
Capitalization means it's become a proper noun. A specific Thing. And everything that comes with that: canned methodologies, books on how to implement specific "Agile" processes, management retreats, certifications and all sorts of other senseless bureaucracy.
In other words, processes, tools, documentation, contracts, and plans. Not quite the kind of agility they were talking about in the manifesto. ( http://agilemanifesto.org/ )
I Can't agree more. I successfully applied agile ONCE during my 6 year tenure at a fortune 100.
When approached with the question of what I thought was needed to make agile work, my reply was simple: "start with a good team who can handle it."
That advice was of course tossed aside as corporations (especially middle managers) whose primary concern is the ability to shift resources cannot fathom the notion that agile starts with the individuals who compose the team. The sad truth of it was that there very few individuals who are actually capable of being "agile" after being subject to heavy handed processes for years.
From being on a few agile projects at Fortune 100s, I can attest to this. I once worked in parallel with another agile team that was doing a separate but integrated portion of an entire solution (it was a bunch of apps under one umbrella/initiative). At the risk of tooting my own horn, the other team was disbanded after three months without progress.
The inherent problem that I saw was how those companies treat developers. I'm a consultant, so I was able to do the projects and move on, but it's clear that the companies look at developers as commodities and cannot fathom why somebody who says they can write code is not equal to another person who can write code. After all, aren't all accountants the same? (They're really not.)
Regardless of whatever process you throw at these large companies, if you have a mediocre staff, expect mediocre results.
Regardless of whatever process you throw at these large companies, if you have a mediocre staff, expect mediocre results.
I agree about staff, but I also think that mindset is a huge component. There are organizations out there that consider software to be a necessary evil, something that they have to do in order to do the real stuff that needs to get done.
Other organizations consider software development to be a strategic advantage. Something that they can do more quickly or more effectively to make the organization succeed.
If you're at a "necessary evil" organization, neither talent nor methodology will save you.
I've worked in a global company pushing agile, as well as currently running my own start-up. My problem with Agile (with a capital A) is that it has become something it didn't set out to be.
Individuals and interactions over processes and tools
Agile has become about processes and tools. Tools for burn rate management, estimation processes, test-driven development is a process.
Working software over comprehensive documentation
Documentation is now done, just in a different place. We seem to document everything but the product. Processes/interactions/estimation accuracy. And then this point is just used to not even produce any helpful/lightweight documentation by the end.
Customer collaboration over contract negotiation
BA's and managers speak to customers. They claim to be collaborating, but in reality, it's just a contract negotiation. You want x, we can deliver y, it will cost z. Even within a corporation, it's just contract negotiation. Devs still aren't pushed to talk to customers, and managers still don't understand estimation. Incentives are poorly aligned.
Responding to change over following a plan
People following Agile still have a plan. It just takes into account burn-rate, and a few other metrics. Changes still hurt, because the deadlines don't change. However, customers are now told it's OK "We can respond to change!".
These days, my aim is to get stuff working and maintain motivation. No more, no less.
I worked at a company which tried to push agile process upon us in a 'top-down' fashion. I believe this is a cargo cult behaviour caused by a lack of understanding of underling development issues.
Late-running projects that don't meet requirements are often a result of poor communication and a lack of concrete requirements. This is clear to most business people. What isn't so noticeable to many is that software development looks a lot like an iceberg. The business domain decisions are visible above the water while the architecture and implementation decisions are below the water line. The fundamental point I am asserting is that software projects grow in implementation complexity at a greater rate than the complexity of the business domain does. Software projects run into scalability issues far earlier than non-software projects.
At low-scale, increased centralisation of decision-making and communication increases quality and timeliness. However, at high-scale most companies make the mistake of continuing to increase centralisation on the business-domain and let this centralisation of project decisions be imposed upon the more complex implementation-domain. Doing so can backfire for the following reasons:
(1) There are too many decisions which need to be communicated and specced out by one person.
(2) Increasing the vertical height of an organisation creates 'chinese-whispers' type communication problems.
(3) Business-domain knowledge held by project managers is often not enough to make informed decisions that require specialist technical knowledge.
Now I have to be blunt. If your employees aren't high quality enough to make decisions or communicate on their own then there is very little you can do to improve your process. You can try to create mini-hierarchies of lead developers smart enough to make informed decisions and communicate in between groups of drones but this which will not solve implementation problems and yet again the centralisation will create similar issues as before. However, if you've hired smart employees you will have much better luck since you can: (1) scale decision-making and communication completely horizontally -- be careful to avoid coupling problems, (2) help people to form an iterative process that takes into consideration long-term goals, and (3) implement social interfaces (stand-up meetings being the classic example) to simplify and increase the quality of communication.
The reason this should be done from the 'bottom-up' is that people aren't so much as following a process but instead are iteratively learning their process. You need to step away to let this happen.
Thanks for this post. I agree wholeheartedly. A lot more effort has to be put into the 'learning your process' part. The problem is that this requires self-reflective and humble individuals who are willing to work on improving their skills. I see far too many who are not willing to improve and learn, and most of these individuals lurk at big companies. Such an individual sees only imposed constraints in the methodology, and no possibilities. Since these passive individuals are easy to control, management has formed as a constraint-imposer and controller above them. Introduce a new methodology, and the workers will only (rightfully) complain while the management translates it into control constraints. So the problem lies not within the process, but the individuals. Start with improving the attitude, then let the team members bring in useful parts from methodologies as tools to reach the goals you have set together.
Badly managed businesses that are run the "top down" way the author describes typically have no respect or desire to understand the product development process they depend so heavily on for success and profit. No amount of "Agile" or "Lean", no consultant or trainer, no Kanban or Scrum board can really ever hope to change this for them if these organizations cannot self reflect and see that they are standing in their own way.
The concepts and learning that has come from the agile and lean software movements are still valuable though. Valuable for those who choose learn and are willing apply them, test them, validate them and actually self discover what is and is not effective. But they are absolutely snake oil and worthless for those who are unwilling to put in this basic effort.
I think you may have accidentally nailed the crux of it in your first sentence. Most enterprises are not software shops and nearly everything represented in this blog post is not talking about product development... and if you're not selling it you're probably not going to be inclined to care about best practices or be passionate about quality or timeliness.
No amount of "Agile" or "Lean", no consultant or trainer, no Kanban or Scrum board can really ever hope to change this for them if these organizations cannot self reflect and see that they are standing in their own way.
You neglect the larger scale life cycle of such companies. Over time, in a large and varied environment, some group manages to do the right things. The real value of such group is recognized for a time, but then is lost to time and attrition. The project ages and becomes another outdated and sclerotic barrier to further progress.
The concepts and learning that has come from the agile and lean software movements are still valuable...But they are absolutely snake oil and worthless for those who are unwilling to put in this basic effort.
Basically, you want good people and a team culture that actually works.
Basically, you want good people and a team culture that actually works.
Bingo. Which is of course, the hard part to do and maintain. The rest of it is really mostly hand waving. I do believe in a lot of the lean philosophy, and I believe that finding a process that works is crucial to success, but I also believe only a team capable of defining, developing and refining a process will ever have one that "works".
The devil is in the details. You still need competent people to determine and estimate tasks, and to implement those tasks. Implementation is the biggie. How do you design the implementation? How complete are your tests? What are the important requirements of the task?
There seems to continue to be this idea in management that process will take the place of skilled programmers. That you can then just plug any programmer in, and the process will produce a usable product.
It's amazing how most other skill based industries require certification exams, to verify that you truly do know how to build a bridge, prepare a legal defense, or perform a hip replacement. Software development is the only place where process is thought to replace skill and knowledge.
No amount of "Agile" brand ointment can miraculously turn a rag-tag team of goof offs into a well oiled development crew. And likewise, a team of skilled developers can actually be quite successful without Agile methodologies.
My own experience with Agile is that it's mostly beneficial to the high priests that make their living from it...
There seems to continue to be this idea in management that process will take the place of skilled programmers.
This is true, but it's not what Agile originators meant. The early Agile stuff was exactly the opposite. E.g. the initial SCRUM paper starts out talking about "Small teams of competent individuals, working within a constrained space that they own and control." Extreme Programming specifically removed from management the power to arbitrarily mandate a schedule:
But if the programmers are running the show, what is management supposed to do? That's why there's a scrum master. Yes, you need a leader, but often that role gets pushed to a manager, who focuses too much on process. They should rename it the scrum playmaker, with each sprint focusing on a new playmaker, returning control to the team.
Managers also have a much different understanding of the word "competent" than the people on the team.
I hate to use the word "Agile" because the very act of naming seems to encourage the nonproductive behaviors the OP is talking about. Calling it a "process" is even worse, because it's a metaprocess, not a process.
Like he says, "it's the doing, the reflecting, and the adapting that count the most." That doesn't really need a name; that's just how you become good at anything. Software, jazz piano, raising chickens...anything.
Any productive methodology is a local adaptation to local conditions. Any codified methodology should be treated as a suggestion, not a constraint. If you stop observing and adapting your methods, they rapidly become less relevant and eventually turn into useless dogma. I'm pretty sure that just about anyone who is good at doing anything does this without being told, or giving it a name.
I hate to use the word "Agile" because it has lots of baggage at this point. There's the lingering smell of snake-oil salesman, the push-back against hardcore zealots who can't cross the gap between theory and practice, and the overly simplistic "I know what the English word 'agile' means" interpretation.
And while I love what a lot of people are doing around "Lean" software/startups, I'm starting to hate using that word, too.
"Those things that looked “shiny” a few years ago, like TDD or continuous integration, have lost their shine, and aren’t attractive anymore. Don’t believe me? check out the big conferences – seen these topics lately?"
I don't know about TDD, but IMO continuous integration isn't mentioned at conferences anymore because it has won. I haven't seen a software shop not doing continuous integration in years: ranging from huge private companies to government contractors to small fly-by-night shops. Outside companies for which software is part of their product, you still see a lack of it, but in terms of the software industry CI has won.
When I started professionally in 2004, I saw my first large-scale CI setup, and I was pretty impressed. You mean we can make computers run tests all day and night? Check to see if code compiles? Tell us when stuff breaks? It's brilliant.
It's the best part of being Lazy -- automate crap so you can concentrate on the fun stuff, like actually hacking code and making software.
I'm not sure how/when CI got lumped in with capital-A Agile. It's just a pragmatic idea.
Never in twenty years have I had the opportunity to do CI with a team, mostly because nobody I've ever worked for invested in such comprehensive automated testing that manual regression tests could be abandoned. Where I'm at now, we literally schedule integrations and releases around what QA has had time to test, and they're guided by project priorities rather than whatever got finished earliest.
This is extremely common in my experience (and the corrollary effect - we cannot refactor that code, it will be too expensive to retest. Dont touch that code path, add a new one with cut and paste)
Anyway, it is possible to bring Agile to a large organisation, but it generally needs first to solve tooling and build problems - I would not start a single sprint until there was source control, complete automated build, and enough tests to be reasonable. Amazing how much pain can be endured by teams. That much pain can teach you not to care.
This is exactly how it is in the company I work for as well. There has been a half-assed attempt at CI, the guy who was setting it up left before it was finished though, its rather sad that its this way but when a codebase is 10 years old it can be hard to inject new processes.
> "IMO continuous integration isn't mentioned at conferences anymore because it has won"
Oh if only that were the case! It hasn't won per se, it's just no uncontroversial. Everyone agrees CI is a good thing, but you'd be surprised how many companies still aren't doing it.
Absolutely this is the case. But not to single Agile out unfairly: Although people are doing a lot of great work researching processes and so on, Software Development as an industry is like this.
Consider Waterfall. If you try it, and it doesn’t work, proponents will tell you that you were doing it wrong with all the changes late in a project. If you try Ruby and your code is a mess, Ruby programmers will say it isn’t the language, you failed to write good Ruby code.
And so on, ad infinitum.
So, if I agree with you, why did I post this? Not because I assert that Agile will “work” if you just do it right, but that I think it’s wrong to modify it and then blame it for not working.
Come to think of it, if you do it “right” and then succeed, I also think it’s wrong to give it the credit. But that’s another story.
That's a really bad analogy. People can, and do, play cut-down versions of baseball all the time (in Britain we call this kind of game "rounders". It has very flexible rules, and yet I've never seen it "not work").
I’ve played rounders in Nigeria. But it isn’t a cut-down version of Baseball, it’s its own game :-)
As far as the analogy is concerned, I don’t see it suggesting that you can’t cut Baseball or Agile or anything else down. For example, XP started as a bunch of practices cut down to what its proponents felt were the essentials! What I personally read in the story is that if you are given something new, you have to embrace it for what it is and learn to do thing sits way if you want to benefit from it.
If you fight against it at every turn because it feels different or awkward, you are going to end up neutering it, and it will not be a surprise that it doesn’t deliver what you hoped.
I have experienced the exact same thing with programming languages. If you try to write Java programs in Clojure, you will walk away very unsatisfied with the result.
Getting things done is not that hard. I don't need a burn down chart and some dude with a Dilbertesque PHB role to do my job effectively. I view anyone waxing on about so-called agile in the same light as those who speak of synergistic paradigm shifts in innovation etc.
I don't need a burn down chart and some dude with a Dilbertesque PHB role to do my job effectively.
Sure, you don't. Once you have a team of people, though, you're going to have some sort of process, be it formal or implicit. Especially so when that team includes technical and non-technical people. And they are externally accountable, or face hard resource constraints.
And for what it's worth, the Agile originators were fighting against Dilbertesque PHB roles.
Is "Agile" is still the best out there? If I have a team of 4-5 bright people, the best way to get them working as a team to produce software for a limited group of people is agile/scrum, right?
As usual it depends. At the risk of spouting a cliche, there are no silver bullets.
The best way to get a team working is to analyze and enumerate the current problems and shortcomings, and find a way to reduce or eliminate them.
Is the problem with your team:
* They don't make deadlines?
* Too many defects?
* Disconnect between software and customer?
* etc.
Armed with that information, you can see if any of the multitude of Agile methodologies addresses the issues you have, and is something that will work with your team. You can see if you can cherry-pick a technique here or there from a given methodology.
Asking other people what methodology is best is like asking if Java/.NET/rails/django/php/lift is the best way to write a website. You'll get a lot of opinions that sound good and point you in the right direction, but ultimately you need to roll up your sleeves, investigate the options, and make decisions based on your current circumstances.
[And ultimately, this is the problem with the current state of agile. A consultant or evangelist will promote their brand of Agile as a silver bullet without taking the time to look at your organization's problems.]
"Agile" has kind of been losing popularity as a term to "software craftsmanship" lately which seems a bit more holistic and less prescriptive than agile/scrum.
Basically it says, "take the parts that make sense from these processes and implement them." Don't do anything without thinking it out fully.
But as bad as the average Agile adoption is, I disagree with the author. Let's take each of his points:
Companies are “doing agile” Yes. As an example, last year I had a huge manufacturing company (which I will not name) ask me to come in to help as an Agile coach. It was a weird gig -- no teams, no traction, just a couple of guys in an office somewhere. It took me a while, but I finally figured out what they wanted. They wanted an "Agile Expert" to help them rewrite all the processes so they could be Agile! To them, it was top-down, create and monitor the process, etc. This was a control game, and they just wanted an imprimatur for the whole thing so they could gain more control.
So you might think that I would agree with this point, but I do not. The reason is the same reason why these guys brought me on: Agile is being implemented from the ground-up whether companies like it or not. The natives are restless, developers are seeing the difference, and change continues to happen. Most of this top-down stuff you see is more a reaction to that than anything else. So yes, the guy in the top office is trying to take a big hammer and "do" Agile like he did everything else, but the guys on the shop floor are doing their best to do it the right way, no matter what.
The business and development divide has grown...we now have project managers as ScrumMasters. Yes we do in many places, but not always. Lots of teams are doing it different ways. I think you could look a this either way, depending on your personal experience. When I wrote a blog article last week about rotating the ScrumMaster role among different folks in the team (http://tiny-giant-books.com/blog/scrummasters-get-no-respect...) I had one CST come by and say basically "That's not a good thing. I've never seen it done like that!" But I also had a few people email me and thank me for bringing it up. It varies.
We can’t even appear as a united front. This is a biggie, but it's a feature, not a bug. Remember: Agile is not like Scrum. Scrum has a certification process and an ivory tower where you go and get blessed. Agile is just best practices for iterative and incremental development. It's a superset of Scrum, sure, but don't mix them up. Because it's best practices, there's a lot of conflict out there. But that's great! Give me a dozen different options and opinions, let's talk it out, and then I can make an informed decision. The noise is a good thing. People get mad because Agile seems so slippery. One guy said, "It's almost like 'whatever works the best' is then just called Agile." Yes! Exactly! It's changing as we continue to try new things and share more.
So cheer up! I'm starting to hear from companies that want to use Agile for all of their business processes, not just technology development. Eric Ries is doing an awesome job of taking Agile/Lean ideas into the startup world. I'm so optimistic I even spent a few weeks putting down my best thoughts for how to start your team on the best foot. (http://amzn.to/xUgFxL) I wouldn't have done that if I didn't think the market was out there for it.
I think Agile adoption is always going to suck -- but it's just because of the nature of trying to get large numbers of people to think differently. If your experience with Agile adoption sucks, it's probably because you are working in a place where a lot of things suck. That's a much bigger problem than just Agile.
I took a quick look at your ebook - $50 seems pretty steep, particularly when I can't view more than the first page, and I don't know how many pages there are. A table of contents would definitely help know what the book's about.
Also, it's confusing how you've laid out the "You're doing it Wrong" parts. I'd merge them, call it "You're doing it wrong if..." (the if... is important) and make it all bullet points or else separate the points a bit to make them stand out more (I'm not 100% sure you can do that last bit in Kindle though).
There are other things, but those are the two that I immediately noticed.
Thanks! I have a TOC -- just need to figure out how to make that available on Amazon.
The book is very short, perhaps 20-30 pages. I make no apologies about that. The point is to be direct and provide lots of instruction. So if you're looking to pay for your book depending on it's weight, not it's usefulness, probably not for you. I know some of the best hacking books I've owned have been short. I'm hoping to make this book more like them and less like those huge encyclopedia monsters I have on my shelf.
For anybody who wants to see some sample pages and the TOC, just shoot me an email. Happy to help out. Also for more book information, check out the landing page I made for it, http://tiny-giant-books.com/scrummaster.htm
The landing page is definitely a better place to send people to than Amazon, at least until you get the TOC issues sorted out. And you probably want to expand the book description on Amazon too - currently it's only a couple of sentences.
Sorry, but I'm not going to use "each person's generic current favorite practice" instead of "best practices", even though that's exactly what I mean. It may be more correct, but it just doesn't sound right to me. You are correct that the term is overused and doesn't provide a lot of descriptive power. Part of the problem is that when I say "Agile is best practices around iterative and incremental development" what I'm _trying_ to say is exactly that: it's fuzzy and difficult to get your head around and understand if you're new to the field. it varies greatly depending not only on the person you ask, but the particular circumstances. Everybody has a different set of favorites. So the things you rant about actually are good things for purposes of my description. The frustration you describe is exactly what newcomers find. I am not saying that there is a best way to do things which I will simply not describe. I am saying that a dozen people will have a dozen best ways, each of which may change drastically depending on the particular situation. I must describe this situation, and the phrase I used appears to me to be the best for doing that.
Isn't this a sign that "agile" as he is defining it simply isn't working?
More importantly, why is this a bad thing? We don't really think that this process was "the" process that would stick for the next hundred years, did we?
When Gandhi was asked what he thought of western civilization, he replied: "I think it would be a good idea." I feel the same way about the widespread adoption of Agile methods.
What most people are doing under the name "Agile" bears little resemblance to what many of the early adopters used the name to mean. Not that anybody thought it would be the One True Process. As one of the XP originators says, "Extreme Programming isn't where you'll end up. But it's where you should start."
I once worked in a very nice team. We used Scrum. We delivered before deadline and the bugs were few. The scrum master was a developer gone PM but with total understanding of which problems you come across as a software developer.
The project would probably have been ok without agile since most of the team was good at their job but I felt it made the process more inspiring and fun. It makes me sad that a lot of people seem to just dismiss it because of bad practices.
It's frustrating to me to see So many discussions around "agile" practices. Agile is a set of principles. That's it. I hear less disagreement with the principles and more disagreement with the process in which case I ask, why are you stuck with that process if it doesn't work for your company? It isn't because you're Agile. It's because you're not.
I don't believe that agile is really going away, but it is definitely not really the hot thing anymore, at this point, many years since it first became a trend.
And probably a growing number of people are going to be put off by their experiences with "agile", if they are anything like mine have been.
Managers love the idea of releasing more often and understand that. And that part can be an improvement.
However, quite a few people seem to think that means you can fit that many more features in, which is false.
This is the big challenge for agile in my opinion: automated testing or even testing at all, is hard for many development teams for a number of reasons. Not to say that the actual skills or knowledge are particularly difficult to acquire. But practically speaking, putting into place a good testing and release system are challenging for many teams.
And doing "agile" (read: code and release more features faster) without solid automated testing or at least one real QA analyst who is busy, will guarantee that you have a ton of regressions and broken deployments. Now, since you are releasing much more often, you will still gain from getting much more feedback from customers, but there will be loud drone of bug fixing conflicting with the "agile" manager's push to release more features.
If your implementation of "agile" actually includes frequent refactoring and you don't have unit or at least integration tests, you are definitely going to be sorry.
Test-driven development requires a little bit more discipline, sometimes more up-front design, and sometimes investment in developer training. If TDD or the like is off the table or only partially implemented but you are still going ahead with "agile", then you still might survive, if you have a real QA analyst and solid process. My perception and experience is that most managers will decide that hiring an additional person "just for testing" is "not in the budget" and if they do bring someone onto the project for QA they will think they can use just any person rather than hiring a single QA professional.
That has been my experience with agile. Several years in, I am still working on trying to discipline myself and with my latest project I am having a bit of success with vows-bdd integration tests. But TDD/BDD etc. is a skillset that takes time to develop and a discipline that I believe young developers will greatly benefit from if it is part of their early training and experience.
Good code is written by good programmers. Bad code is written by bad programmers. Bad management can hobble good programmers, but no form of management can make bad programmers write good code.
(I'm not saying agile is bad, necessarily, just that it does not live up to its hype.)