I like how he tries to supplant agile with something that doesn't even attempt to solve the same problems that agile (like waterfall before it) attempts to solve.
Both agile and waterfall methods attempt to give you a pattern by which you can predict when software will be delivered. That's the main business value. The suits don't care how the devs operate, provided they can get things on time.
Waterfall tried to do this by trying to understand the problem as fully as possible, to liken it as much to previously solved problems, and to involve gurus to say "that kind of problem will take X amount of time to resolve", and thereby create milestones and delivery dates.
Agile instead said "look, we don't know enough at the start of a project to do that. Let's instead keep track of everything we want to do. Let's try and estimate each set of tasks (stories), individually. And then let's rank them in priority. We can measure how good our estimates are, we can modify them, we can generate more data and determine what we'll have at a milestone, and then can either push back the due date, or at least recognize that we won't be able to ship the entire feature set at that time.
Continuous delivery can be done with either one of those (it almost never is in waterfall, but it -could- be). But by itself it offers a business nothing for planning purposes. All it does is allow an immediacy, a "as soon as it's done it's out in front of the users". This may or may not be a good thing to the business, but it doesn't solve the basic issue that business people want to know when they can expect a given set of features to be live.
That's one of the most level-headed and succinct description of the differences between Waterfall and Agile I have read.
I would also add that, in my completely personal and anecdotal experience, they are not one-fits-all methods: in general I find Agile much more suited to developing a "product", while Waterfall makes more sense when the aim is to deliver a "project".
My takeaway from the post is that he wants to replace Agile (clause 1: "individuals and interactions over processes and tools") with an ill-defined process (CD) and a big old pile of tools.
Call me an agile zealot fanboy or whatever, but that doesn't feel to me like progress (and, inter alia, more or less the opposite of what Dave T was complaining about in his own agile is dead thing).
I'm not a management guru, so I might be off-base but Agile isn't about sprinting tasks. Tasks, broken up enough, are much smaller than a sprint. It's about having a deliverable at a certain short term goal post. You cut corners and smash half finished features together to reach that goal so that you have a product you can pivot around (the word "pivot" always makes me throw up a little - maybe... reassess the direction of the project?). With test driven development and continual integration tools you constantly have a deliverable, so there is no longer any reason to sprint. You can "pivot" or whatever at any point in time
The real question is test driven development / continual-integration worth doing? CI isn't too controversial, but for TDD there is no clear answer and it really depends on the domain and what language you work in etc. etc.
Product owners create stories. These are titled things like "As a (type of user), I want to be able to X". The point of the title is to determine who this actually benefits. Then, they attempt to define it with acceptance criteria. These are a list of "what does it mean to solve this need". Ideally, it implies a set of tasks, and gives a decent starting point for QA to start testing. This is stuff like "When the user clicks X, the system shall Y" and "Should the system fail to do Y, it will instead (failure mode), and (inform the user? Stay silent? Whatever). Sometimes the product owner needs help from the devs to determine this.
The devs will then add tasks to the story. The story should be able to be completed in one sprint; the tasks are, indeed, much faster. They're tracked only insofar as to see progress towards the story's completion, but they're not nearly as important as the story itself. A story with half of its tasks complete is not done; the feature is not implemented, it's not ready to go out. When all the tasks are done, the story is handed off to QA to vet; at that point the story is done, and it can be shipped.
The dev team is only ever committing to what can be done in a sprint. They should have an idea of how many story points they can handle in a sprint, such that they can work with the product owner to determine the stories they'll work on in that sprint.
When the estimates start lining up with what is actually achieved (that is, the team has a velocity of, say, 40 story points per sprint. And they're completing 40 story points per sprint), the product owner can start planning around it. "We have four sprints until the business wants the next milestone. As such, I have assigned 160 points worth of stories to try to get in for that". And that's reasonable. And then, if anything emergent comes up, or new stories take priority (a 'pivot', if you will), the product owner knows they can't manage it; they either need to replace a currently existing story with that emergent/newly prioritized story, or, they need to slip the schedule.
And that can actually work. It requires honesty, transparency, and a desire to actually get shit done, but it can work. The problem is oftentimes people or cultures value CYA, politics, and 'leadership' over getting shit done, and all of those make agile little more than a scapegoat for why everything is on fire.
And it's only one way of being agile. Kanban, for instance, still has stories and tasks as described, and it still offers velocity, though measuring it slightly differently, but it isn't concerned about the sprint boundaries.
What always annoyed me about Story Points is that it's a measure of complexity, not time. At least that's what Story Points are supposed to stand for.
When you state that a team should know how many Story Points it can handle, it would make much more sense to see Story Points as some measure of time. If you've completed last few Sprints 38 to 42 Story Points, with 2 full-time devs, than one could state that a single developer on average can finish 20 Story Points in Sprint. And if a Sprint is 2 weeks (10 days), that would mean maybe 2 Story Points per day for each dev.
Yet, in every business environment I've been in, the SCRUM fanatics always state that "No, story points are a measure of complexity". In practice, to me, this makes no sense. At least not if one wants to use historic Story Points to estimate how much work can be completed in the next Sprint.
The problem is that time isn't something you can chop up neatly.
For ease of use, let's say that a story point represents 8 hours of actual work. That gives us 168/8 points a week, or 21 points. Of course, we lose 7 of those to sleep, 4 to the weekend and another 5 to the evenings during the week, leaving us with 5 points per week. You probably lose a point to lunch, coffee, bathroom breaks and mingling with coworkers and another point to meetings and interruptions.
That leaves you with 3 points out of 21 potential points in a week, and you aren't getting 3/5 done per day. Using story points instead of hours tries to get people to treat the sprint as a unit instead of time as a unit, since time can be split up to a very fine degree. Just because you work 40 hours a week doesn't mean you can complete 20 2 hour tasks.
I'm not that dogmatic. But complexity should loosely correspond to time, I would think. A deeply complex task is going to entail more work than a simple one.
That said, there's a good reason to say it measures the complexity, and not the time. You want to keep distance between your estimates and time, or else the business people are likely to come to you and say "You said this will take 6 hours, so I expect it tomorrow". Nevermind that it requires another task to be done first, that it's blocked on getting something from another department, that you only ever committed to delivering it at the end of the sprint, and that you underestimated how long it would take anyway (but it doesn't matter because you overestimated something else so it comes out in the wash), -you- said it would only take 6 hours!
Thanks for the explanation - that really clarified things for me. The breakdown between story, task, sprint and milestone is very interesting and well thought out. Good estimate generation is the holy grail
Well as a freelancer it's something I've sorta given up hope on being able to do. Thanks again for the information. Hopefully it'll give me some ideas.
The trouble is that in dynamic environments, the discipline necessary to properly practice scrum isn't really possible. It takes longer to define the requirements than it does to code, and once coded they change mid-sprint.
The trouble with waterfall is that it tries to predict beyond the scope of a sprint, which just isn't valid. Project estimates are asymmetrical curves (likely poisson?) and you can't add them up and expect them to cancel out: http://www.sketchdeck.com/blog/why-people-are-bad-at-estimat...
The problem is like recursive state estimation, except when you break it down just to estimate and add it back up you aren't actually taking any new measurements.
On most projects it's usually just better to estimate based on relative size to your last project, start delivering continuously, and do a forecast of completion (as opposed to an estimate):
The trouble with waterfall is that it tries to predict beyond the scope of a sprint, which just isn't valid.
This assertion is nonsense. There's nothing magical about a couple of weeks such that it forms a boundary outwith lie impossible predictions. The validity of predictions depend entirely on the understanding of the problem domain and the complexity of the solution space.
The single biggest benefit from agile in theory, IMO, is controlling risk by getting the customer in front of the software sooner, so it can be iterated based on feedback. The primary risk being controlled is building the wrong thing. But you wouldn't develop e.g. an autonomous driving subsystem for a car that way.
Agile (scrum, specifically) in practice is too often used simply to chop large tasks into bite-size stories to be fed on a conveyer belt to a team of more or less replaceable programming cogs; the sprint scope keeps blinkers on everybody so they don't look too far in the future, they just keep munching through stories.
And when agile is used in this way, not only can it be demoralizing, but also extremely inefficient: a focus on user stories typically encourages building small features that involve narrow vertical slices through the stack of an application. That's hard to parallelize effectively - related stories will affect the same bits of code and cause conflicts. If you can bundle a bunch of related features together based on how they are likely to be implemented, you can slice them up horizontally, and implement the different layers separately, using things like APIs and data models at the boundaries. This paralellizes quite well at the team level.
It's still not great for software design. It's still a very blinkered approach; you're not going to design an application-specific framework that makes implementing features easy. It doesn't allow any space for experienced developers who have foresight, and relies on refactoring to create reusable domain-specific abstractions. But refactoring isn't a user story, and a team munching on stories isn't in a good position to think holistically about a problem.
If you've got a bunch of good, productive developers, you've got to work really hard to get anything but good software out of them. They will probably self-organise into an effective team, whatever the methodology.
On the other hand, if you have a bunch of inexperienced, mediocre developers (myself included) then "bite-size stories being fed on a conveyer belt" is probably a good way to get productivity out of them. It's certainly a lot better than "you have 3 months to build this enormous system based on this 1 paragraph brief" which is pretty common.
You'll get a blob that gets harder and harder to modify over time. For a throwaway system and / or with piles of money, it may work until the system needs replacing.
Different people working on similar vertical slices through the system leads to slightly different parallel implementations and probably some duplication of helper logic. Refactoring won't get scheduled and it'll become technical debt in the way all duplicated logic is: not a big deal to start with, but an increasing source of bugs and features fixed / implemented in one place but not another.
Inexperienced developers won't be disciplined in keeping their abstractions separate; they tend to intermingle their abstractions so that the boundary is fuzzy. Specifically, they lack layering discipline. Instead of libA using libB which uses libC, they'll pass bits of libA into libB and return bits of libC. When implementing a complex algorithm that joins libA to libB, they'll write code that zips the two together within its convolutions, rather than creating adapters for libA and libB so that the algorithm follows naturally. And they'll model the domain, but nothing much more abstract, and write convoluted procedural algorithms in VerbingClasses (new ThingDoer(x).doThing(y)), possibly with interfaces for mockability.
And to be frank, many line of business applications can cope with this. The developers are cheaper and easier to find, and IT was always a cost centre anyway. It's no way to live if you love code, though.
Of course. If we lived in a world where good, productive, experienced developers were cheap and plentiful, we could do things very differently. As it is, it can be more efficient to pay cheap developers to continually patch (or even rebuild) big balls of mud than pay expensive developers to build it properly in the first place.
And I'm not going to complain, because I benefit from the current system.
If you've got a bunch of good, productive developers, you've got to work really hard to get anything but good software out of them. They will probably self-organise into an effective team, whatever the methodology.
I'm sure there is an element of truth to this, but I doubt it's as decisive as you're suggesting in real life.
If you've got a good team of expert developers, people who are both individually skillful at producing useful code but also good team players and able to co-operate, I'd say you have a decent chance of them self-organising. I'd omit the "whatever the methodology", because these are exactly the kinds of teams who don't want or need some consultant's pet methodology limiting their options.
Of course, there is more to building useful real world software projects than just producing a good design and implementation. Even the most technically brilliant team also needs a clear goal to be effective in practice. Capturing the requirements and turning them into actionable specifications is a significant challenge in its own right, one which requires a very different skill set that even exceptional developers won't necessarily have.
I suggest that one of the big differences between a highly skilled and experienced team and a team with more modest capabilities is that the former will immediately recognise the need for clear specs and try to do something about the lack of them. The kinds of development processes and methodologies we're talking about today are designed in part to shield the latter from the same responsibility, but consequently they also rely on having very good people to do that work instead (and by corollary tend to fail hard if the communication with customers and resulting planning work aren't up to standard for whatever reason).
The big difference is that three months into a two year waterfall project, if someone complains that we're behind schedule, no one believes you or gives a shit. They keep hoping for a miracle.
Three months into an agile project and everyone already knows there's a problem.
"you're not going to design an application-specific framework that makes implementing features easy. It doesn't allow any space for experienced developers who have foresight,"
I wonder if this is an extension of 'programming by poking'[1]. You replace serious thought and planning with a piecemeal, try-it-and-see-what happens process.
I don't know. Programming by poking works pretty well in Haskell, a language that's seldom accused of dumbing down. (And they've recently made it easier with typed holes.)
"It takes longer to define the requirements than it does to code, and once coded they change mid-sprint"
I've experienced one -or- the other of those, rarely both. The only times I've experienced both were due to product owners who wanted to be managers, to bring 'leadership', and so insisted on wasting my time with meetings that didn't actually lead to well defined stories.
If I've had a product owner who wasn't a waste of space, who met with the customer(s), actually understood what they needed, and then met with us to help define a story, then while that story might take a good 10-20 minutes to fully flesh out with good acceptance criteria, it almost never changed during the sprint. We might, on the demo and/or release of the feature, realize modifications were in order, but that wasn't because we did things incorrectly at first; we did them correctly, with something releasable, and useful, and that allowed us to learn something that helped us to refine it.
However, more often I've had product owners who are wastes of space. They'll slap a story together, maybe with acceptance criteria of a line or two, maybe nothing, and then hand it off to the devs to go do. We roll our eyes, make some assumptions for all the missing details, do something, demo it to the customer, and get "That's not what I wanted! Change it!"
The former is infinitely preferable. Even if it's a highly complex feature, that takes a while to get understanding and consensus on, it almost never changes once we do and the devs, customer, and product owner are all on the same page. The problem is it requires a product owner who doesn't suck, and the reality is most of them suck.
We recently lost two or three bigger costumers because of stuff like this.. No one talked to the actual users.. The program was full of functions, workflows and solving problems no user wanted to get solved.
They just never used the programs only for situations like : we have (another new useless feature, please test)
Uh, for me sad part is when I try to make those people write stories in more detail like "what should happen if data fails to load", "what should be default state", "Ok we have adding for an element, how we handle deletes". I got question "So should we go back to the waterfall?". Because we are so agile that not everything should be specified up front. But they do not get the scope, single story does not fall into waterfall...
The story is the last possible moment of decision making before the developers go and develop something. Obviously it may need further refinement at a later date once you have learned something. But, by definition, you now know -everything that can be known before development starts-, because development is about to start. As such, anything, -anything- that remains unspecified, is, again, by definition, undefined. It may be worth explaining that if failure conditions remain undefined, then you will ignore them for convenience, because whatever you decide to do will almost assuredly be wrong. If the product owners want -any- say in what it does in the event of something going wrong, and don't want it to be a surprise, and require even more work, then now is the time to say something.
> The trouble is that in dynamic environments, the discipline necessary to properly practice scrum isn't really possible. It takes longer to define the requirements than it does to code, and once coded they change mid-sprint.
Scrum is not trictly bijective with agile.
The concept of a sprint creates an entirely arbitrary deadline with an arbitrary box of work.
(I have a dog in this fight: XP with Lean trimmings, since I work at Pivotal and learnt it in Labs)
> It takes longer to define the requirements than it does to code, and once coded they change mid-sprint
It sounds like your product owner isn't doing his job. What specifically does it mean for requirements to change? If the specifications were unclear or incomplete, well you should have held out for clear and complete specifications. If you can't do that you have an organizational problem. But did the customer change their mind about what they want? Not likely, but possible I guess. The only thing left is that the owner didn't really understand the customer's needs.
Agile isn't about predictability so much as it is about 1: ensuring productive work is always going on; 2: ensuring the thing that ships is what the client wants; and 3: ensuring that something functional will actually ship.
Agile grew as a bulwark against the bad old ways of the "enterprise" trenches that led to the original "software crisis". There was a time when big software projects would not deliver, at all, as often as they would succeed. And even when they succeeded they often delivered the wrong thing. Daily builds, continuous integration, using the software itself as the source of truth (instead of elaborately negotiated specs), and focusing on iteration, these are how you can extract productivity from any team, and how you can keep on target and deliver something of value. It's not necessarily the best way, but it's one reasonably reliable way to get stuff done.
>>> We can measure how good our estimates are, we can modify them, we can generate more data and determine what we'll have at a milestone, and then can either push back the due date, or at least recognize that we won't be able to ship the entire feature set at that time.
Good waterfall project managers do that too. You can't bend reality to fit a gantt chart.
I stopped reading at "Agile had a great run in it’s youth by taking on some obviously bad practices such as 'Waterfall' software development."
Nothing about waterfall is obviously bad. It's how most engineering projects are done. Heck, it's how most projects generally in the world are done.
There's a reason it took a long time to move away from waterfall. It solves obvious problems and meshes with how the rest of corporations typically operate. In fact, with software delivery methods which were dominant back then, agile probably would have completely failed.
---
In the end, I ended up reading the full piece. One doesn't get much more oblivious and buzzwordy than this:
> We are now more squarely in the age of Microservices, Mobile first, Polyglot, post-Java JVM languages, GitHub, Docker and the emergence of a world being eaten by Software.
How is this piece on the top of Hacker News? This piece reads like it was written by a marketing consultant with no idea what these terms actually mean or how software is actually built.
Yeah and I like how he goes in on 'Agile Gurus' and then at the end of the article plugs his continuous delivery conference like it isn't the exact same thing in a different hat.
> We are now more squarely in the age of Microservices, Mobile first, Polyglot, post-Java JVM languages
Gradle provides only one language, i.e. Apache Groovy on the JVM, for specifying build configuration info. Before talking about "Polyglot, post-Java JVM languages", perhaps Gradleware needs to eat its own dogfood and provide an API so users can write their build configs in Jython or JRuby if they want. Be polyglot before preaching it.
Yeah, let's replace one consulting fad (the 4 or 5th I've seen in my career, I entered when "Waterfall" was still in vogue, then XP, then Agile, some variations of each too) with another.
Kent Beck who created much of XP, is quite sarcastic in what he writes at times. I think it's meant to be sort of sarcastic tongue and cheek sort of title so giggling at it is probably the right response. XP is not extreme at all, in fact it's quite reasonable.
People tell me that waterfall was never really a thing, and it was only ever a name for how things should not be done and that nobody ever actually advocated it.
Is this not true? I'm too young to remember what came before agile.
Pure waterfall as it's depicted by Agile consultants was never a thing. Think about it, how could you possibly code an entire app and only then debug it?
However, there was a waterfall-like process that was widely used. After getting the reqs and design in place, you started coding and you mostly coded. You would compile as you went along and made sure that your code worked on the handful of tests you put together as you went along. When the app was mostly coded, you then started more serious testing: writing more elaborate tests and verifying functionality. This step often highlighted buried errors that you then spent time debugging, writing additional tests, and so on.
So, the biggest innovation in Agile was the latter's orientation towards developer testing concurrently with coding. Which, of course, has had many other ramifications.
While the old way of doing things might sound klunky by today's standards, when done right it actually worked better than its reputation. Because devs hated spending hours in the debugger, they tended to code very carefully. The concept of "let's code this and run it through some tests to validate it" was unknown. Rather you coded so that you were pretty much sure that what you were writing worked properly if it compiled without error.
> Pure waterfall as it's depicted by Agile consultants was never a thing.
Yes it was.
> Think about it, how could you possibly code an entire app and only then debug it?
Which is why programmers hate waterfall so much, but you're absolutely wrong to think this isn't exactly what was being attempted. Time and time again, management in an attempt to cut the cost of programming time, thought the way to do it was to first build specification for everything and try and prototype the entire app up front, storyboard every screen, build out mountains of specs because naturally they think that's how you build things, with detailed blueprints so all the decisions have already been made. They couldn't be more wrong, but it's certainly the most natural way to think if you aren't a programmer and don't know better.
> So, the biggest innovation in Agile was the latter's orientation towards developer testing concurrently with coding. Which, of course, has had many other ramifications.
Agile in general had nothing to do with testing, the big change in agile was removing the big design up front, the specs and meetings and months spent planning something before being developed. Some agile methods like extreme programming certainly had testing as a big part of their process, but what differentiated agile from waterfall was introducing iterative programming where work was done in short week or two cycles and then delivered whereas waterfall wastes enormous time trying to nail down details that simply ended up being wrong come programming time.
tldr; agile is not about testing, it's about iterative development in short cycles with little planning and always has been and that's what made it different from waterfall; no "Big Design Up Front".
"Pure waterfall as it's depitected by Agile consultants was never a thing."
This is correct.
The first paper to describe a stepwise model was by Royce in 1970 [1]. The model he is describing is hypothetical and does not use the term Waterfall.
The first use of the word "waterfall" (including the quotation marks) is from 1976 [2] and specifically refers to [1], the hypothetical model. In [2], the writers specifically state that "so few" projects fit this scheme.
If you were programming in the 80s and early 90s, you would know that no one in programming ever referred to a "waterfall" model.
Even Kent Beck's seminal book on XP, written in 2000, describes many failures of software development in those days, but he does not once use the word Waterfall.
So, in summary: the paper that supposedly describes it describes a hypothetical system; the paper that first uses the term mentions how little the model is used; the term wasn't used by people during the era it was supposedly most popular; and the folks who initiated the new generation of software dev don't refer to the model.
I think it's safe to say that it was not a thing. Or, to be more accurate, if it was a thing, it was never the thing it became until the Agile consultants used is as a strawman.
but it absolutely was very common to use the process that it implies.
many man-years of gathering requirements, building lists of features, interrogating stakeholders.
people knew they would never get a second chance to get what they wanted, so they would throw every feature in that they could think of.
Then the developers would build it. more man years building. mostly they tested as they went along, but....different features and areas of the app were often build in isolation, so the most you could say was that you had tested what you had built.
the stakeholders were consulted with screenshots, and sometimes the app, but the Big Fear was new features and additional requirements (which would cause massive contractual headaches) so any contact was VERY carefully controlled.
Also it was assumed every project was massively valuable IP, so detailed designs and features were held very close to the chest..again, any contact was VERY carefully controlled.
Eventually, after months or years of effort the various parts of the app would be "merged". omg the cluster fuck that would occur, the compromises, and arguments, the agonizing...
..then more testing..
Once that was completed the brilliant new creation would be seriously unveiled to the users to cheers and relief.
Finally, the contractual battles would begin as the paying company realised they had asked for entirely the wrong thing, and the development company tried desperately to cover their ass and make a profit.
The single biggest problem that Agile fixes isn't the testing, its the disconnect between what the client wanted and what the developers actually deliver.
I was there in the mid-nineties, and it was absolutely a thing in enterprise style IT.
There were also several books on the matter, and also revised waterfall models. The very term was not always used early on (though around the nineties it did), but the schemes were the same.
Lots of public projects, including some I've been involved, were designed and managed in such a way.
There were no changing requirements until the things were delivered, which could take 1-2 full years. The design phase resulted in monstrous 400 or 800-page documents covering every aspect, and to apply for such a software tender you needed to write those in excruciating detail (I did that too for several projects).
The waterfall model, with the name and all, was also taught at our university (early/mid nineties). XP wasn't even mentioned back then at our level, though we discovered it on our own, reading Fowler, Beck et al in the late nineties.
There's some tinfoil conspiracy theory thrown around that Waterfall was never a thing etc, and it was just used to push Agile etc. I wonder what its adherents did in the 80s and 90s, but surely not programming in large enterprise/public/military etc software projects.
(That Waterfall cannot work is another thing altogether -- nobody did it 100%, as nobody does Agile 100% today).
People would refer to "The Software Development Life Cycle", which was by all means a waterfall-esque process (look it up on Google, you'll see the classic waterfall diagram used for it). The word itsself may have been coined as something to compare Agile against, but the process itself existed, and existed long after Agile had "won" in more traditional companies.
I ran projects that ran through a VERY traditional waterfall process as recently as 2010.
Saying that the term was never used isn't really refuting the point. It doesn't matter whether you coded back in those days. Anyone who looks at (or used) the software being produced back then can see that continuous software delivery wasn't really the norm, probably due at least in part to technical limitations. I'm thinking about your average Microsoft product, but there are plenty of examples even today of poorly-designed waterfall projects (e.g., healthcare.gov for Obamacare).
You're wrong. Look at the responses you're getting, many of us were there in the 90's and if you think waterfall wasn't the dominant method of developing then you just don't know what was going on back then. It doesn't matter if they called it "waterfall" or not, it matters that it's simply what they were doing. Massive requirements gathering and spec writing with development being considered like construction of a blueprint at the end of a months/years long cycle. That's how development was done before agile came around.
Waterfall still isn't dead and will never die because it's how people think by default, plan it all out, then build it. That's how buildings are made and thus it becomes the default methodology of all new managers who don't know what they're doing.
This is absolutely wrong, I have seen first-hand people seriously advocating (and actually trying to practice) it, even within the past 10 years.
The thing to remember though is that the 'pure' version of the process (in which you never return to the previous level - hence the name 'waterfall') was regarded as an ideal, not as something achievable. Nobody expected that there would be zero bugs found in testing that would require more coding work (why test otherwise?), but it was regarded as a kind moral failing, to be met with a lot of hand-wringing about why we can't be like real engineers, who never make mistakes &c. And the same applied to revisiting the design or requirements phases.
The next level of idealism held that even if avoiding returning to a previous stage was impossible, you could at least limit it to the immediately preceding stage. As Royce pointed out in the first paper to provide a convenient diagram of the established software lifecycle to cite, this never happened in practice either.
Over time the necessity of doubling back became less of a moral failing and was incorporated into the teaching of the process, without any acknowledgement that the entire concept was fatally flawed. It was also succeeded by several iterative but still waterfall-like processes (starting with Royce's paper and eventually followed by Spiral and RUP). Thus there are many folks who still believe that the pre-Agile process (they may not recognise that there was more than one) was just fine, and that the pejorative 'waterfall' was applied to something that never really existed. In a certain sense that's true, but it gives a completely misleading picture of the history.
Agile's contribution was the idea that any or all of the phases could be happening simultaneously - and, more importantly, that was OK. Plenty of people had been ignoring the orthodox methodologies and working that way, of course, but don't believe anyone who tells you that was an orthodox idea before Agile.
Much like most "scrum" teams don't really do by-the-book Scrum, it was very rare to do by-the-book waterfall.
If you defined waterfall to mean
- get every requirement written up in absolute detail before you do any design
- finish the design in great details before you write any code
- if the design process identifies problems with the requirements (missing/ambiguous/etc), then stop the design work and go back to requirements phase
- write all the code before you start testing.
- if during development you find an issue with the design, stop development and go back to design
etc
Then I never saw any project that worked that way.
But it was quite common to have a requirements gathering exercise, write up a document that covered the requirements, get that "signed off", then do a multi-stage design phase (usually we'd start that before requirements were signed off, since the probability of the requirements being rejected in their entirety was pretty low), sign that off, and then move into development to implement that design. And then test the whole thing at the end once it was "done".
If at any stage we found missing requirements, that would get raised as a scope change. If, during development, we found that the design was broken in some way, we'd have a design change (which was usually lightweight and was a couple of emails saying "we can't do X, so we're doing Y, OK?")
There's "Agile", the buzzword that all sorts of advice-givers have capitalized on commercially, and then there's the Agile Manifesto [1]. The Agile Manifesto has merit and is a set of values, not a methodology - it would precisely agree with your advice to chiefly "use your brain". It's unfortunate that the latter has been turned into a Frankenstein's Monster of the former. [1] http://agilemanifesto.org/
I hate to disagree since you support me, but thats not true.
The agile manifesto has plenty of problems that should be obvious with some thinking. Some of the statements are outright ludicrous and naive as much as others are valuable insights (but even then, only if you haven't thought about the problem very much and tried to approach it in reality).
I shipped software very quickly before learning much about agile at all... I ship it even faster afterwards, but not because I agree with it all... in fact I am convinced I am better off for ignoring the more harmful aspects and selectively choosing the ones that have a measurable benefit.
> The agile manifesto has plenty of problems that should be obvious with some thinking. Some of the statements are outright ludicrous and naive as much as others are valuable insights
Are we talking about the same document? What are the problems and harmful aspects that you're referring to? (Please quote the relevant part) This comment feels either a bit like a content-free middlebrow dismissal, or we're referring to different documents.
> Individuals and interactions over processes and tools
Have you seen this in practice and experienced how people work day-to-day? Leave them to their own devices and it varies a lot by personality. Some people need micro-management for instance... there is also thing that process is very important, and this point is self contradictory given that a lot of the agile manifest essentially describes process or a strategy for creating one... and as for ever relegating the importance of good tools.... ick.
?
(p.s. thanks for calling it out, its good to take these things seriously imo. its shows passion and care. :) )
I think one's assessment of the manifesto is largely dependent on how generously you read it.
> Individuals and interactions over processes and tools
needs to be read in conjunction with:
> while there is value in the items on the right, we value the items on the left more
Which means I can interpret it a way that is true, but not that interesting.
That is, I believe that a great developer with crappy tools will produce better software than a crappy developer with great tools
But I'm not sure what that really tells me other than "hire great people".
I guess, to some extent, it means "if the process says I should do X, but the team thinks it's a bad idea, then I should listen to them", but even that's is not a necessary conclusion from the statement.
In the non-software world, you only need to spend a little bit of time looking at workplace deaths to see that many of them are caused by teams that decided to ignore safety protocols.
Ultimately the only thing I really take from the agile manifesto is that doing the exact opposite is bad, but so what?
I don't want to ever work in development team that:
- Thinks that "following a process" trumps "investing in good people"
- Prioritizes writing documentation for software rather than making it work
- Has watertight contracts but ultimately disappoints the customer
- Sticks to "the plan" even when it's obvious that the circumstances have changed.
"Individuals and interactions" is not saying "hire good people", although I can see why you could read it that way. I've always understood it to be saying, "Get people together in a room and talk and figure out what to do", instead of "rigidly follow an arbitrary process".
For example, let's say that we did some project planning a month back, which says that we should start on a certain new project tomorrow. Everyone on the team is grumbling and no longer feels like that's a wise expenditure of resources. What should we do? Stick to our goal and follow the process, or get people together, work it out, and set new goals? Agile is saying, especially when you consider other bullets like "Responding to change over following a plan", that people subscribing to the Agile Manifesto value getting individuals together, collaborating, and responding to change more than following their plan.
But that is the point of the sentence. A process does not care about individual variation. What the manifesto is saying is exactly what you are saying.
Yeah, and that's a difference between a mature engineering discipline and software development. Though one could argue that software is so young as an industry that we have no clue what processes could be actually effective, so it's too early to set up one.
I totally agree ... Don't waste your time reading this article. And if you ever have the opportunity to write an article yourself, remember one simple thing - if you're going to write several hundred words of content, you should try to tell your target audience something they don't know!
Well it's working, it's on top of HN. The problem is the people clicking in the end. I didn't, cause I know with a title like that this gona be some cheap clickbait.
Sure, and I agree to a large extent, but if I'm doing woodworking and my blades are so fragile that they snap with the tiniest amount of friction, I'm probably not going to be very happy working like that for very long.
But why would you ever intentionally use (or vigorously defend) demonstrably bad tools? Just because you could do well with a bad tool doesn't mean you should or that you couldn't do better with easy-to-get other, better tools.
If you are forced to use an inferior tool like Agile, then sure, you want to make the best of it and be a craftsman who can still succeed.
That is an endorsement of being adaptable and self-reliant as an engineer, not an endorsement of Agile.
And if a tool is bad like Agile, it's useful to point it out and slowly steer the bureaucracy that feeds it into bastardizing whatever the next tool is, but hopefully inching forward to a better global state as well.
I'm an XP fan, but I'm always wary of saying "they didn't do real agile". It can descend into "No True Agile" pretty quickly, with a dash of victim-blaming.
Agile is actually hard to get to grips with. It requires lots of discipline. So did the pre-agile methods that worked best. And it gets half-done for the same reason that the pre-agile methods got half-done.
It's hard. Hard to do, hard to learn, tempting to stop.
Perhaps, but one problem often seen in large orgs (I've been there personally twice, and heard many more stories from close colleagues) is that they adopt the agile methodology (sprints, stories, etc), without adopting the culture shift (shit does not get added into the middle of a sprint, ever) necessary to make it work.
Agile process without agile culture objectively isn't True Agile. It's worse than everything it sets out to solve with the alternatives.
I disagree that Scrum is the whole of agile, so the concept of "sprints" doesn't exist for me. If the product manager wants to rearrange stories, that is his or her business. My business is advising on the relative complexity of stories and then undertaking to deliver the next story in the backlog. The roles are well-defined, after which, we talk to each other like we're adults trying to achieve the best outcome for everyone.
Putting aside that particular nitpick, it's easy to cargo-cult the practices. I sometimes use the analogy of the introduction of lean manufacturing in the US. At first what was introduced were the tools: boards, line-stoppages etc. But tools are just tools, by themselves they're not enough.
The uncomfortable, expensive and difficult truth in our industry is that people, process and tooling are not substitutable. You need the best of all three that you can get.
I'd prefer to say "misinterpret" than pretend to follow... but even so, following it as presented to its totality is still going to be sub-par.
Some of the points in the Agile manifesto are just arrogant, unjustified statements of fact or vague and useless comments.
Some people hate face-to-face interaction... and what the hell is a 'regular interval'... and sometimes a clear vision and direction from above achieves the very greatest of results that the team without it would have failed to produce.
Its not useless, but its not without fault either.
Clearly the article is just a thinly veiled advertisement, but leaving that aside:
Apparently the author hasn't really read the agile manifesto, because nowhere does it talk about scrum or sprints or xp or any of those things.
The agile manifesto is essentially a few sentences, that very eloquently say
"Don't be a dick and understand that the product people or customers don't really know what it's supposed to do either."
Much like the bible took Jesus's essential point "Don't be a dick." and wrapped a terrible mess of a system around it, the various agile methodologies have done the same, but the essential point of the manifesto is as relevant today as it was when it was invented.
(Note: I'm not actually religious, the Jesus thing was just a comparison)
Projects die all the time. Even languages can (mostly) die. The death of a project is a valid part of a project lifecycle and knowing how it dies can affect how you deal with previous project phases.
Agile never assumes project death is a reality. It philosophically implies that you can always measure and iterate your way to the next phase... but that's just not true. No one measures and iterates their way to a dead state on purpose and yet, projects end up there all the time.
So how are the projects getting there and what is Agile doing about it? And don't give me the Agile vs. "Agile" argument. Assume a flawless execution of agile principles with completely accurate measurements and successful iterations took place when answering that question.
To me, agile just assumes infalsifiability because it refuses to acknowledge, let alone, prepare for project death. And worse, according you agile, you can even measure and iterate your way out of the grave.
The operation of a projects has more in common with biology than immortality. Treating projects like immortals flies in the face of the reality of project lifecycles and it brings very bad mental states to problem solving. It makes far more sense to treat a project in lifecycle phases (birth, child, youth, adult, mature, elderly, dead) than assuming a singular monolithic philosophy that completely ignores the arc of a project.
A: "The code base is too unwieldy, no one understands it any more"
B: "Oh, just measure and iterate"
A: "But that just creates more unwieldy code!"
Agile makes sense as a subphase of a project lifecycle for when that lifecycle is spinning its wheels in the mud, not as a end-all-be-all philosophy.
TL;DR: Agile should used if a project stalls, not as a guiding philosophy.
Exactly, every 2 weeks, sometimes less, we consistently release what has been accomplished in a sprint. This "blogger" is talking out of his ass. Like always, declarations like this stir the anti-agile rage pot. It's getting more and more laughable as the years go on.
The problem with these buzz words is it seems most people can't seem to agree on what they mean. This author seems to think continuous delivery replaces agile, case in point.
It'd be nice if people could just layout concrete development / release plans and not assume everyone has the same interpretation for the buzz word that happens to be in vogue for a particular management team.
Oh, you didn't read all those development methodology books?!
I'd sort of like to speak plainly and get programming done, personally.
- Agile is Dead because people abused it, so we should move on from the term.
- Instead use the term Continuous Delivery. By the way this term doesn't mean anything, use whatever system you want.
- Lots of buzzwords.
I see it more as death by ubiquity. When the last laptop is sold, we'll proclaim the computer is dead, but we'll do so from our phones, tablets, headsets and whatever we use to tap into vast AI's that help us navigate the unfathomable complexity and depth of an ocean of information.
We will proclaim agile is dead without realising we constantly adapt our code to changing needs of our users and their representatives who are very deeply involved in the whole process and who give us feedback as we constantly release new functionality.
I see no indication that agile is dying. The consultant speak cult that developed around it is. The hugely processed SCRUM version of it seems to be. Yet the fundamental idea behind Agile (in the manifesto) seems to be as relevant as ever:
>Individuals and interactions over processes and tools
>Working software over comprehensive documentation
>Customer collaboration over contract negotiation
>Responding to change over following a plan
The process we've designed and use daily at Gridium most definitely adheres to those principles. We've tested and abandoned the idea of "sprints" and weekly estimation cycles. We have no idea what our "velocity" is.
Yet we collaborate with customers nearly constantly. We are highly responsive to changes in the business. We integrate continuously. We deploy very often. All ideas that I think where present when we all fell in love with the idea of "agile".
* Edit: Shame on me for not reading the article, because I basically summarized it :) Heap your downvotes on me HN!
Yep. If people checked their actions against the agile manifesto they would immediately see that most scrum teams violate "Individuals and interactions over processes and tools" and "Responding to change over following a plan". A lot of scrum teams I see are fully process and tool oriented and have no ability to respond to change. I hate when I hear "Yes you are right but we can't make changes because the sprint would not finish".
I also have given up on fixed sprints for my (not anymore) scrum team. We have a prioritized backlog and whatever is next gets done. More like Kanban. As long as people behave like adults and do their work efficiently it works pretty well. We still have a weekly velocity and that number is surprisingly reliable.
The daily standup ("scrum") did make some sense in the context of agile development. The idea is that you embrace the unpredictability of software and respond to change. So you take on a task, but you accept that the task is likely to not go as you expected. You may need to change direction. You may need to take a different approach. Or you may need 10 weeks instead of 1, and even that is getting fuzzy… under these circumstances, it can be useful to remain in frequent contact with the development team and business/end users users.
But as you pointed out, daily scrums can essentially devolve into what is essentially a policing of developers to make sure they are staying the course and are reminded daily of their deadlines.
It's actually just an unusually unfriendly version of waterfall. Why? Cause at least in waterfall, the business users are forced into something unfair and impossible as well. It's impossible for developers to accurately estimate completion dates, but it's also impossible for business units to accurately and fully spec out a software application. "You didn't meet your estimate"... "Yeah, well, you changed your mind".
It's a nasty business, but there you go. Best to avoid the entire thing and work on projects that are very high value but aren't deadline dependent, where the value of a developer is measured by working software at reasonable intervals rather than daily discussions of sprints, stories, and deadlines.
Actually, in some ways, that sounds more like what agile is supposed to be than all this scrum/velocity/stories stuff that isn't in the manifesto in the first place.
>Yes you are right but we can't make changes because the sprint would not finish".
That makes absolutely no sense because the sprint always finishes, on a scale far basis.
The only way to make a sprint not finish is to submit code that cannot be safely deployed, and that is a bug in the code, nothing to do with changes in plan.
A better analogy, is that some of the more advanced examples have "sublimed" as in Iain M. Bank's "Culture" series.
I was just having this thought last night. "eXtreme Programming" and Scrum seem to either exist in a clunky bureaucratic mutant form or have evolved into companies that use CI and prioritize really good communication. (Yes, people in the late 90's and early 00's did sometimes write it like that.)
> If you’re unable to keep pace with the rate of change of the open source components you depend on, you will be frozen and stuck and unable to ship software properly, a state we at Gradle call “Build Hell”.
Or do it like anyone who wants to build a business that lasts longer than 2 years: use stable, battle-proven software only and don't run after every hipster trend (remember EmberJS and friends). Before you use any software component, evaluate its support lifecycles and how well it is being maintained - especially with the trend of "modularization", where stuff like a string pad function is a separate module, every additional dependency you pull is a potential security issue.
Oh, and don't make the cardinal error of putting your node_modules/vendor folders into .gitignore. THAT is build hell once someone unpublishes a popular module...
Agile is "dead" because it isn't practiced. Where and when it is implemented, it's just fine. But 90%+ of "agile" is actually a sloppy, lazy, underspec'ed waterfall process hoping to achieve results in a stealth fixed-bid model, tracked on a kanban board.
The other side of the coin is that defending agile falls into the No True Scotsman trap too easily: Since people don't practice agile, failed projects aren't agile. Well, of course.
The real problem is how do you get the people who could be doing agile to be motivated to access the benefits of agile, and how do you get the borderline cases to acknowledge their limitations and address them.
It's when you do software development for services that are constantly changing and need to be up all the time, and you are not only responsible for the development, but also for the services being up all the time.
I work on a non-dogmatic agile team. It works pretty well, but only because we have engaged, technically sophisticated stakeholders. We have a process, but the personal relationships and collaboration are really the meat of it.
I think our process (which is fairly informal) would fall apart if different people tried it.
Yes, I have worked on agile Kanban teams. The experience was more or less the following...
1. What's the next task/bug/story we should be working on?
2. Build & test it.
3. Ship it.
4. Rinse & repeat.
We'd have meetings periodically to define new sets of tasks and figure out priorities, and/or to reflect on things that hadn't gone well. But these weren't necessarily scheduled on a regular basis. We did them as needed.
I used to study construction engineering and now I work in construction. "Waterfall" is still in use and going strong. Applying that directly to coding seems bit stupid.
I then swithed major to mechanical engineering, and there "toyota production system" is the thing. Agile and lean are just taking the underlying idea bit further. The original idea is to eliminate intermediate storages. This happens by making every production cell to order shit from previous production cells via kanban card. Previous idea was to just optimize volumes of every cell while giving zero fucks who, when and why the product was needed. But even inside the realm of mechanical engineering this sometimes is plain stupid. For example the workflow of large castings doesn't follow rythm of manufacturing cells. It follows the rythm of curing molds, heating metal and waiting the metal to cool down. What is "Agile" in software seems to follow TPS somewhat. You do one unit of work, you cash that in and start on second one. Id guess it's just coincidence that short compiling times and intermediate storage elimination happened during same timeframe and that resulted one term for both "agile".
I have very limited experience on coding, but neither of the previous seem to fix any underlying hard problems in coding. When I code, I struggle with interfaces, specs, debugging and data flow. The only thing that comes close to coding is design of mechanical parts. But that has lot less degrees of freedom. And there seems to be pretty much zero management fads applied to mechanical design anymore. The only thing everybody is saying is that "try to keep open mind in the beginning of process. And spend some serious time defining your problem, because that's what affects total costs most". The rest is just going by gut feeling.
We have ISO and ASME, but still interfaces are some of the most tricky subjects around in mechanical design. I'd expect the next meaningfull paradigm for coding to focus on interface design, as it's the one that requires communication between human beings. And it sets limits to what individual coders can achieve.
"Never stand between craftsman and his professional pride" Quote by Kalevi Aaltonen, but I guess he got it from somewhere.
I don't know what to think about it. The gradle.org website screams consulting with its slickness and free user guides/books etc. It does not look like honest, no-nonsense build software.
The source code is about 6000 Java files + 3200 groovy files besides many others. To me it appears excessive and over designed software. I will be very reluctant to use a build software which needs 300+ pages user guide.
TBH it is entirely possible that I just do not work on such complex software projects which needs tools like these.
After James Strachan was knifed 2 years after he created Apache Groovy, that ecosystem has been all about consulting and conferences. Software such as Grails and Gradle is shoved down people's throats with promises of managing complexity and giving users choices, but in reality these tools add complexity and the documentation obscures. Then the sales calls start touting conferences and paid services to manage it all.
There will be many comments pointing, "agile works great when implements the right way". While that might be true I'm willing to bet good money that agile has turned out to be really bad for 90%+ of the teams. I tried hard to save my org from agile and failed. Everything crashed and burned soon after.
This is my experience too. Companies bring in agile consultants to tell them how to release software better. But things never improve, and in two years, they do it over again (but with a different buzzword).
If your company has trouble developing software, the problem has nothing to do with development processes. The problem is always management.
With all this talk of Agile, continuous delivery, Waterfall etc, it reminds me of a Software Engineering course I did at University. Specifically one of the methodologies mentioned which was a Spiral model. The gist was that instead of having one monolithic Waterfall process, you would still do the steps in waterfall, but you would release more often and have smaller releases. So that after a few iterations you would spiral in on the desired solution.
Upon reflection this seems like a proto-agile methodology, and because it doesn't have its own set of buzzwords and extra processes it seems like it could be a good reactionary (as a reaction against full Agile) methodology to use.
The author mentions the Agile Manifesto. It's worth a read, and the concepts it focuses on are somewhat different than the impression that you'd get by reading about Agile in popular blog posts or websites: http://agilemanifesto.org/
Manifesto for Agile Software Development
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
That's the gist of it: being Agile means valuing the things on the left more. It is a set of values, not a methodology.
I think this manifesto is wise. Like Zen koans, contemplating and discussing it provides an understanding about what Agile is. Fortunately it's not cryptic, although the meaning behind the statements might become clearer with consideration and experience. "Individuals and interactions" is saying, focus on getting people together and talking and working things out, not following a predefined process or using complex tracking tools. "Customer collaboration over contract negotiation" is written, I assume, in the context of contract work, but it applies generally to the relationship between the software team and stakeholders: collaborate with your stakeholder directly, rather than trying to negotiate some kind of specification (contract) up front. The Twelve Principles of Agile Software are also worth a read: http://agilemanifesto.org/principles.html
These principles are what Agile is about -- not some particular methodology. Indeed, blindly following a methodology like Scrum would violate the Agile Manifesto ("individuals and interactions over process"). One of the things that I like about the Agile Manifesto and Principles is that they're not prescribing a particular method; they are instead providing some principles to consider while thinking about a situation. I've always liked Amazon's Leadership Principles for the same reason: https://www.amazon.jobs/principles - insight comes from thinking about what it means to embody a principle in a given situation; or what it means to value one principle over another, in what situations, and why. It's not prescriptive nor a method, but it's a starting point for discussion, and a way of talking about what we value as individuals, a team, or a company.
> "Customer collaboration over contract negotiation" is written, I assume, in the context of contract work
This should be about contracts between software development firms and their customers, where the development firm gives a quote to complete a project based on very detailed and legally binding specifications - which is a big problem because you can't know every detail before the project even starts. Hence a big part of the "Agile" philosophy.
This declares Agile dead without explaining specifically why. From what I gather, Agile is "dead" because no one actually does Agile, and only takes the "grab bag" of Agile terms without going fully Agile.
If no one's doing Agile, how could it be dead, since it was never alive? Maybe we should start trying to actually "do" Agile, to see if it might work, since no one's ever tried it yet, according to this blog.
We really need a bullshit register for this kind of people. They should not be able to get away unpunished for releasing something they already know is bullshit.
I think what features this article is exactly what we see the one facet of agileness of its practitioner. The distortion of science and engineering methodology and practice. Before you do a big thing, you have to prove a bunch of small things work as the foundation. Any increment will not be linearly scalable. Finally, what you claim should be strengthened by your data, charts, plots that are verifiable by others.
It's funny. Our devops team is in the midsts of a long migration from cfengine to Chef, and I recently started managing the project more like agile, with sprints and task planning meetings and so on.
Sure, for our clients and internal developers, devops makes CI happen. But as far as the devops themselves are concerned, Agile hasn't gone anywhere.
Waterfall is still the right choice for many products. The fact that the author can't even acknowledge that shows how little he understands about software development. The whole article is mainly a bunch of garbage from one narrow minded point of view with an obnoxious title.
Not sure if I agree with his post, but to hell with the gurus! It's a mentality we need to move away from, and it is nauseating the way people drool over so-called 'gurus'. Software is not religion... it's just software
Unfortunately it sells, and idiotic managers love it because it gives them a ready-made recipe, instead of having them come up with something that works for their product and their team.
This is just an advert. Can we please stop supporting all of these "XYZ is Dead" articles with no real solutions provided and an obvious unlying financial agenda.
"We are now more squarely in the age of Microservices, Mobile first, Polyglot, post-Java JVM languages, GitHub, Docker and the emergence of a world being eaten by Software."
sigh
I can see how Docker is relevant, Microservices are arguably relevant, maybe even GitHub, but that still leaves 4 out of 7 items that have nothing to do with continuous delivery.
Both agile and waterfall methods attempt to give you a pattern by which you can predict when software will be delivered. That's the main business value. The suits don't care how the devs operate, provided they can get things on time.
Waterfall tried to do this by trying to understand the problem as fully as possible, to liken it as much to previously solved problems, and to involve gurus to say "that kind of problem will take X amount of time to resolve", and thereby create milestones and delivery dates.
Agile instead said "look, we don't know enough at the start of a project to do that. Let's instead keep track of everything we want to do. Let's try and estimate each set of tasks (stories), individually. And then let's rank them in priority. We can measure how good our estimates are, we can modify them, we can generate more data and determine what we'll have at a milestone, and then can either push back the due date, or at least recognize that we won't be able to ship the entire feature set at that time.
Continuous delivery can be done with either one of those (it almost never is in waterfall, but it -could- be). But by itself it offers a business nothing for planning purposes. All it does is allow an immediacy, a "as soon as it's done it's out in front of the users". This may or may not be a good thing to the business, but it doesn't solve the basic issue that business people want to know when they can expect a given set of features to be live.