> But my fundamental reaction is, what does failure mean. Because in my world view failure should be expected and accepted and learned from.
That's fine in some domains. It's not acceptable in others (i.e. you cannot just try things and see if they stick).
But regardless of that, you still want to minimize failure. So if one methodology (Agile) pushes to spend less time on fixed and clear requirements upfront, and this leads to higher failure rates, this is still an issue.
Agile is about reacting to changing requirements. If the requirements never change, then agile doesn't promise to be better than other methodologies. It came from a rapidly changing crashing markets in the .com bubble. How would the other methodologies fare if requirements changed two or three times in the middle of the project? That's when you need to be agile.
How did they even measure "success" if the requirements weren't given up front? What did they expect at the end?
Agile means delivering very small increments to users at a frequent pace, allowing them to provide feedback early and often. When done correctly, this eliminates a significant amount of guesswork. However, if the client (or, indeed, your own team or management) does not fully understand the process, things can go spectacularly wrong in many different ways.
We still need to explain what "agile" is. Ask yourself why do we "need" to deliver small increments at frequent pace? Isn't it to be able to react to changes?
It's main purpose is to react to changes, and make the whole process sustainable. It's literally one of the 4 main lines of the manifesto and the second of the 12 principles, and included in the meaning of the term "agile" itself, while guesswork isn't mentioned. But I can see how trying to predict what the markets will look like in 6 months involves a lot of guesswork, which is eliminated. I hope you can see that the only reason why guesswork is tricky is that there are a lot of changes in the meantime, so the ability to react to them is the deeper, actual main purpose.
I mean you are entirely discounting that it also allows engineers to pick their tasks and pace them as they see fit, as well as have input into the creation and prioritization of those tasks.
Everytime I hear an engineer complain about this, I ask them if you wish the PM would just write a PRD full of guesswork tasks and you execute on them unquestioningly.
Some engineers complain about having the ability to pick their tasks and have input in prioritization and planning? This has always been the most attractive part of agile for me. Self-organized team. Built around motivated individuals. Striving for technical excellence.
That's cargo scrum, very common disease. Teams without autonomy or self-organization. The queue should consist of higher level tasks, milestones, and the team should rework them into the small tasks that will actually go into sprints. Of course there is a motivation to pick the higher priority tasks first, but team members should be able to pick a task with which they can most efficiently contribute to the sprint goal, it's not always the top one.
Projects with fixed, clear and unambiguous requirements (and the necessary counterpart - fixed budget) I worked on typically turned into a nightmare, because they were not actually fixed, clear and unambiguous. You could pretend everything was fine for like 80% of the allocated time, the next 80% were spent in bickering over possible interpretation of some not-so-unambiguous line in the spec, death marches, blaming each other (client and supplier, but unfortunately also within the company).
The main innovation of agile is showing failures early and thus being able to quickly react and correct course, meanwhile in the waterfall the project spent most the time in a quantum superposition without possibility for correction.
> It's not acceptable in others (i.e. you cannot just try things and see if they stick).
Why not?
> So if one methodology (Agile) pushes to spend less time on fixed and clear requirements upfront, and this leads to higher failure rates, this is still an issue
That is an incorrect characterization of Agile. Maybe some "agile" companies use it as an excuse to avoid requirements, but that is definitely not part of the process.
If you fulfill the role of a supplier of a component in a bigger product (especially in an industrial setting), you cannot "try to build something".
People are not happy if you tell them that you want to iterate on their control system and deliver incremental value, or tell them that it's not possible in the contractually agreed way.
> That is an incorrect characterization of Agile.
These are also the cases where you just have to work out overarching requirements upfront.
Ah, fair. This is a startup site so my mind was on organization that have the freedom to pivot.
It is supposed to be that new requirements come from the customer though, working closely in conjunction with engineering team reps. So it’s more like “deliver MVP control system software, test it out with actual users, then incorporate their requested changes into the next sprint.”
I’m not surprised if most so-called agile teams didn’t do that though.
Yes, that's possible for product development in a startup (even necessary, because you don't know what you need to build), and in web dev, end user facing apps,...
For many other domains, the full incremental / Agile loop is not possible, and becomes disaster if it results in skipping upfront planning:
The counterparts you need to integrate with are often not done yet. Maybe the hardware is not finalized. The end user is years away. Maybe each test run is hugely expensive.
There is no MVP here, as in the system works as expected or it doesn't.
You cannot meaningfully iterate the core software running an MRI machine, a surgical system, financial clearing, pick-and-place machines, automatic rail protection, etc. towards correctness with the end user.
My startup is a hardware company where we are our own customer. You absolutely can iterate these kinds of projects instead of top-down waterfall planning based development. It just requires dog-fooding the product and using mockups or simulations for parts that haven’t been implemented yet.
> So if one methodology (Agile) pushes to spend less time on fixed and clear requirements upfront
Fundamental to agile is to have clear requirements, the time is not fixed.
With reference to the quality (scope), time, and cost triangle, each of the major methods picks one point to emphasise, another to fix (not allow to change), and another it manages the change of.
Agile wants speed so time is its emphasis, it does that by fixing cost (over a release period) and allowing changing scope, and it does that by limiting the features that go into a release. That doesn't mean that the features required should be unclear. I will deliver these 5 features out of 30 in the next 2 weeks at X dollars. If one of the features isn't ready, we'll drop it. No, I'm not doing features 6,7,8 till later. Yes, you can change you mind about what you want for later releases.
Lean emphasises quality and fixes cost while allowing time to change. You can get your burger for X dollars but you're 5th in the queue, you'll have to wait.
Traditional is about predictable cost, it (tries to) fixes time and allows change in quality to remain on budget. If everything goes to plan you should come in on time and on budget but maybe with less features or worse features than you wanted.
Of course, we know that traditional done badly produces over-budget, over-time projects that don't do what they were supposed to. It seems that agile methods are being used as cover for bad management because, as this entire thread is evidence of, there is widespread misunderstanding of the basics of project management. If you're going into an "agile" project and the requirements that you have are unclear then you need to kick up a fuss and hold the PM and the product owner to account. They should certainly be clear by the beginning of each sprint.
That's fine in some domains. It's not acceptable in others (i.e. you cannot just try things and see if they stick).
But regardless of that, you still want to minimize failure. So if one methodology (Agile) pushes to spend less time on fixed and clear requirements upfront, and this leads to higher failure rates, this is still an issue.