I think there are a couple of different problems driving overly complex software.
1. Cargo-culting. The article talks about this, but one thing I've seen consistently over the years is that something new comes along, and people latch onto it like it's the solution for every problem, everywhere and you're stupid or outdated for not thinking the same. It's the whole, "RDBMS's are dead!" thing.
2. Shiny-new-thingism. This is slightly different than cargo-culting in that people don't necessarily think the new tech is the ONLY way to do things, it's just that people want to use something new because it's hot. Then people get deeper into it and realize that there are some unknown unknowns that bite them in the ass. This becomes more likely the more abstracted things are, and/or the more moving parts there are. It's not that old technology doesn't have that problem too, but there are more charred carcasses of other people's failures to learn from.
The other thing is, as a developer there is a strong urge to try to use new tech because 'hot tech' = 'higher pay'.
3. The business requirements were not well-understood when the project was started (and may still not be well understood). Sometimes you're told to start building X, but you find out over time you really should have been building Y, while the customer asks for Z (no wait, W!) and you never get the time to pay off that technical debt. This is probably the hardest problem to address and is a large reason agile caught on.
4. Bad software architecture. Maybe you inherit software that has a crapload of technical debt because of point #3, or maybe it's because the previous team made some really bad design decisions from the get-go due to incompetence/inexperience. So then your choice becomes, do you tear it all down and start over, or do you limp along and add to the skyscraper made of popsicle sticks and bailing wire? Oh, and there's no requirements!
So you're left performing software archaeology like Indiana Jones, trying to figure what it was that a previous civilization was trying to do. Then you have the horror movie moment where you realize, "This...never worked, it was wrong all along..."
>The other thing is, as a developer there is a strong urge to try to use new tech because 'hot tech' = 'higher pay'.
That's a great point. We developers do often engage in "resume driven development" – often not intentionally, just subconsciously observing that some tech is hot and there's a lot of demand for it. So we look for excuses to use it, which is of course backwards.
1. Cargo-culting. The article talks about this, but one thing I've seen consistently over the years is that something new comes along, and people latch onto it like it's the solution for every problem, everywhere and you're stupid or outdated for not thinking the same. It's the whole, "RDBMS's are dead!" thing.
2. Shiny-new-thingism. This is slightly different than cargo-culting in that people don't necessarily think the new tech is the ONLY way to do things, it's just that people want to use something new because it's hot. Then people get deeper into it and realize that there are some unknown unknowns that bite them in the ass. This becomes more likely the more abstracted things are, and/or the more moving parts there are. It's not that old technology doesn't have that problem too, but there are more charred carcasses of other people's failures to learn from.
The other thing is, as a developer there is a strong urge to try to use new tech because 'hot tech' = 'higher pay'.
3. The business requirements were not well-understood when the project was started (and may still not be well understood). Sometimes you're told to start building X, but you find out over time you really should have been building Y, while the customer asks for Z (no wait, W!) and you never get the time to pay off that technical debt. This is probably the hardest problem to address and is a large reason agile caught on.
4. Bad software architecture. Maybe you inherit software that has a crapload of technical debt because of point #3, or maybe it's because the previous team made some really bad design decisions from the get-go due to incompetence/inexperience. So then your choice becomes, do you tear it all down and start over, or do you limp along and add to the skyscraper made of popsicle sticks and bailing wire? Oh, and there's no requirements!
So you're left performing software archaeology like Indiana Jones, trying to figure what it was that a previous civilization was trying to do. Then you have the horror movie moment where you realize, "This...never worked, it was wrong all along..."