I guess my problem with that claim is that, like, there are process issues, but software is mostly complex because the problem it's solving is complex, and "simplifying" past a certain point means actually not addressing the problem at hand.
The author's terms are not really well-defined. From the text of the article (which, admittedly, I only skimmed), it appears that he is talking about complexity for its own sake, in the form of trying to force a project to use microservices or all sorts of mumbo-jumbo that is not actually needed to solve the problem at hand. In essence, the argument is against over-engineering, and that is a problem that causes us a lot of grief.
I mean, yeah, there is ultimately a judgment call involved. The problem is that many people make judgment calls that are clearly biased toward complexity, with vague non-technical justifications like "This will be useful when we get 10x bigger", "<Last-years-buzzword> is outdated technology", and "This is based on a whitepaper from Google." Unless the project is explicitly intended for experimentation, these are not good justifications!
If you're using something like Docker or Kubernetes, good justifications are like "Docker replaces a bunch of scripts we used to have to use to deploy our compiled binaries over a fleet of servers" and "Kubernetes really helped us measure and manage the shared hosting environment that runs our lightweight services".
How often do you hear these justifications, or others that refer to actual improvements in the technical processes? Usually it's just "containers are really cool d00d" or "We spend way too much on AWS, and since the concept of not using AWS is unfathomable to us, we are collapsing everything down to 8 instances that run Kubernetes and spending tens of thousands of dollars on the engineering overhaul to support that instead of just provisioning cheaper servers."
I see. Yes, that's a fair point that I think maybe I've gotten a little numb to because the place I work at kind of has the opposite bias (i.e., "we got burned in the past so now you have to fight tooth and nail to be allowed to introduce tools that would make work on the project significantly more productive").
Yeah, there absolutely has to be a balance. The important part is basing the arguments on sound technical and business justifications. "No, we can't move off of Perl because it's what I learned in college in 1996" is not really any better -- but then, Perl isn't really so bad. ;)
In my (perhaps myopic) experience, over-engineering is a rare situation. And statistically we can assume most "engineers" fall in the 40-80% behavior range. The 1%ers write entirely inline Python scripts, and the 99%ers write Java EE structures.
The 40-80% try to find some balance between "now" and "the space of potential futures".
Point being, most developers aren't maniacs nor idiots.
Not my experience. Overengineering is rampant. I think I've seen more webapps that were using an internationalisation framework than not, despite not one of the apps I've worked on ever having been used in a non-English market. The whole ESB market seems to not have any actual use case. Every time I've worked with a "software architect" they've made the project worse...
Most developers, myself included, are ignorant and do things without quite understanding the reasons for them. Given how much easier it is socially to make the case for "we should follow this best practice" than "we should do the cowboy thing here", overengineering is the natural way of things. All we can do is fight to keep it down to a manageable level.
I'd say you're either pretty new, or you've been pretty lucky for this to have been your experience.
It's not that other developers are "maniacs or idiots". The issue is much finer-grained than that. Without strong technical leadership and responsible system architecture, it is easy for any project to go afield, no matter how talented the individual developers are in isolation.
Independently, most professional developers who've had a job for a while and jumped across a few different companies will be able to write acceptably reasonable code for the type of problems they're used to seeing. Even if they're using the cutting-edgiest, most fragile tooling, it will basically be fine within the context of a small demo and/or side project. Like you said, most of them are not maniacs, nor are they idiots, and they can easily manage something that is entirely under their sole control.
Without strong technical authorities and coordination in a team setting, however, you get these independent units trying to plug together, only to find that the couplers are fundamentally incompatible. Lots of individual developers are doing things that are reasonable in isolation and within the constraints of their specific timelines and concerns (some of these concerns are personal but still work-related, e.g., looking smart to peers), but which require huge effort to fix and make cross-compatible.
This is not just a matter of matching an API, though that may be part of it. The processes, incentives, and overall mindsets have to fit together well. Dev X needs to be able to read Dev Y's code from last week and understand what they were trying to do.
It's harder to read code than to write it, since writing is just an attempt to brain dump a structure you already hold, whereas reading is attempting to load a structure of someone else's conception -- a totally foreign construct -- based only on the slow, imperfect dump they were able to get on the day they wrote the code. You'll sometimes hear talk of catharsis in writing code, because it's a pleasure to express this mental model in a convenient reproducible context. You basically never hear the same of reading code.
Now, if you get Dev Y and Dev X on the same wavelength, then this isn't too big of an obstacle, and you have a sane, productive, happy group that is working under strong technical leadership to make cohesive, reasonable choices in the context of the broader team and company needs.
But if you can't synchronize the wavelengths, Dev Y moans about how stupid Dev X is the whole time, how it's better to use Pet Thing Z instead of Pet Thing A, so anything he touches gets its PTA ripped out for a PTZ, and how dev W should be moved over to replace dev X, since they're much more cooperative.
This continues for months and years. Stack enough of this together and it all just starts to blur. By the time you get to the wide scale view of the company's whole ecosystem, you're ready to trade in your keyboard for a mop and become a janitor.
tl;dr "a person is smart; people are dumb, panicky, dangerous animals and you know it".
My experience is generally the opposite - software is usually complex because people lack the skills, time and/or inclination to make it less so. The problems themselves are generally simple.
The world is chock full of developers who want to reinvent the scroll bar, badly, in JavaScript and seriously, genuinely believe this is both a good idea and a wise use of their company's precious development time and money.
It often looks akin to a bunch of civil engineers building a large rail bridge with no up-front design by making short-sighted decisions every two weeks as to which bit to add next, eventually sticking enough random bits of steel on to get a bicycle safely over it, before declaring it "finished".
It is often both. The intrinsic problems aren’t always sexy. We create other problems and focus on those so we can stay engaged.
So we have parts of the system that try to abstract away core business concerns and some big in-house framework that is poorly written and that you can’t put on your resume.
