The insidious thing about over engineering is that it's usually committed by very experienced engineers. Experienced engineers rarely under-engineer, that tends to be fixed very early in one's career.
As we get more competent and read more books, we have the tendency to get enamored by new fancy abstractions. We get too clever and then we get in our own way.
Best real-world example: I inherited a project that was an giant "microservice" with 50-ish endpoints, and a Mongo database and dozens of collections. After probably 3 months of wrestling with this thing I had a realization: "this whole thing can be just a simple command line tool and one collection in Mongo". That reduced the code by almost half and it became so much easier to work with. It's frustrating that it could have just started this way.
I used to work for a company which the founder (and original engineer) was both very curious about how compilers worked and paranoid about having the company's code base stolen. That came to fruition with the following idea: "if I develop my own language and compiler, even if someone steals part of the code they still won't be able to run without the language specification and compiler!".
He designed his own programming language, compiler and, why not, the database and tools as well. It was a mess. It got all the limitations of a poorly projected side project and no benefit in the long run besides from huge technical debt.
15 years later, with dozens of clients' sites with the software installed and running locally, the company was locked in a horrible software stack that was next to impossible to move away from by gradually replacing modules, because the programming language had zero interoperability, the original creator was retired and no one besides himself had worked on it for years; while continue giving support to existing customers, because there were fires everyday, and with the same engineering team headcount.
That is one of the top 5 "oh my" in my career. I'm glad I left that behind.
Clojure's Rich Hickey? No. I'm not too familiar with him or Clojure, but it seems he put a lot more thought on it than the co-founder in my story could ever dream of.
OP suggests there's a hump in the curve of experience vs over-engineering; more experience correlates to over-engineering until it gets to a certain point, at which even more experience leads to less over-engineering.
Is it true? I don't know, I think it matches my... experience.
Part of it is that the "experience" needed to avoid over-engineering is helped if it's not just engineering experience, but domain experience too. If someone is constantly jumping industries/domains, it might take longer to get there. I think they still would eventually.
Often I'll tend to over-engineer the initial solution, but as I think about all the complex edge cases that needs to be handled etc I often find myself asking "does the customer really need all this".
After discussing it with the customer, a simpler solution often emerges, where they change the requirements slightly allowing for a much simpler solution that might even solve the actual needs better.
While it's not just down to experience, I'd say it has a strong influence on being able to see beyond the given requirements towards a better outcome.
My own personal learning curve matches that graph almost perfectly. I'm probably not as close to the right side as I'd like to think I am, but I'm always trying to move in that direction.
Over the last years I've been working with an audio library that makes several simplifying assumptions, like
- Audio interfaces are never added or removed over the app's lifetime
- Audio is never rerouted between interfaces, or at least the app doesn't need to know about it
- The details of an audio interface, like number of channels or supported sample rates, do not change while the interface is running
- Audio latency never changes and callback timing is always precise, so timestamps are not really needed and a single number is enough
Of course, none of these are really true and adding support for some of them would require rewriting a few APIs and many, many implementations. On the overengineering side, of course it has its own smart pointers, string class, thread implementation etc just in case someone needs to build for a target that doesn't support C++11.
In the world of hardware, which I realize is a parallel universe to the one in this thread where "product" apparently means "app", the answers are "all the time" and "no", respectively.
If someone didn't take the time to do actual "engineering" - which is to say, using mathematics to formalize design requirements (again, possibly a foreign concept to the app-building class) and just YOLOed the design based on intuition, you can end up with something so fundamentally broken in concept that "fixing" it requires a bottom-up redesign starting from fundamentals. A child's drawing of an airplane is not progress towards a blueprint of one!
quite a bit. under engineered as in 'never really tested, is known in general not to work'. usually, yes in theory it should be easy to fix - but since the parent organization doesn't value getting a handle on what's really going on, it never does.
As we get more competent and read more books, we have the tendency to get enamored by new fancy abstractions. We get too clever and then we get in our own way.
Best real-world example: I inherited a project that was an giant "microservice" with 50-ish endpoints, and a Mongo database and dozens of collections. After probably 3 months of wrestling with this thing I had a realization: "this whole thing can be just a simple command line tool and one collection in Mongo". That reduced the code by almost half and it became so much easier to work with. It's frustrating that it could have just started this way.