Yup, but it goes both ways. I end up over-eningeering as some feeble attempt to avoid technical debt, only to realise my over-eningeering /is/ the technical debt. Or I end up with relatively simplistic / specialised code that needs an entire rewrite and migration process any time something is added. Either way, it's a bunch of rewritten code mixed with paralysing anxiety about writing bad code.
Get used to, and even good at rewriting. I enjoy it, and I realize that I may never have the best idea of what's appropriate at one point in time, but rather accumulate an approximation over time as I reshape the code to the best of my knowledge. And I develop tools and practices to aid me in refactoring faster and with more confidence. Most importantly, keep throwing yourself into it.
I think we as developers underestimate the value of a 100 line Python script. Everything could be hard coded and inflexible, but it's still easy to refactor because you can keep the whole thing in your head.
My tendency now is to aim for over-simplicity in hobby projects. I already have to deal with over-engineered garbage at work I have no choice but to accept, so I don't want to bring those headaches home.
I also did a lot of hobby projects and open source work. Still, in my early career, I was over-engineering everything including my own hobby projects. I think I was trying to put my signature on the work and I unwittingly achieved this through unnecessary complexity.
I was thinking of good code as something I had to invent, but now I feel like it's more like something I have to discover.
Lock-in factor and billable hours can be motivations for contractors.
But, for contemporary software developers in general, I'd guess more often it's either resume-driven development (e.g., add the big complex new framework keyword to your resume) or not yet having enough experience to know what complexity is worthwhile for a situation.
In my experience most over-engineering can be explained by a lack of understanding. When we design it, we don't know what matters and our guesses are wrong; and when we modify it later, we don't have time to figure out how it really works. Both of those problems can be fixed by simply spending more time to understand, but time is money.
Not a lack of understanding, a misalignment of values.
I had a team once implement the MediatR pattern for a < 5k LoC codebase (I'm guessing at its size but it was made larger due to the overhead of the MediatR pattern).
When I asked them to remove it, it became a political fight that went to the VP because they were convinced that sort of flexibility was a good idea. Fast forward a year and they have a new technical leader and he thought I wanted that complexity until we had a conversation and when I mentioned I didn't like it he confessed to me he wanted to rip it all out.
not being absolutely perfect suggests a lack of understanding, but at that point we're engaged in a tautology.
Values are those things that help guide you in the face of imperfect information. Not having a crystal ball that can predict the future perfectly means you have imperfect information.
Perfection is not necessary for you to be confident that you have a reasonable understanding and can build something to match. It is possible the confidence will prove to be misguided, but you can deal with that later.
To prioritize flexibility means that one lacks the understanding required to even build a misguided confidence.
That does not imply there is a lack of understanding in the moment. Understanding does not seek perfection towards future events. However, if you do not even understand what is known in the present, that is when you will start to lean on flexibility.
I've often say "design for deletion": Most of those long-term "flexibility someday" needs are best-met by making sure the inflexible modules or flows can be clearly identified and ripped out for replacement.
This leads to a certain kind of decoupling, although with a higher tolerance for coupling that can kept in check by static analysis.
I think, related, is the hope that switching to a new framework will somehow solve everything annoying about the old stack, and let you undo past mistakes. In reality, though, if you're lucky enough not to end up supporting two stacks at once for a long time, you end up making lots of new mistakes again.
I never though so either, but then I worked at a place that had a stale product, product teams powerless and developers rejecting most features/writing random code all the time.
It took me a month to realise that the staff developers very much revelled in and protected their bad code and bizarre domain choices.
It was so far gone that there was no way to get rid of them and the product was just slowly dying and burning the remaining cash.
Then the mergers happened and they all got let go, only retaining the name for brand power and the entire stack was quietly moved over to another similar product which was rebranded.
Separately, there were absolutely very large consultancies that had a programming style/rules based on making their implementations difficult to read/modify, needing to call their COE back in to fix their code or add features - with it being very hard to modify. Talking entire codebase structured with ridiculous levels of abstraction and annoying code style. Bad integrations requiring their tooling to work and make sense of etc.
They target traditional orgs where the management just wants to get a project through and then bleed them over years.
>It took me a month to realize that the staff developers very much reveled in and protected their bad code and bizarre domain choices.
I think personality can account for this without any reference to incentives, which come in to explain how this personality problem can be so common among successful engineers.
Today's best post on complexity to me was the I accidentally saved my company half a million dollars, which is a story filled yes with a lot of poor developers but much much worse is a story filled with Conway's Law style lessons. Of madcap organizations & wild legacy systems that we live atop & typically just have to make do with. https://news.ycombinator.com/item?id=38069710https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-...
So much of the grug-brained argumentation is same-side violence, is developers roasting developers. Assigning bad motives & declaring immoral bad people among us probably takes already same-side disdain & amplifies it, forments real & powerful hatred. Yes, various petty or criminal over engineering happens some, yes. But I usually think there are simpler Hanlon's Razor explanations that aren't even the individual's fault, are just the story of an org: an organizational body where so many constituent members of the body-whole have so little idea what others are doing or have done, and access to so few who can offer informed situational-appropriate wisdom for whatever dark end a squad finds itself tasked with.
In some ways, this has been the actual story of open source. We have not been end user facing. We have grow communities of practitioners with individual knowledge & experience that ports & can be compared & discussed with others, from outside our company. We get peership that companies can rarely afford or find after they grow mid-sized.
The Influx folks just talked about their 3.0, & replacing their custom built engine with Flight, DataFusion, Arrow, and Parquet. Maybe they might miss one box or two on neat nitty gritty optimizations (maybe this guy elsewhere in this thread, insisting on diy'ing binary encodings because protobuf doesn't have 16-bit ints can find some deficiencies, https://news.ycombinator.com/item?id=38078133), but the nice to haves of ecosystem integration seems like an intangible, but one that lets little good things out there plug in & help out, and that seems invaluable.
https://news.ycombinator.com/item?id=38013714https://www.influxdata.com/blog/flight-datafusion-arrow-parq...
The top comment to me epitomizes the worst of grug brainedness. It's smug & certain of itself, a strong & divisive Hanlon's violation of high degree. That kind of blue-on-blue attitude to me is far worse than 99% of the people who end up over engineering. Few organizations have real defense, lack coherency, to help themselves away from complexity, or they have such tight constraints that they overprune & disaffect those who do care and have good ideas. These are organizational issues, and having an org that can somehow see itself & what's afoot is the huge challenge. Stop hating on devs, please.
I don't think this makes sense. For example currently I have a big inclination to think bad about developers working for Vercel/recommending NextJs because they are invariably the same ppl and do it because of monetary benefits. The intentional over engineering they are adding to make a profit is insane and evil. There is no way I can make this point without "hating" the people doing it.
In my experience these are rarely, if ever, the reasons for over engineering.