Using a lightweight, comprehensible framework is good, until you hit the limits of that framework and start pulling in lots more libraries to fill the gaps (the Sinatra/Cuba world). Using a heavyweight, complete library is good, until you start suffering from bugs caused by incomprehensible magic buried deep inside (the Rails world).
I see the same problem in microservices versus monolithic apps. If you use microservices, you have a lot of duplication and inter-component configuration management complexity. If you use monolithic apps, you get the big ball of mud.
Or, as I sometimes put it, "Which kneecap do you want to get shot in?"
The underlying problem isn't fashion, or bloat. It's that software is very, very complex. Unless you're doing hardcore embedded work in assembly language, you're building a raft on an ocean of code you won't read and can't understand.
A friend of mine put it well once. He said that you should have deep understanding of systems one layer from yours (ie your frameworks), and at least a shallow understanding of things two or three layers away (operating systems, etc). If you don't have some grasp of the things you depend on, you're relying on magic and are vulnerable. But you can't realistically expect to know it all.
"I don't believe in the no-win scenario." -Captain Kirk
A lot of the problems we face in software engineering are Kobiyashi Maru tests. They're no-win scenarios, tests of character rather than ingenuity. There's a certain irreduceable complexity to all interesting problems, so at some point, you're not solving complexity, but merely pushing it from one place to another.
There's a certain irreduceable complexity to all interesting problems, so at some point, you're not solving complexity, but merely pushing it from one place to another.
Right, but I'm not about to believe we're even close to that limit. Every day we have people writing bugs which, from a state of the art perspective, are already solved problems. It's like people are out there riding horses while others are driving past in their cars.
I think it's a little better to say that a lot of the problems we face in software engineering are cultural, not technical. So many people adopt these tribalistic mindsets when discussing their preferred technologies rather than allowing themselves to be open to better ideas.
> It's like people are out there riding horses while others are driving past in their cars.
This is a good analogy but in software development, as in life, there are roads and there are trails. Some developers try and drive their car on the trail and their horse on the road! But sometimes you need a horse and face already solved problems because your environment requires it (e.g. using C for embedded development).
> So many people adopt these tribalistic mindsets when discussing their preferred technologies rather than allowing themselves to be open to better ideas.
The problem is there is always a better idea. You have to strike a balance between being open to new ideas and just getting the job done. But there is definitely tribalism because that is human nature; You earn your way into a technology and community through pain and suffering so you become attached to it. Maybe some people can avoid that but I believe it's pretty hard. Especially when the differences between technologies are mostly cultural. You're less likely to be tribal when considering C vs. Ruby but perhaps more so when faced with Ruby vs. Python.
Sure, there are lots of projects that are cranking out junk redundant code on legacy systems. But even if you do use the best tools, the best processes, the best analysis, the irreduceable complexity remains.
For example, I have to read data from one source, and write it to another. I can have a single app provide api for both the writing source and the reading source, or I can have two separate apps and two separate apis. But I can't get away from the problem of reading and writing. That's irreduceable. If I have separate apps for the two apis, I have a configuration management issue. If I have a monolithic app that does both, I have a coupling issue.
Imagining that Bleeding Edge Technology of the Month will make this go away is wishful thinking. But facing the truth is awful, so we choose the wishful thinking.
I'm not merely talking about junk redundant code but entire classes of problems such as memory errors (null pointers etc.) and race conditions (in many, though not all, cases).
If you compare software engineering to another field like aerospace or structural engineering it's laughable how little rigour and formality takes place in most software. It's shocking how much disdain programmers show for mathematics which gives us the tools to prove that our applications are correct.
It's shocking how much disdain programmers show for mathematics which gives us the tools to prove that our applications are correct.
I personally think these things are great but too expensive. My customers care about solving their problems not perfection. There's a balance to be struck and I'm not excusing bad software, but if the customer isn't willing to pay for better and is happy with the current quality, there is little incentive to put in this extra effort (and therefore cost) when the resources could instead be put to work on providing value in ways the customer does care about.
I guess after you exceed the customers quality requirements/expectations, there are diminishing returns.
I'd love to produce perfect software, but nobody will pay me to do it.
I personally think these things are great but too expensive. My customers care about solving their problems not perfection. There's a balance to be struck and I'm not excusing bad software, but if the customer isn't willing to pay for better and is happy with the current quality, there is little incentive to put in this extra effort (and therefore cost) when the resources could instead be put to work on providing value in ways the customer does care about.
I didn't say you had to do all of this yourself. I don't believe every programmer has to have a degree in mathematics. Where I take issue is with the refusal to take advantage of the hard work of others. There exist languages, tools and libraries which have a strong mathematical basis that are ignored in favour of the latest fad. Places like stackoverflow are full to the brim with people asking for help solving problems that they wouldn't have to deal with if they'd chosen better tools.
Ah, you're thinking down at the lines-of-code level. I'm thinking at a much higher, big-project requirements level. It's hard to say which one is worse.
I do, however, think there's a "which kneecap" problem to rigor as well. Rigor isn't free - it comes with costs, in learning curve, in the quality of developer required, etc.
I've been bouncing back and forth between not-rigorous Ruby, kinda-rigorous Go, and pseudo-rigorous Java. I don't think rigor is a killer solution, nor do I think it's a gruesome waste of time. But I find the really ugly problems to be at the requirements and architecture level, not the code level. Not surprising, considering coding isn't my primary work.
After reading that I feel about as lost as I did after that XKCD about correlation and causation https://xkcd.com/552/
Also, just because your comment is popular does not mean your comment is good.
The popularity heuristic may be a little better than the OP suggests though, maybe.
Using a lightweight, comprehensible framework is good, until you hit the limits of that framework and start pulling in lots more libraries to fill the gaps (the Sinatra/Cuba world). Using a heavyweight, complete library is good, until you start suffering from bugs caused by incomprehensible magic buried deep inside (the Rails world).
I see the same problem in microservices versus monolithic apps. If you use microservices, you have a lot of duplication and inter-component configuration management complexity. If you use monolithic apps, you get the big ball of mud.
Or, as I sometimes put it, "Which kneecap do you want to get shot in?"
The underlying problem isn't fashion, or bloat. It's that software is very, very complex. Unless you're doing hardcore embedded work in assembly language, you're building a raft on an ocean of code you won't read and can't understand.
A friend of mine put it well once. He said that you should have deep understanding of systems one layer from yours (ie your frameworks), and at least a shallow understanding of things two or three layers away (operating systems, etc). If you don't have some grasp of the things you depend on, you're relying on magic and are vulnerable. But you can't realistically expect to know it all.