Complex solutions are easy. Simple is hard. Often simplicity takes time, iteration and understanding that comes from actually operating a system. I think the missing link here is the Kaizen-oriented refinement that turns complex into simple over time. I find that modern OO-languages frustrate this process by needing cross-cutting changes to refactor for incremental improvements. Expression-oriented languages (like Clojure) are much more fluid, enabling the incremental refactoring required to transform the initially complex and awkward system into the simple and refined scalable system. Unfortunately, just like other languages, it is possible to write difficult-to-change systems in Clojure. And that seems to be often the way it is done.
Specifically for clojure, do you have any recommendations about how to make systems easy to refactor? I've often read discussions about how dynamic languages are quicker to write and more difficult to refactor than static ones, and I've had my fair share of dealing with issues during refactoring of clojure code, but I don't remember having read anything particular around clojure best practices. Googling for a bit does not produce particularly insightful results, so if you have any insights that'd be awesome.
EDIT: Part of the things I'm finding are my own comments in HN from some years back, in fact :(
Yes, I like to call the style Procedural Composition. The key characteristics are no logic with IO, and all system functions are assembled from higher-order middleware functions. This is the way ring, the web framework, and various client libraries like clj-http work. One of the best ways to get familiar with it is to look at the ring libraries.
What this allows one to do is assemble handlers from lots of small higher order functions. It has the downside of one needing to have initialization well organized and thought out. But the fractal-cul-sac as one co-worker has nicknamed it of ever dividing and increasingly specialized functions with IO and logic twisted together is effectively prevented. One can always modify one handler without affecting others or changing how existing functionality works. And it is always testable in small unit tests because the logic is pure.
Systems built in this style can radically reform themselves in a controllable, reliable and consistent way indefinitely. They are immune to the "apogee" phenomena whereby a system gets to the size and fragility where it can no longer be modified without causing unforeseen regressions, triggering the need to rewrite it to move it forward.
This style is for handler-oriented procedural software common in web services etc. It does not apply to embedded state-machine oriented software that operates real-time control etc. That is a different problem space.
The reason it's hard to refactor to simplicity is that one must feel free to modify foundational assumptions, and the repercussions of that change are both hard to implement (you must recapitulate all following changes with the new assumption) and test. By "foundational assumption" I mean architecture at the high level, and general code organization at the lower level. It is the "second system" problem, writ small but many times over.
Given that code grows over time in accordance with selection pressure, these lateral, foundational changes are always going to be difficult because recapitulating the following growth in general takes the same amount of time as the original system took to grow. The irony is that if that system is making money, it is even LESS likely that the will to simplify it will arise; if the system doesn't make money, then there will be neither will nor resources to try it.
"Physics progresses one funeral at a time", Planck's Principle [1], applies here. Software gets simpler but not laterally within a working project; it only gets simpler with new projects. Luckily software devs seem more open-minded than physicists so we can progress faster than funerals, but not arbitrarily so.
Changing and refining Clojure code is a remarkably pleasant experience. There's very little state, so you can usually just grab what sexp you need and extract it.
I saw a 2x2 matrix recently that showed how NP-hard problems require exponential time to solve but polynomial time to verify the solution. It gave examples for every other cell of the matrix besides (exponential time to solve, exponential time to verify), which it left as blank or N/A.
I think we have found something that fits that square: truly simple solutions. It's easy to come up with a simple solution that is unsatisfactory, that's not what I'm talking about.
I mean a solution that is so simple and complete, that it was HARD to come up with. It's easy to verify that such a solution is simple, sure, but it is HARD to verify that the solution is truly complete or satisfactory
> It gave examples for every other cell of the matrix besides (exponential time to solve, exponential time to verify), which it left as blank or N/A.
I would be interested in seeing this matrix, because it sounds incomplete to me.
Any NP-complete problem, as in a problem that is exponential to solve but polynomial to verify, is a "decision problem" version of a "optimization problem" that is exponential to both solve and verify.
For example, the decision-problem version of the traveling salesman is, "given a graph of nodes and weighted edges, does there exist a route that visits all nodes with a total edge cost of less than N?" Solutions are hard to compute (evaluate many paths to find a good solution) but easy to verify (evaluate the given path exactly once and compare the cost to N).
Meanwhile, the optimization-problem version of the traveling salesman is "given a graph of nodes and weighted edges, what is the route that visits all edges with the least cost?" The solution is now both hard to compute and hard to verify, because to verify it you also need to consider the entire decision space to determine whether or not there's a route with a lesser cost.
