IMO it's not that the simplest solutions are the best but that the "better" complex solutions are not actually available upfront. They can only be made with hard-won domain knowledge. The design policy for lld v1 per this article encoded many assumptions that turned out to be untrue in practice (like the importance of platform independence). If they had been true then the extra complexity might have been worth it. Over time the simpler lld v2 might accrue its own complexity that better reflects learned experience.
Code is a tool for exploring and understanding problems as much as it is about solving them. Sophisticated solutions can't be designed before they are validated.
The software engineering field is terrible at handing down domain knowledge.
Part of the problem is that it's next to impossible to separate the facts from the religion in software domain knowledge. Software artifacts are simultaneously art, technology. math and religion.
It's also hard to separate the abstract knowledge that is broadly applicable from its original context where it was hard won: the particular operating systems, toolchains and tech stacks.
Even that knowledge which is separable is mired in the jargon of those systems, so that it looks like it is specific to them. If those systems are long obsolete, then it's easy to dismiss anything that is robed in their jargon as being obsolescent by association.
If civil or electronic engineering were like software, then every five years, someone would be reinventing a class B push-pull emitter follower amplifier output stage, or Pratt truss, under different names.
My thing is I go simple. I'm good, maybe messy. Finally see enough / think I know a good abstraction and pull the trigger and later I find out ... oh man I was wrong. Knowing when you know is the hard part.
I've got some abstractions out there I wrote early on when I didn't think I coded well and they have lived for years and saved tons of time.
So many times I've taken code that was a mess because someone tried blindly DRY code for the sake of being DRY and rub abstractions on top of it and I've reverted the code back to being simple copypasta and its become so much more clearer and robust. Then you can look at the result and a simple abstraction may pop out which can reduce enough code duplication that the result is satisfactory (it may require a bit more boilerplate in the subclasses or whatever, but nothing likely to be brittle under future fixes).
The nice thing about dealing with an overly-DRY codebase is you can unwind the abstractions.
But an underly-DRY codebase... good luck. Copypasta tends to accumulate minor differences and figuring out if those minor differences matters can become very time consuming.
All the libraries I love and use the most have tons of parameters but they are really well documented including their parameters' interactions with each other.
I don't mean well-considered and documented libraries, I mean the thing where someone goes to DRY up a codebase and keeps thinking "oh, if I just add a parameter this function can also handle this one-off case over here..." over, and over, and over, until you've got this function with a name that makes it look straightforward but for some reason it wants three strings, two of which should actually be nulled in most cases (and god help you if you fill in a value when you don't need them), an integer for unclear reasons but it's the same value in every call you can find, and half a dozen booleans that you'll have to go read the source to figure out, because some of them do more than the variable name in the signature implies.
Good intentions at every step, but a Frankenstein's monster of a function (or entire class, sometimes) in the end
It's the DRY equivalent of having 20 slightly-different copypastes all over a codebase.
If it's unclear, then it's poor code or poor documentation or both. Sane developers will provide a way to obtain default or common configurations rather than forcing you to figure out the 100 different tunable options.
I would say, theory as in "music theory", not as "theory of general relativity".
A person who possesses music theory can explain why they are composing something they way they are, and suggest better notes to someone who is trying to embellish the music with a harmonization or whatever. Just like the people who held the "theory" of the compiler were able to spot better ways of implementing the ideas of the second group without spoiling the design.
Over time, I have come to believe that the problem is overly-aggressive abstraction. It is very tempting for most developers, especially good developers, to reach for abstraction as soon as things get complicated. And this can pay off very, very well in some cases. However, too much abstraction in a system leads to a very, well, abstract code base that becomes hard to get a handle on: there's no there there. You see this in the java world with AbstractFactoryBuilderLookupServiceBuilders and so forth, and with very elaborate type setups in functional programming languages.
Concretizing the crucial bits of your system, even if that means a few large, complex and gronky methods or classes, ends up often making things more understandable and maintainable and, certainly, debuggable.
John Ousterhout wrote a book that makes this point as well, advocating for "deep" rather than "shallow" classes and methods:
> Over time, I have come to believe that the problem is overly-aggressive abstraction.
Sometimes, yes, overly-aggressive abstractions are a problem. But the author describes policies of v1 and v2 of the linker. And I would say the most critical difference between them is, that the authors of v2 had a better understanding of the requirements that actually mattered. Therefore they were in a better situation to evaluate the architecture and they were able to make better trade-offs.
Deciding to trust object files as inputs might raise a red flag for some people. In principle it could allow attacks/exploits of the linker. But in reality for most threat models this does not matter. Because the compiler generating object files has the same level of trust as the linker. Companies that want to provide an elf linker as a service product are out of scope. Deciding what is in scope and what is out of scope is probably one of the hardest decisions in product engineering. Because many of us software engineers lean towards perfectionism or, especially inexperienced developers are searching for the silver bullet, that set of rules that enables them to develop every product successfully.
I suspect the "Worse is Better" essay has caused untold harm. It is written from the standpoint of a purist offended that the world doesn't appreciate purity, complaining that the things people end up using are built by pragmatic people. The lesson seems to be that doing things better is punished. But that is the wrong lesson.
The correct lesson is that the real world is not obliged to conform to your personal model of "better". You have a personal obligation to continuously adapt your model to match the real world. This is the model of science, and is opposed to Platonism. After you have adjusted your model, it is certain to still not be right, and need further adjustment.
Usually "the world" you are obliged to adjust to has its own problems. There are powerful forces making us favor accommodating a Microsoft execution environment, even though that execution environment has always been a cesspit. It represents its own poor abstraction. Posix file system semantics are another example. Von Neumann architecture and the C abstract machine are not the only, or best way to organize computational resources. It is important to recognize when somebody else's pragmatic failure threatens to taint your own models.
Lisp, RG's hobbyhorse, didn't get sidelined because of Philistines. Lisp turned out not to be better, despite how strongly RG felt about it. Instead of figuring out what about Lisp was not right, he called things that were, along axes that matter, more right "worse", preserving his personal model and lessening readers' ability to reason about merit.
I don't agree at all. I think he was able to give a clear and accurate analysis of why lisp failed, despite liking the language so much. He also wrote critical responses to his own essay under a pseudonym, with a back and forth that is quite funny and demonstrates the ability to understand both sides of the argument. In "Worse is better" he explicitly mentions how that approach favors real-world application, because it is so simple it is fast and is "good enough" and then can be moved to 90% of the right thing.
All of this is to say: I don't agree with you, but I also agree with you and I suspect he would as well, with qualifications. And he would probably also disagree with you.
Not buying it. He still says "worse" and still writes "90%". What he means is "not conforming to my personal value system". He is dodging the crucial step of discovering how his personal value system got so off-kilter as to lead him to wrong choices, and fixing it.
Lisp machines were clearly abandoned because of their price. Every time I find some history about someone that actually made that decision, the reasoning was exactly alike, those machines costed more to keep than the Unix ones to install, and were less capable due to outdated hardware.
Yet the essay goes all over the place, citing time to market (that was completely irrelevant, UNIX was the newcomer, Lisp machines were there already), university-based prejudice (yet every single one decided the same at around the same time), and blaming the user. The essay doesn't even talk about money.
It would probably help you to understand the essay if you knew that it was written by Richard P. Gabriel, the head of Lucid, the leading competitor to Lisp machines. His company's product was a Lisp system that ran on Unix machines. What he's personally best known for (aside from this essay) is showing that smarter Lisp compilers on commodity hardware like a 68020 or a SPARC could deliver performance that first equaled, then exceeded the performance available from custom silicon; the book he published on this subject was so rigorous that many of the tests in it are still used today for judging the performance of implementations of high-level languages such as LuaJIT and V8.
You seem to think he was advocating Lisp machines, scoffing at his essay based on that misconception. But if there was a single person in history who worked hardest to destroy Lisp machines, it was probably RPG.
Yeah, that's more like it. RPG worked a lot to bring sophisticated Lisp to UNIX systems. He worked on bringing a standard OOP system to Lisp (incl. Lisp Machines, where Lucid (and he personally) collaborated with Xerox PARC and Symbolics to develop CLOS as a part of the ANSI Common Lisp standard).
Then he tried to develop and market similar technology in the form of a databased-backed incremental C++ development system for UNIX (called Energize). https://www.youtube.com/watch?v=pQQTScuApWk
He hoped that the market for C++ environments was larger than the shrinking Lisp market and that C++ developers would want an interactive system with integrated code management in an object store. That did not went well.
Reports were that Energize customers were relieved when Lucid folded, because they would then not get a whole new set of compiler bugs to discover every quarter. They had, instead, the bugs they already knew about and had learned to work around.
Lisp Machines anyway raised the standard of quality in CRT monitors. Manufacturing their own monitors has to have contributed substantially to their downfall, but we all benefited in the end. Well, all but them.
There is an unfortunate fallacy here. Your notion "the real world" conflates the physical world with a social milieu. The physical world is (at this scale) immutable, so of course we must conform to it and update our science-like models.
But there is no absolute requirement to conform oneself to a social milieu. A social milieu changes. It can be altered. It supports a vast number of models. And milieus overlap so densely that one can just go play somewhere else.
Disclosure: I used to be a Lisp bigot, but I got better.
Agreed it is complicated. We have the principles of physics and computational irreducibility, then physical realization of state-machine designs to exploit those principles, then social conventions around what is valued, languages influenced by conventions, operating environments for programs in them, and finally actual code in those languages.
What makes a "better" Lisp program differs from what makes a "better" C++ or Rust program. Besides fitness for purpose, there is maintainability, energy cost to execute, and results per unit time. What did you learn coding it? Can it be the basis for something more ambitious? Is it secure, deadlock-proof? Are its results accurate, aesthetically pleasing, generative of insight?
We can get mired in detail, obscuring important truths. We talk about performance a lot, not because we are obsessive, but because it is a measure that is hard to fake. A faster program says something fundamental about how your computational resources are being directed to the target results.
We can be misled by details of realizations of computation. What is fast on a PDP-11 is not necessarily fast on a Ryzen 5. But submitting to the rigors of performance for the best machines we have is a discipline that connects us, howsoever imperfectly, to the physical principles of computation. It destroys illusion: if a variation seems like it ought to be faster, but is actually slower, there is no sugar-coating the fact. Hewing to physical performance enforces a kind of honesty we don't get any other way.
> The correct lesson is that the real world is not obliged to conform to your personal model of "better".
(Annoying as fuck, that. ;-)
And that seems to indicate that a(n initially) half-assed, but improvable, model of it that is then iterated upon is a good way to build one's model. Feels to me like that is exactly what "Worse is Better" is about -- shouldn't your beef here rather be with "the MIT model"?
> You have a personal obligation to continuously adapt your model to match the real world.
A bit hard to get a consensus model out of that, since everyone's perception of the real world -- heck, everyone's actual "real world" -- varies.
> This is the model of science, and is opposed to Platonism.
Again, what feels like setting up a Platonic ideal to me is more "the MIT model", rather than "Worse is Better".
> After you have adjusted your model, it is certain to still not be right, and need further adjustment.
Ah, dangit, at some point you just gotta say "Screw it, good enough!" (See, for instance, "The saddest 'Just ship it!' story" the other day.)
This will sound crazy from our perspective this year, but from the perspective of the essay 31 years ago, Lisp dominates software development today; we just spell it without parentheses.
The top ten languages in https://www.tiobe.com/tiobe-index/ right now are Python, C, Java, C++, C#, Visual Basic, JS, assembly, SQL, and PHP. Of these, the "dialects of Lisp" include Python, Java, C#, VB, JS, and PHP.
Remember that in 01991 all "serious" modern software was either C, C++, Pascal, or assembly. BASIC, whose only data structures were the string and the array, was for amateurs or the MIS department, which mostly used COBOL, assembly, and JCL. Fortran was established but was considered antiquated (except by supercomputer people) and didn't have pointers.
If we compare these languages on the points pg lists in http://www.paulgraham.com/diff.html, the earliest versions of Lisp score 9/9, Python is 7/9, Java is 7/9, C# and VB are Java with different syntax, JS is 7/9, and PHP is 5/9. By contrast, C is 2/9, C++ is 3/9, Pascal is 3/9, assembly is 0/9, COBOL is 0/9, Fortran 77 is 1/9. You can quibble a bit about these numbers (do Pascal procedure parameters qualify as "a function type" even though you can't store them in variables? Does the Python expression "intern(s) is foo" qualify as "a symbol type"?) but the broad division is very clear.
I think it was in the conversation where he originally wrote that essay that Guy Steele said of his own work on Java that they had managed to drag all the C++ programmers kicking and screaming about halfway to Common Lisp, so we should be happy. I've lost my archives from that time, so I can't be sure.
In terms of syntax, Python or Java have nothing in common with Lisp. But in terms of the issues you raise — accommodating a Microsoft execution environment, POSIX filesystem semantics, the Von Neumann architecture, the C abstract machine, or just the tools they give you to analyze problems — they're just dialects of Lisp with slightly different syntax (and more hair on eval, and sort of broken symbols or no symbols).
In terms of "worse is better" of course Python and C# lean just as hard on "worse" as C does.
If we restrict the sense of "Lisp" to languages with S-expression syntax like Emacs Lisp, Common Lisp, Scheme, and Arc, then Lisp did fail to (at least) become popular — but it's plain to see that when people abandoned Common Lisp and Scheme, they were mostly moving to languages like Python and JS which adopted Lisp's most appealing ideas, not to C++.
> Your notion "the real world" conflates the physical world with a social milieu. The physical world is (at this scale) immutable, so of course we must conform to it…
> But there is no absolute requirement to conform oneself to a social milieu. A social milieu changes. It can be altered. It supports a vast number of models. And milieus overlap so densely that one can just go play somewhere else.
In that vein, it's worth noting that Linux got pretty far before it ever had to accommodate a Microsoft execution environment, though I did have coworkers in 01997 who thought I was hopelessly unhip for preferring Unix (which they thought of as antiquated) to Microsoft Windows.
> This will sound crazy from our perspective this year, but from the perspective of the essay 31 years ago, Lisp dominates software development today; we just spell it without parentheses.
It is true that some good ideas originate from Lisp. However, by saying that 'Lisp dominates software development today', you are giving the false impression that the Lisp heritage is the only one that matters, as if Lisp were some sort of linguistic asymptote toward which all other programming languages must inevitably converge. I take issue with that. Creating a new programming language involves copying the good ideas from previous languages and rejecting the bad ones. The notion that this or that modern programming language is a 'dialect of Lisp with slightly different syntax' completely ignores all the bad ideas from Lisp that were not copied and all the good ideas from other languages that were.
Yes, Lisp itself abandoned the idea of car, cdr, and cons as the foundation of all data structures about 50 years ago. On the other hand, many languages further from Lisp than Python or JS did adopt the idea of car, cdr, and cons as the foundation of some data structures, notably Prolog, OCaml, Standard ML, F#, and Haskell.
Well, I suppose that's true, although mainstream languages that are uncontroversially Lisps, such as Emacs Lisp, Scheme, and Common Lisp, have also abandoned a lot of Lisp's bad ideas. And I do agree that Lisp isn't something toward which all other programming languages inevitably converge—neither any particular historical Lisp, nor some Platonic ideal of Lisp.
But I do think it's a sort of "linguistic extreme" that defines a dimension in programming-language space, and new popular languages tend to be pretty far in the Lisp direction. In TIOBE's top ten languages, the four I identified as "not Lisps" are pretty old: C is from, say, 01973; C++ is from 01982; SQL is from 01974 (and wasn't Turing-complete until a recent unfortunate accident, therefore not a programming language); and assembly language is from somewhere between 01948 and 01968. The other six popular languages I listed are all from 01989 or later; C# and arguably VB are from 02000.
There are several other linguistic extremes that things could have moved towards instead. For example, FORTH, Self (the essence of Smalltalk), miniKANREN (or maybe Mozart/Oz/Prolog), Coq (or at least some kind of cleaned-up ML), the π-calculus, m6, Inform 7, some kind of lazy language (maybe Haskell), and now something interesting is emerging in Rust. You could imagine a history in which all the new languages adopted key controversial ideas from FORTH or ML instead of from Lisp, but that mostly isn't the history we're in.
Java/C#/VB in particular takes pervasive ad-hoc polymorphism from Smalltalk, parametric polymorphism from ML, and type declarations from, I guess, Fortran, though detecting errors at compile time wasn't a motivation for having them in Fortran. And Python takes pervasive ad-hoc polymorphism from Smalltalk and gets its parametric polymorphism for free with the dynamic typing it takes from Lisp. (Python's predecessor ABC was a teaching language derived from ML, which is why Python spells null as None, but Python dropped the static typing.)
Here are some candidates for Lisp's abandoned or not-adopted bad ideas that come to my mind:
1. Dynamic scoping as default. PostScript and TeX do do this, but nothing else does. Even Common Lisp and Emacs Lisp have abandoned it. (And Scheme never had it to begin with.) Nevertheless, it was viewed as a defining attribute of Lisp.
2. The β-reduction model of program execution, in which the output of the program is a transformed version of the program; and, more generally, there is no clear distinction between the program and the data it processes. This was sharply abandoned in Scheme, but without abandoning metaprogramming. Mathematica is the only current language I know of that works this way. (Maybe Q?)
3. Textual serialization (READ, PRINT) as the standard and indeed only serialization. (The original Lisp and many of its successors also have pervasive serializability, which pg inexplicably omitted from his original essay: any data object can be written to a byte stream, and the same or equivalent data object can later be read back from it.) Variants of this are pretty popular; JSON.stringify, Python repr(), Firefox .toSource(), Python pickle, etc., but in most cases modern systems relieve the tensions between the different uses of READ and PRINT by providing several different serializations.
4. The general idea that the important part of software development is getting a prototype working, after which making it robust is comparatively trivial. This idea probably does not originate with Lisp, and it is popular in startup business literature, but not in current programming language design. But people use Python in Jupyter a lot to interactively explore computational ideas.
5. Symbols as the only string type, which is to say, hash-consing all your strings. This is sometimes a worthwhile thing to do in a particular library but not a good tradeoff to impose on all users of the language. Lisps stopped doing this about 50 years ago too.
I'm interested to hear which bad ideas from Lisp you're thinking of!
I guess that's true. But maybe those projects should have been done in C++ (or ML or something) instead of Lisp in the first place, because the things that Lisp is good at are very much not the things C++ is good at. They're the things Python, Java, and JS are good at. As Perlis wrote in the Foreword to SICP:
> Pascal is for building pyramids—imposing, breathtaking, static structures built by armies pushing heavy blocks into place. Lisp is for building organisms—imposing, breathtaking, dynamic structures built by squads fitting fluctuating myriads of simpler organisms into place. The organizing principles used are the same in both cases, except for one extraordinarily important difference: The discretionary exportable functionality entrusted to the individual Lisp programmer is more than an order of magnitude greater than that to be found within Pascal enterprises. ... To illustrate this difference, compare the treatment of material and exercises within this book with that in any first-course text using Pascal.
You can easily substitute "C++" for "Pascal" here and "Python" or "JS" for "Lisp", and indeed a JS translation of SICP has been completed, including a column-by-column comparison: https://sicp.sourceacademy.org/chapters/5.2.3.html
> But maybe those projects should have been done in C++ (or ML or something) instead of Lisp in the first place, because the things that Lisp is good at are very much not the things C++ is good at.
Common Lisp was not developed as a scripting language like Python or JavaScript. It was developed for complex applications (100k to 10M lines of code were not rare - for example the PTC CAD system written in C and Lisp had 7 MLOC Lisp code already over a decade ago) and derived from a language with was actually a systems programming language: Lisp Machine Lisp.
Thus there were a bunch of ambitious projects initially written in Lisp and then continued in C++ (and also in C) for better performance on smaller machines, smaller memory footprints and less need to mix C and Lisp.
I know it was, and today Emacs has over a million lines of Lisp even without considering Elpa, but I think Lisp embodied some ideas about the best ways to structure large programs that turned out to be wrong.
In particular, Lisp in general (including Common Lisp) is designed as far as possible to maximize flexibility, and I think there's an unavoidable tradeoff between flexibility and correctness. Oversimplified, programs are flexible when they can do things their original author did not anticipate; they are correct when they cannot do things their original author did not anticipate.
Of course, in reality, it's not that simple. But this obviously false formulation contains important bits of truth. Testing a flexible program is more difficult, and more bugs are likely to slip through your testing, and each one usually takes more effort to diagnose and fix. And of course in general any two virtues trade off against one another in the limit, because if you take every possible measure to increase the first, some of them will happen to decrease the second. (By the same token there are some measures you can take that will increase both.)
Yeah, or sometimes by abusing the fuck out of templates, though that wasn't a possibility yet in the early 01990s. Emacs uses the same approach (adding an embedded dynamic scripting language), and I think PTC CAD does too, and in these cases the "embedded dynamic scripting language" is a Lisp. (AutoCAD, too, but I don't think they wrote significant parts of AutoCAD in AutoLisp.) It's a good approach that avoids paying the cost of flexibility except where you need it.
Since Lisp is often compiled to machine language, C/C++ is often only glue to the OS and large parts of the software is actually written in Lisp. GNU Emacs now has a variant with native compilation, so it there would possible to get rid with much of the C code without much loss of performance. Thus in a large CAD system, compiled Lisp is not just the scripting language, but can be much of the implementation.
Yeah, as I said above, it's not mostly about the performance¹ but about the bugginess, which is to say, the comprehensibility. There are arguments both ways about whether C or Lisp is more bug-prone (less comprehensible) but I think the results are in. And there are new languages like TypeScript which combine and exceed the advantages of both.
______
¹ Though GCC still routinely generates much better code than SBCL, much less the new Elisp compiler, that's not the primary consideration. I'm not sure why you're mentioning it.
It can be worth understanding why the Java ecosystem evolved that way. It's not because simplicity wasn't valued, but rather it's not code simplicity that's valued. Rather, much of the Java ecosystem is designed to allow dozens of independent teams, totaling hundreds of developers, within a company (or across companies) to build their small piece of the application, with whatever build process is used by them. Then you put all the .jar files in one bundle, and the AbstractFactoryBuilderLookupServiceBuilders puts the pieces together at runtime.
It's complex as hell code-wise, but it simplifies the amount of cross-team/cross-company alignment and synchronization that has to happen in order to cut a new version of the application containing hotfix 8495 for the part of the app maintained by Steve's team.
There's actually a decent parallel between that and Microservices. Microservices make maintenance of the whole more complicated by introducing the network between pieces, but allow each piece more flexibility in how it's developed.
The folks at Lucid were starting to get a little worried because I would bring them review drafts of papers arguing for worse is better, and later I would bring them rebuttals against myself. One fellow was seriously nervous that I might have a mental disease.
after over a decade of thinking and speaking about it . . . [I wrote] "Back to the Future: Is Worse (Still) Better?" In this short paper, I came out against worse is better. But a month or so later, I wrote a second one, called "Back to the Future: Worse (Still) is Better!" which was in favor of it.
This is the heart of engineering or politics: finding the optimal compromise.
The kind of codebase you describe with "elaborate type setups" (multiple levels of inheritance, design patterns like Facade used for "decoupling" etc.) make it very hard to read the code and understand what it's doing (unless it's very well documented), and because of that also make it harder to extend without resorting to kludges. Or, if you want to extend it in a way the original designer didn't foresee, you now have to change 5 different classes to be able to access some private property in some class.
One of the best programming aphorisms I've heard is "Everything in programming can be solved with another layer of abstraction except for the problem of to many layers of abstraction."
Also, I often see developers running to the abstractions they know (cough design patterns...) mostly because they're very afraid of acknowledging they haven't grasped the real complexity of a domain. By the time a pragmatic efficient design emerges, you're on the second or third rewrite...
Worse is better is simply a local maxima of features - schedule - cost tradeoffs, and there not being enough of a sustained investment to "hop out" of the local maxima.
When those maxima are things like operating systems and programming language ecosystems, those are very very big hops to make.
Abstraction always costs. This implies any abstraction you put in that doesn't deliver commensurate benefits makes your code fundamentally worse.
If you have an abstraction with a name that seems to say it does X, but to be sure I have to trace through four other source files plus an unknown number of dead ends just to see if it really does exactly just X, and in the end it could have just been coded in place, it has already cost way more than any benefit it could yield.
Abstraction is an engineering tool, not a moral imperative. You can always add abstraction later.
So, "Worse is Better" is at best misleading. Better is better. But the measure of "better" you have been using is likely to be way off. People who think of themselves as smart tend to undervalue simplicity. It is a personal failing.
It sounds like what you're talking about is indirection, not abstraction.
If you have to verify that "it does X" by following source files and tracing execution then you're talking about indirection.
An abstraction has mathematically sound laws that can be proven and introduce precise, new semantic layers to a program. By definition one doesn't have to think about the layers underneath the abstraction and can instead think entirely in the abstraction.
The difficulty with abstractions is that few practising programmers know how to think even informally about abstractions. The mistake of substituting abstraction for indirection leads to the misguided notion that virtual methods and classes are "abstractions." However if you try to formalize these abstractions with relations and properties I think you will find most of these proofs difficult, if not downright impossible write. That's a good sign you don't have an abstraction.
Right. All that work on type systems is purely indirection. I want to run my code and the compiler keeps telling me I can't access this place in memory because some other part of my code is borrowing it. Or, what do you mean the commutative property of this type class doesn't hold for my type? Purely directing me away from programming, for sure.
Not always. In C++ and Rust, in many cases the abstraction penalty is completely eliminated by inlining or by making an efficient concrete specialization. If this can be achieved the abstraction penalty is paid only in compile time.
That is just the compiler optimizing out the runtime indirection. The effect on you as a programmer is unchanged: you have to look through the same number of source files either way.
The flip side is that without abstraction your code becomes an unmaintainable, fragile disaster, leading to bad outcomes for the user. It’s not as simple in a complex system as you make it out to be.
Abstraction needs to be made to earn its keep. The massively abstracted linker in TFA was, exactly, an unmaintainable, fragile disaster. It was so bad they had to rewrite it from scratch. The new one certainly has abstractions of its own. Less harmful ones.
With way too much abstraction your code also "becomes an unmaintainable, fragile disaster, leading to bad outcomes for the user". Both extremes are terrible. And GP is not advocating using no abstractions.
This ain’t “worse is better”. If the end-user has no worse of a user experience with the supposedly worse-is-better interface or program—most notably in this case if the linker never in practice gives a bad user experience on bad input (perhaps because it never happens)—then it’s not really “worse”.
The central tenet of worse-is-better is to prioritize implementation simplicity over user experience. But if you simplify the implementation since some features are never used and not needed then you haven’t even had to make a choice on that spectrum—you have just cut out unneeded cruft.
In fact the user experience has improved since it is faster...
In fact the V1 implementation was not simpler. So, it was just worse on every axis except its adherence to what people apparently are taught in CS programs.
I have seen "worse is better" trotted out many many times, used mostly as an appeal to authority instead of a clear instantiation of the argument presented in the essay. I think this is because the argument in the original essay is not crisply presented (at least not to me). So authors take "worse" and "better" on some random dimension that's beneficial to them in the moment, and then appeal to the authority of this well respected essay to "prove" that their approach is better.
This post is stating that they tried to generalize too much instead of building to narrow use cases first. There is no need to bring "worse is better" into it at all, IMO
As I understand it, this all goes back to the original writings / theories / practices that "Worse is Better" was a reply to, which encouraged a "too theoretically pure" way of writing software: Everything abstracted away, generalised, (theoretically) infinitely flexible, etc -- at the cost of code complexity that in practice often turns out to be unnecessary for the vast majority of use cases.
In that sense, every more or less radical simplification that sacrifices some of those fine (theoretical) advantages for simplicity can be said to be an example of "Worse is Better". Because what "Worse is Better" means is just that "theoretically 'worse', but much simpler code is actually better than theoretically 'better' but much more complex code". It's "worse" vs "better" on all those fine more or less theoretical dimensions against "worse" vs "better" on the single dimension of code complexity / simplicity.
So I'd say this article, which was all about how a radical simplification of the code -- sacrificing the (unnecessary) generalization and flexibility they'd first tried to build into it -- turned out to bring a lot of other practical advantages, is a prime example of "Worse is Better". As I understand it.
I wouldn't called it "worse is better". Why not "less is better" or "simpler is better"?
My take is that one should program in function of what the code should do, and not in function of what it's comfortable to me (as a developer).
Yes, it's a great feeling when your code fits like a jigsaw puzzle, but also more complexity = more code being executed. Behind that RAII, behind that "operator=" and that "p = new Struct", etc. there might be extra complexity for the sake of developer's readability and comfortability. There is little or no added value for the end user or the purpose of the program itself.
Also the code should be written "for the now", not for "that future feature it would be awesome to have someday like making it compatible with every other library X, etc.".
At the end of the day, even without realizing it, your program is slow.
I remember a developer where I work did a C# implementation of an AT command parser, in which every AT command was a separate DLL. It was very complex, and super slow. But the developer argued "if I need a new AT command, I'll just add a new DLL". It might have been better for him as a developer, but it was worse for the end user and the system in general. The code died the day that guy left the job.
> I wouldn't called it "worse is better". Why not "less is better" or "simpler is better"?
The original article called 'worse is better' was pretty successful and widely read, and part of it was the title grabbed people's attention. You could argue it's a rephrasing of "less is more" or "do one thing well" concepts for the click bait zeitgeist. That's not a bad thing - an old concept wrapped up in a new way of expressing can keep the idea alive - anyway it's funny and a little confusing, which might trigger someone to engage with it differently. There's many concepts that I've heard n times expressed in different ways only to have it click on the n + 1 way of phrasing it.
Also, the "PC-losering problem" presented as one of the central examples in the original 'Worse is Better' really is an example of 'worse' -- the unix approach gives a modest reduction in kernel implementation complexity at the cost of every single userspace program either taking a complexity hit (retrying on EINTR) or being buggy in corner cases. This isn't just "less" or "simpler" -- it's actively accepting "worse" in order to get a basically functioning thing out the door sooner and maybe fix the most-annoying warts later. (Minimum Viable Product, anybody?)
> Yes, it's a great feeling when your code fits like a jigsaw puzzle, but also more complexity = more code being executed. Behind that RAII, behind that "operator=" and that "p = new Struct", etc. there might be extra complexity for the sake of developer's readability and comfortability. There is little or no added value for the end user or the purpose of the program itself.
'Yes, it's a great feeling when your code fits like a jigsaw puzzle, but also more complexity = more code being executed. Behind that function call, behind that nested expression and that "p = malloc(sizeof(*p))", etc. there might be extra complexity for the sake of developer's readability and comfortability. There is little or no added value for the end user or the purpose of the program itself.'
See? The same argument can be used against C in favor of assembly. Abstractions allow us to write safer, more correct code in less time, which actually does offer a lot of added value for the end user. C makes many useful abstractions harder or downright impossible, which is one reason why software written in C is generally such a shitshow from a security perspective.
> Behind that function call, behind that nested expression and that "p = malloc(sizeof(*p))", etc. there might be extra complexity for the sake of developer's readability and comfortability.
There is no way to avoid most of things you listed. It's not comfort; it's a necessity.
> software written in C is generally such a shitshow
Whatever language you use to code, all your calls will sooner or later pass through a function call, a nested expression or a malloc()/free() call, somewhere in the huge stack of C code your language needs, depends and relies on. There is no escape. Basically, today your preferred language exists and might be able to something just because C code exists.
> There is no way to avoid most of things you listed.
Oh yes, there definitely is: There are no function calls, nested expressions, or malloc in assembly – those are abstractions provided by the language and translated by the compiler, for comfort, not necessity. In fact, you can simulate all that in C itself: use gotos and an explicitly managed memory area as stack, and manually transform nested expressions into SSA form.
You are still paying for the register allocator. That is not a trivial abstraction; it is wildly complex and carefully tuned, and there is ongoing research on how to do it better.
> Also the code should be written "for the now", not for "that future feature it would be awesome to have someday like making it compatible with every other library X, etc.".
The folksy software adage for this is YAGNI. (You Ain’t Gonna Need It)
I’ve been thinking a lot about this kind of problem lately. My thinking was triggered by a description of real-world contracts by Lawrence Lessig, in a talk about crypto. [1] His point was that in most real world contracts there are many, many undefined contingencies for rare occurrences; the time and effort to nail-down what each party should do in those cases is just not worth it, and if any of them do actually occur everyone will go to court for adjudication.
The software industry might be better off if we start making explicit that we have a real trade off of time and effort between designing a system that can handle all contingencies, and one that needs adjudication occasionally, i.e. throws an error or crashes, needs patching, etc. [2]
Notice that Rui’s solution here was to basically allow lld to just crash under rare, extraordinary cases. This seems reasonable when written here, but in the real world suggesting that a system be allowed to crash, ever, often gets very hard pushback.
This is where the programming-language exception feature is valuable.
You could crash, but you can also throw. Somebody somewhere sometime might be able to do a better job with the situation than you can right there and then. Checking status codes all up and down the call graph does not increase the likelihood that something could be done, but costs in the meantime. Hiding the checking behind abstractions does not reduce those costs.
> You could crash, but you can also throw. Somebody somewhere sometime might be able to do a better job with the situation than you can right there and then.
This is a very big assumption.
Edit: The assumption is that this day can ever come. The author decided that in his system that day would likely never arrive.
> when they can just as well leave the choice to somebody else, at exactly zero cost.
This is where the confusion lies. “Zero cost” here means “it’s a few lines of code, a few cpu cycles to handle it. What’s the big deal?”
But I’m not talking about a few lines of code or cpu cycles, I’m talking about developer time. Leaving the decision to someone else may take little no time to you, today, but might incur a steep cost on someone else’s time tomorrow. Even worse is when you, with deep domain knowledge, tempt a far less knowledgeable dev into “handling” an error at a time and in a place that will be difficult or impossible to handle, wasting their time.
You are still not getting this. It costs exactly zero more developer time to type "throw x" than to type "abort()". It places exactly zero demand on anybody else.
If somebody decides they want to catch the exception and try something else, that is totally their choice, and they can devote as much or as little time to getting their idea working as they like. If they get it working, good. If they give up first, that is fine too.
> You are still not getting this. It costs exactly zero more developer time to type "throw x" than to type "abort()". It places exactly zero demand on anybody else.
OTOH, it could be that you're just not getting this: Those exceptions that can't ultimately be handled will perhaps throw a (more or less informative) error message onto the screen and into a log file, but then they'll have to do a (more or less controlled) shutdown of the app.
One form of shorthand for this that I could see myself using is... "just crash". So perhaps that was what was originally meant: Maybe your "throw an exception for godssake!" is "just crash".
I understand perfectly well what you’ve said, and I don’t disagree with any of it. The point I’m trying to make (badly, apparently) is not that.
My point (and I believe the point of the linked article, and, I mistakenly thought, my original comment) is that decisions by a few can have enormous impacts on the productivity of many, including design decisions like “should we bother polishing the errors from these corner cases at the expense of delaying new feature work, or just let people report a bug if they run into a corner case and get on with new features?” Rui chose to group a whole swath of errors in lld into the “not worth polishing” category so they could move on with more important work.
My apologies that my point somehow came across as “always crash your program, never throw exceptions!” Thanks for your patience and taking the time to read generously and deeply and to really understand my point.
I apologize for my impatience. The original description really could plausibly be read to suggest he did not even check for bad input, and just assumed input was OK, maybe with assertions that would run in debug builds. It is true that not even coding sanity checks is less work than arranging to abort or throw, and marginally faster.
Producing faulty output is a more likely response to unchecked faulty input than crashing. It would have been better for the author to mention that, if true.
> Since the linker's input file is created not by humans but by the compiler, it is unlikely that the linker takes a corrupted file as an input. Therefore, the policy did not actually increase a crash rate. The code that trusts input was much simpler than the one that does not trust any byte of an input file.
Interestingly I have encountered crashes in Ninja (not lld), caused by corrupted on-disk state I had to delete: https://github.com/ninja-build/ninja/issues/1978. I think I traced it down to a memory indexing or null pointer error, which would've been caught by asserts but they were disabled in release builds.
There's some commonality here between this sort of "worse is better" and the observation that a meticulously neat and tidy, fastidiously clean {desk, notetaking system, editor configuration, ...} is a good indicator that its owner doesn't do anything worthwhile with it.
The real world is messy. Things that come into contact with the real world will acquire a little bit of wear and mess. If it's all still brilliantly clean and tidy, you can't have done anything useful yet!
> Here is the rule: if a user can trigger an error condition by using the linker in a wrong way, the linker should print out a proper error message. However, if the binary of an input file is corrupted, it is OK for the linker to simply crash.
One of the most useful ideas in developing reasonably complex systems that I encountered is treating different types of errors differently. There's a place for both panics/exceptions on one hand result monads/error messages on another.
I've always considered the EINTR example in that article to be a bad one (although I think I agree with the article in general). Having blocking syscalls interrupted in case of an asynchonous signal is the right thing, because it allows the program to act on the signal. Think for example of a terminal user pressing Ctrl+C to interrupt blocking I/O to return to the shell prompt.
The problem is that this does't go far enough - decades ago machines were running on a single CPU and OSes were focused on the scheduling of processes. Syscalls were all blocking, so for each individual process there could only ever be one syscall ("request") in flight at a time. Now, we're seeing a change (for example with io_uring) towards fully asynchronous I/O exposed to the userland, which allows submitting multiple requests to various I/O devices simultaneously, which has the potential to improve throughput a lot.
What a lot of people seem to not understand is that mistakes or failings are sometimes on purpose. If software is slow, sometimes (not always!) it’s because the software that focused on performance didn’t get traction with users and failed. Too many programmers just see the immediate failings and not the larger, successfully avoided failings that the software prevented due to a tradeoff.
Likewise many programmers don’t seem to get that the incentives of a programmer are quite different than the incentives of a manager. Programmers think “aw man my stupid manager is making me push out features instead of refactoring, if I were in charge I’d focus on code quality and performance”, not understanding that maybe, just maybe their manager might have different incentives and a different perspective.
Worse is better is essentially a shorthand for understanding product management and scoping.
> "It says that lazily-looking code that does not provide a consistent interface is sometimes actually better than neatly layered, consistent one."
I find this an odd take on that paper. To me it has always been about how simplicity is more important than correctness. And while the authors take doesn't conflict with WiB, I do think they miss the point.
> "Simplicity of implementation is very, very important, and you can sometimes sacrifice consistency and completeness for it."
They almost get it in the conclusion. They are starting to see the important of simplicity, but still are fixed on correctness. Simplicity is not something you sacrifice correctness and completeness for "sometimes", it always wins if there is a contest. It is always more important (at least in line with WiB).
> Simplicity is not something you sacrifice correctness and completeness for "sometimes", it always wins if there is a contest.
Not quite: The absolutely simplest code is an empty code file. But that doesn't do anything when compiled, so I doubt it would become very popular; people want their software to do something. And if that "something" is equivalent to "rm -rf /", they'd prefer it to do something correct in stead. So code simplicity doesn't always win; having at least some darn somewhat-correct functionality easily beats it.
> This may seem like an amateur-level programming mistake, but in reality, it's much easier to write straightforward code for each target than writing unified one that covers all the details and corner cases of all supported targets simultaneously.
This is one of the reasons sum types are so critical in my opinion. They let you write code in that style, where OOP forces you to make everything look kinda the same.
Not a good example of "Worse is Better". This is "Worse is faster" which is much less interesting. The power of WIB is that is applies even if the programs are equally fast.
The "PC losering" anecdote in Gabriel's original essay is vert dated.
In fact, neither design is the "better".
In not-so-modern-anymore POSIX, you can choose whether a system call will be restarted after a signal is handled, or whether it will terminate with an error. Both requirements are needed.
It is signals themselves that are "worse". But they let you have asynchronous behaviors without using threads.
Sometimes you want a signal handler to just set some flag. This is because you have to be careful what you do in a signal handler, as well as how much you do. And then if you want the program to react to that flag, it behooves you to have it wake up from the interrupted system call and not go back to sleep for another 27 seconds until some network data arrives or whatever.
In addition to sigaction, you can also abort a system call by jumping out of a signal handler; in POSIX you have sigsetjmp and siglongjmp which save and restore the signal mask. So that would be an alternative to setting a flag and checking. If you use siglongjmp, the signal itself can be set up in such a way that the system call is restarted. The signal handler can then choose to return (syscall is restarted) or bail via siglongjmp (syscall is abandoned). I wouldn't necessarily want to be forced to use siglongjmp as the only way to get around system calls being always restartable.
Anyway, the Unix design showed to be capable of being "worse for now", and have space to work toward "better eventually".
In the present story, the monolithic linker design isn't "worse". Let's just look at one aspect: crashing on corrupt inputs. Is that a bad requirement not to require robustness? No; the requirement is justifiable, because a linker isn't required to handle untrusted inputs. It's a tool-chain back-end. The only way it gets a bad input is if the middle parts of the toolchain violate its contract; the assembler puts out a bad object file and such. It can be a wasteful requirement to have careful contract checking between internal components.
Gabriel naturally makes references to Lisp in the Rise of Worse is Better, claiming that Common Lisp is an example of better. But not everything is robust in Common Lisp. For instance the way type declarations work is "worse is better": you make promises to the compiler, and then if you violate them, you have undefined behavior. Modifying a literal object is undefined behavior in Common Lisp, pretty much exactly like in ISO C. The Loop macro's clause symbols being compared as strings is worse-is-better; the "correct requirement" would have been to use keywords, or else symbols in the CL package that have to be properly made visible to be used.
I don't think that Gabriel had a well reasoned and organized point in the essay and himself admitted that it was probably flawed (and on top of that, misunderstood).
The essays is about requirements; of course the assumption is that everyone is implementing the requirements right: "worse" doesn't refer to bugs (which would be a strawman interpretation) but to a sort of "taste" in the selection of requirements.
Requirements have so many dimensions that it's very hard to know which directions in that space point toward "better". There are tradeoffs at every corner. Adopt this "better" requirement here, but then you have to concede toward "worse" there. If we look at one single requirement at a time, it's not difficult to acquire a sense of which direction is better or worse, but the combinations of thousands of requirements are daunting.
If we look for what is the truth, the insight in Gabriel's essay it is that adherence to principled absolutes is often easily defeated by flexible reasoning that takes into account the context.
3.1415926 is undeniably a better approximation of pi than 3.14. But if you had to use pencil-and-paper calculations to estimate how many tiles you need for a circular room, it would be worse to be using 3.1415926. You would just do a lot of extra work, for no benefit; the estimate wouldn't be any better. Using the worse 3.14 is better than using 3.1415926; that may be the essence of "worse is better". On the other hand, if you have a calculator with a pi button, it would be worse to be punching in 3.14 than just using the button, and the fact that the button gives you pi to 17 digits is moot. A small bit of context like that can change the way in which the worse-is-better reasoning is applied.
Code is a tool for exploring and understanding problems as much as it is about solving them. Sophisticated solutions can't be designed before they are validated.