Over time, I have come to believe that the problem is overly-aggressive abstraction. It is very tempting for most developers, especially good developers, to reach for abstraction as soon as things get complicated. And this can pay off very, very well in some cases. However, too much abstraction in a system leads to a very, well, abstract code base that becomes hard to get a handle on: there's no there there. You see this in the java world with AbstractFactoryBuilderLookupServiceBuilders and so forth, and with very elaborate type setups in functional programming languages.
Concretizing the crucial bits of your system, even if that means a few large, complex and gronky methods or classes, ends up often making things more understandable and maintainable and, certainly, debuggable.
John Ousterhout wrote a book that makes this point as well, advocating for "deep" rather than "shallow" classes and methods:
> Over time, I have come to believe that the problem is overly-aggressive abstraction.
Sometimes, yes, overly-aggressive abstractions are a problem. But the author describes policies of v1 and v2 of the linker. And I would say the most critical difference between them is, that the authors of v2 had a better understanding of the requirements that actually mattered. Therefore they were in a better situation to evaluate the architecture and they were able to make better trade-offs.
Deciding to trust object files as inputs might raise a red flag for some people. In principle it could allow attacks/exploits of the linker. But in reality for most threat models this does not matter. Because the compiler generating object files has the same level of trust as the linker. Companies that want to provide an elf linker as a service product are out of scope. Deciding what is in scope and what is out of scope is probably one of the hardest decisions in product engineering. Because many of us software engineers lean towards perfectionism or, especially inexperienced developers are searching for the silver bullet, that set of rules that enables them to develop every product successfully.
I suspect the "Worse is Better" essay has caused untold harm. It is written from the standpoint of a purist offended that the world doesn't appreciate purity, complaining that the things people end up using are built by pragmatic people. The lesson seems to be that doing things better is punished. But that is the wrong lesson.
The correct lesson is that the real world is not obliged to conform to your personal model of "better". You have a personal obligation to continuously adapt your model to match the real world. This is the model of science, and is opposed to Platonism. After you have adjusted your model, it is certain to still not be right, and need further adjustment.
Usually "the world" you are obliged to adjust to has its own problems. There are powerful forces making us favor accommodating a Microsoft execution environment, even though that execution environment has always been a cesspit. It represents its own poor abstraction. Posix file system semantics are another example. Von Neumann architecture and the C abstract machine are not the only, or best way to organize computational resources. It is important to recognize when somebody else's pragmatic failure threatens to taint your own models.
Lisp, RG's hobbyhorse, didn't get sidelined because of Philistines. Lisp turned out not to be better, despite how strongly RG felt about it. Instead of figuring out what about Lisp was not right, he called things that were, along axes that matter, more right "worse", preserving his personal model and lessening readers' ability to reason about merit.
I don't agree at all. I think he was able to give a clear and accurate analysis of why lisp failed, despite liking the language so much. He also wrote critical responses to his own essay under a pseudonym, with a back and forth that is quite funny and demonstrates the ability to understand both sides of the argument. In "Worse is better" he explicitly mentions how that approach favors real-world application, because it is so simple it is fast and is "good enough" and then can be moved to 90% of the right thing.
All of this is to say: I don't agree with you, but I also agree with you and I suspect he would as well, with qualifications. And he would probably also disagree with you.
Not buying it. He still says "worse" and still writes "90%". What he means is "not conforming to my personal value system". He is dodging the crucial step of discovering how his personal value system got so off-kilter as to lead him to wrong choices, and fixing it.
Lisp machines were clearly abandoned because of their price. Every time I find some history about someone that actually made that decision, the reasoning was exactly alike, those machines costed more to keep than the Unix ones to install, and were less capable due to outdated hardware.
Yet the essay goes all over the place, citing time to market (that was completely irrelevant, UNIX was the newcomer, Lisp machines were there already), university-based prejudice (yet every single one decided the same at around the same time), and blaming the user. The essay doesn't even talk about money.
It would probably help you to understand the essay if you knew that it was written by Richard P. Gabriel, the head of Lucid, the leading competitor to Lisp machines. His company's product was a Lisp system that ran on Unix machines. What he's personally best known for (aside from this essay) is showing that smarter Lisp compilers on commodity hardware like a 68020 or a SPARC could deliver performance that first equaled, then exceeded the performance available from custom silicon; the book he published on this subject was so rigorous that many of the tests in it are still used today for judging the performance of implementations of high-level languages such as LuaJIT and V8.
You seem to think he was advocating Lisp machines, scoffing at his essay based on that misconception. But if there was a single person in history who worked hardest to destroy Lisp machines, it was probably RPG.
Yeah, that's more like it. RPG worked a lot to bring sophisticated Lisp to UNIX systems. He worked on bringing a standard OOP system to Lisp (incl. Lisp Machines, where Lucid (and he personally) collaborated with Xerox PARC and Symbolics to develop CLOS as a part of the ANSI Common Lisp standard).
Then he tried to develop and market similar technology in the form of a databased-backed incremental C++ development system for UNIX (called Energize). https://www.youtube.com/watch?v=pQQTScuApWk
He hoped that the market for C++ environments was larger than the shrinking Lisp market and that C++ developers would want an interactive system with integrated code management in an object store. That did not went well.
Reports were that Energize customers were relieved when Lucid folded, because they would then not get a whole new set of compiler bugs to discover every quarter. They had, instead, the bugs they already knew about and had learned to work around.
Lisp Machines anyway raised the standard of quality in CRT monitors. Manufacturing their own monitors has to have contributed substantially to their downfall, but we all benefited in the end. Well, all but them.
There is an unfortunate fallacy here. Your notion "the real world" conflates the physical world with a social milieu. The physical world is (at this scale) immutable, so of course we must conform to it and update our science-like models.
But there is no absolute requirement to conform oneself to a social milieu. A social milieu changes. It can be altered. It supports a vast number of models. And milieus overlap so densely that one can just go play somewhere else.
Disclosure: I used to be a Lisp bigot, but I got better.
Agreed it is complicated. We have the principles of physics and computational irreducibility, then physical realization of state-machine designs to exploit those principles, then social conventions around what is valued, languages influenced by conventions, operating environments for programs in them, and finally actual code in those languages.
What makes a "better" Lisp program differs from what makes a "better" C++ or Rust program. Besides fitness for purpose, there is maintainability, energy cost to execute, and results per unit time. What did you learn coding it? Can it be the basis for something more ambitious? Is it secure, deadlock-proof? Are its results accurate, aesthetically pleasing, generative of insight?
We can get mired in detail, obscuring important truths. We talk about performance a lot, not because we are obsessive, but because it is a measure that is hard to fake. A faster program says something fundamental about how your computational resources are being directed to the target results.
We can be misled by details of realizations of computation. What is fast on a PDP-11 is not necessarily fast on a Ryzen 5. But submitting to the rigors of performance for the best machines we have is a discipline that connects us, howsoever imperfectly, to the physical principles of computation. It destroys illusion: if a variation seems like it ought to be faster, but is actually slower, there is no sugar-coating the fact. Hewing to physical performance enforces a kind of honesty we don't get any other way.
> The correct lesson is that the real world is not obliged to conform to your personal model of "better".
(Annoying as fuck, that. ;-)
And that seems to indicate that a(n initially) half-assed, but improvable, model of it that is then iterated upon is a good way to build one's model. Feels to me like that is exactly what "Worse is Better" is about -- shouldn't your beef here rather be with "the MIT model"?
> You have a personal obligation to continuously adapt your model to match the real world.
A bit hard to get a consensus model out of that, since everyone's perception of the real world -- heck, everyone's actual "real world" -- varies.
> This is the model of science, and is opposed to Platonism.
Again, what feels like setting up a Platonic ideal to me is more "the MIT model", rather than "Worse is Better".
> After you have adjusted your model, it is certain to still not be right, and need further adjustment.
Ah, dangit, at some point you just gotta say "Screw it, good enough!" (See, for instance, "The saddest 'Just ship it!' story" the other day.)
This will sound crazy from our perspective this year, but from the perspective of the essay 31 years ago, Lisp dominates software development today; we just spell it without parentheses.
The top ten languages in https://www.tiobe.com/tiobe-index/ right now are Python, C, Java, C++, C#, Visual Basic, JS, assembly, SQL, and PHP. Of these, the "dialects of Lisp" include Python, Java, C#, VB, JS, and PHP.
Remember that in 01991 all "serious" modern software was either C, C++, Pascal, or assembly. BASIC, whose only data structures were the string and the array, was for amateurs or the MIS department, which mostly used COBOL, assembly, and JCL. Fortran was established but was considered antiquated (except by supercomputer people) and didn't have pointers.
If we compare these languages on the points pg lists in http://www.paulgraham.com/diff.html, the earliest versions of Lisp score 9/9, Python is 7/9, Java is 7/9, C# and VB are Java with different syntax, JS is 7/9, and PHP is 5/9. By contrast, C is 2/9, C++ is 3/9, Pascal is 3/9, assembly is 0/9, COBOL is 0/9, Fortran 77 is 1/9. You can quibble a bit about these numbers (do Pascal procedure parameters qualify as "a function type" even though you can't store them in variables? Does the Python expression "intern(s) is foo" qualify as "a symbol type"?) but the broad division is very clear.
I think it was in the conversation where he originally wrote that essay that Guy Steele said of his own work on Java that they had managed to drag all the C++ programmers kicking and screaming about halfway to Common Lisp, so we should be happy. I've lost my archives from that time, so I can't be sure.
In terms of syntax, Python or Java have nothing in common with Lisp. But in terms of the issues you raise — accommodating a Microsoft execution environment, POSIX filesystem semantics, the Von Neumann architecture, the C abstract machine, or just the tools they give you to analyze problems — they're just dialects of Lisp with slightly different syntax (and more hair on eval, and sort of broken symbols or no symbols).
In terms of "worse is better" of course Python and C# lean just as hard on "worse" as C does.
If we restrict the sense of "Lisp" to languages with S-expression syntax like Emacs Lisp, Common Lisp, Scheme, and Arc, then Lisp did fail to (at least) become popular — but it's plain to see that when people abandoned Common Lisp and Scheme, they were mostly moving to languages like Python and JS which adopted Lisp's most appealing ideas, not to C++.
> Your notion "the real world" conflates the physical world with a social milieu. The physical world is (at this scale) immutable, so of course we must conform to it…
> But there is no absolute requirement to conform oneself to a social milieu. A social milieu changes. It can be altered. It supports a vast number of models. And milieus overlap so densely that one can just go play somewhere else.
In that vein, it's worth noting that Linux got pretty far before it ever had to accommodate a Microsoft execution environment, though I did have coworkers in 01997 who thought I was hopelessly unhip for preferring Unix (which they thought of as antiquated) to Microsoft Windows.
> This will sound crazy from our perspective this year, but from the perspective of the essay 31 years ago, Lisp dominates software development today; we just spell it without parentheses.
It is true that some good ideas originate from Lisp. However, by saying that 'Lisp dominates software development today', you are giving the false impression that the Lisp heritage is the only one that matters, as if Lisp were some sort of linguistic asymptote toward which all other programming languages must inevitably converge. I take issue with that. Creating a new programming language involves copying the good ideas from previous languages and rejecting the bad ones. The notion that this or that modern programming language is a 'dialect of Lisp with slightly different syntax' completely ignores all the bad ideas from Lisp that were not copied and all the good ideas from other languages that were.
Yes, Lisp itself abandoned the idea of car, cdr, and cons as the foundation of all data structures about 50 years ago. On the other hand, many languages further from Lisp than Python or JS did adopt the idea of car, cdr, and cons as the foundation of some data structures, notably Prolog, OCaml, Standard ML, F#, and Haskell.
Well, I suppose that's true, although mainstream languages that are uncontroversially Lisps, such as Emacs Lisp, Scheme, and Common Lisp, have also abandoned a lot of Lisp's bad ideas. And I do agree that Lisp isn't something toward which all other programming languages inevitably converge—neither any particular historical Lisp, nor some Platonic ideal of Lisp.
But I do think it's a sort of "linguistic extreme" that defines a dimension in programming-language space, and new popular languages tend to be pretty far in the Lisp direction. In TIOBE's top ten languages, the four I identified as "not Lisps" are pretty old: C is from, say, 01973; C++ is from 01982; SQL is from 01974 (and wasn't Turing-complete until a recent unfortunate accident, therefore not a programming language); and assembly language is from somewhere between 01948 and 01968. The other six popular languages I listed are all from 01989 or later; C# and arguably VB are from 02000.
There are several other linguistic extremes that things could have moved towards instead. For example, FORTH, Self (the essence of Smalltalk), miniKANREN (or maybe Mozart/Oz/Prolog), Coq (or at least some kind of cleaned-up ML), the π-calculus, m6, Inform 7, some kind of lazy language (maybe Haskell), and now something interesting is emerging in Rust. You could imagine a history in which all the new languages adopted key controversial ideas from FORTH or ML instead of from Lisp, but that mostly isn't the history we're in.
Java/C#/VB in particular takes pervasive ad-hoc polymorphism from Smalltalk, parametric polymorphism from ML, and type declarations from, I guess, Fortran, though detecting errors at compile time wasn't a motivation for having them in Fortran. And Python takes pervasive ad-hoc polymorphism from Smalltalk and gets its parametric polymorphism for free with the dynamic typing it takes from Lisp. (Python's predecessor ABC was a teaching language derived from ML, which is why Python spells null as None, but Python dropped the static typing.)
Here are some candidates for Lisp's abandoned or not-adopted bad ideas that come to my mind:
1. Dynamic scoping as default. PostScript and TeX do do this, but nothing else does. Even Common Lisp and Emacs Lisp have abandoned it. (And Scheme never had it to begin with.) Nevertheless, it was viewed as a defining attribute of Lisp.
2. The β-reduction model of program execution, in which the output of the program is a transformed version of the program; and, more generally, there is no clear distinction between the program and the data it processes. This was sharply abandoned in Scheme, but without abandoning metaprogramming. Mathematica is the only current language I know of that works this way. (Maybe Q?)
3. Textual serialization (READ, PRINT) as the standard and indeed only serialization. (The original Lisp and many of its successors also have pervasive serializability, which pg inexplicably omitted from his original essay: any data object can be written to a byte stream, and the same or equivalent data object can later be read back from it.) Variants of this are pretty popular; JSON.stringify, Python repr(), Firefox .toSource(), Python pickle, etc., but in most cases modern systems relieve the tensions between the different uses of READ and PRINT by providing several different serializations.
4. The general idea that the important part of software development is getting a prototype working, after which making it robust is comparatively trivial. This idea probably does not originate with Lisp, and it is popular in startup business literature, but not in current programming language design. But people use Python in Jupyter a lot to interactively explore computational ideas.
5. Symbols as the only string type, which is to say, hash-consing all your strings. This is sometimes a worthwhile thing to do in a particular library but not a good tradeoff to impose on all users of the language. Lisps stopped doing this about 50 years ago too.
I'm interested to hear which bad ideas from Lisp you're thinking of!
I guess that's true. But maybe those projects should have been done in C++ (or ML or something) instead of Lisp in the first place, because the things that Lisp is good at are very much not the things C++ is good at. They're the things Python, Java, and JS are good at. As Perlis wrote in the Foreword to SICP:
> Pascal is for building pyramids—imposing, breathtaking, static structures built by armies pushing heavy blocks into place. Lisp is for building organisms—imposing, breathtaking, dynamic structures built by squads fitting fluctuating myriads of simpler organisms into place. The organizing principles used are the same in both cases, except for one extraordinarily important difference: The discretionary exportable functionality entrusted to the individual Lisp programmer is more than an order of magnitude greater than that to be found within Pascal enterprises. ... To illustrate this difference, compare the treatment of material and exercises within this book with that in any first-course text using Pascal.
You can easily substitute "C++" for "Pascal" here and "Python" or "JS" for "Lisp", and indeed a JS translation of SICP has been completed, including a column-by-column comparison: https://sicp.sourceacademy.org/chapters/5.2.3.html
> But maybe those projects should have been done in C++ (or ML or something) instead of Lisp in the first place, because the things that Lisp is good at are very much not the things C++ is good at.
Common Lisp was not developed as a scripting language like Python or JavaScript. It was developed for complex applications (100k to 10M lines of code were not rare - for example the PTC CAD system written in C and Lisp had 7 MLOC Lisp code already over a decade ago) and derived from a language with was actually a systems programming language: Lisp Machine Lisp.
Thus there were a bunch of ambitious projects initially written in Lisp and then continued in C++ (and also in C) for better performance on smaller machines, smaller memory footprints and less need to mix C and Lisp.
I know it was, and today Emacs has over a million lines of Lisp even without considering Elpa, but I think Lisp embodied some ideas about the best ways to structure large programs that turned out to be wrong.
In particular, Lisp in general (including Common Lisp) is designed as far as possible to maximize flexibility, and I think there's an unavoidable tradeoff between flexibility and correctness. Oversimplified, programs are flexible when they can do things their original author did not anticipate; they are correct when they cannot do things their original author did not anticipate.
Of course, in reality, it's not that simple. But this obviously false formulation contains important bits of truth. Testing a flexible program is more difficult, and more bugs are likely to slip through your testing, and each one usually takes more effort to diagnose and fix. And of course in general any two virtues trade off against one another in the limit, because if you take every possible measure to increase the first, some of them will happen to decrease the second. (By the same token there are some measures you can take that will increase both.)
Yeah, or sometimes by abusing the fuck out of templates, though that wasn't a possibility yet in the early 01990s. Emacs uses the same approach (adding an embedded dynamic scripting language), and I think PTC CAD does too, and in these cases the "embedded dynamic scripting language" is a Lisp. (AutoCAD, too, but I don't think they wrote significant parts of AutoCAD in AutoLisp.) It's a good approach that avoids paying the cost of flexibility except where you need it.
Since Lisp is often compiled to machine language, C/C++ is often only glue to the OS and large parts of the software is actually written in Lisp. GNU Emacs now has a variant with native compilation, so it there would possible to get rid with much of the C code without much loss of performance. Thus in a large CAD system, compiled Lisp is not just the scripting language, but can be much of the implementation.
Yeah, as I said above, it's not mostly about the performance¹ but about the bugginess, which is to say, the comprehensibility. There are arguments both ways about whether C or Lisp is more bug-prone (less comprehensible) but I think the results are in. And there are new languages like TypeScript which combine and exceed the advantages of both.
______
¹ Though GCC still routinely generates much better code than SBCL, much less the new Elisp compiler, that's not the primary consideration. I'm not sure why you're mentioning it.
It can be worth understanding why the Java ecosystem evolved that way. It's not because simplicity wasn't valued, but rather it's not code simplicity that's valued. Rather, much of the Java ecosystem is designed to allow dozens of independent teams, totaling hundreds of developers, within a company (or across companies) to build their small piece of the application, with whatever build process is used by them. Then you put all the .jar files in one bundle, and the AbstractFactoryBuilderLookupServiceBuilders puts the pieces together at runtime.
It's complex as hell code-wise, but it simplifies the amount of cross-team/cross-company alignment and synchronization that has to happen in order to cut a new version of the application containing hotfix 8495 for the part of the app maintained by Steve's team.
There's actually a decent parallel between that and Microservices. Microservices make maintenance of the whole more complicated by introducing the network between pieces, but allow each piece more flexibility in how it's developed.
The folks at Lucid were starting to get a little worried because I would bring them review drafts of papers arguing for worse is better, and later I would bring them rebuttals against myself. One fellow was seriously nervous that I might have a mental disease.
after over a decade of thinking and speaking about it . . . [I wrote] "Back to the Future: Is Worse (Still) Better?" In this short paper, I came out against worse is better. But a month or so later, I wrote a second one, called "Back to the Future: Worse (Still) is Better!" which was in favor of it.
This is the heart of engineering or politics: finding the optimal compromise.
The kind of codebase you describe with "elaborate type setups" (multiple levels of inheritance, design patterns like Facade used for "decoupling" etc.) make it very hard to read the code and understand what it's doing (unless it's very well documented), and because of that also make it harder to extend without resorting to kludges. Or, if you want to extend it in a way the original designer didn't foresee, you now have to change 5 different classes to be able to access some private property in some class.
One of the best programming aphorisms I've heard is "Everything in programming can be solved with another layer of abstraction except for the problem of to many layers of abstraction."
Also, I often see developers running to the abstractions they know (cough design patterns...) mostly because they're very afraid of acknowledging they haven't grasped the real complexity of a domain. By the time a pragmatic efficient design emerges, you're on the second or third rewrite...
Worse is better is simply a local maxima of features - schedule - cost tradeoffs, and there not being enough of a sustained investment to "hop out" of the local maxima.
When those maxima are things like operating systems and programming language ecosystems, those are very very big hops to make.
https://www.dreamsongs.com/WorseIsBetter.html
Over time, I have come to believe that the problem is overly-aggressive abstraction. It is very tempting for most developers, especially good developers, to reach for abstraction as soon as things get complicated. And this can pay off very, very well in some cases. However, too much abstraction in a system leads to a very, well, abstract code base that becomes hard to get a handle on: there's no there there. You see this in the java world with AbstractFactoryBuilderLookupServiceBuilders and so forth, and with very elaborate type setups in functional programming languages.
Concretizing the crucial bits of your system, even if that means a few large, complex and gronky methods or classes, ends up often making things more understandable and maintainable and, certainly, debuggable.
John Ousterhout wrote a book that makes this point as well, advocating for "deep" rather than "shallow" classes and methods:
https://www.goodreads.com/en/book/show/39996759-a-philosophy...