The best way to achieve a good abstraction is to recall what the word meant before computer science: namely, something closer to generalization.
In computing, we emphasize the communicational (i.e. interface) aspects of our code, and, in this respect, tend to focus on an "abstraction"'s role in hiding information. But a good abstraction does more than simply hide detail, it generalizes particulars into a new kind of "object" that is easier to reason about.
If you keep this in mind, you'll realize that having a lot of particulars to identify shared properties that you can abstract away is a prerequisite. The best abstractions I've seen have always come into being only after a significant amount of particularized code had already been written. It is only then that you can identify the actual common properties and patterns of use. Contrarily, abstractions that are built upfront to try and do little more than hide details or to account for potential similarities or complexity, instead of actual already existent complexity are typically far more confusing and poorly designed.
Yes, abstraction and generalization are properties you'd rather look for the second time around. Someone was already warning about this 25 years ago [1]:
You have a boring problem and hiding behind it is a much more interesting problem. So you code the more interesting problem and the one you've got is a subset of it and it falls out trivial. But of course you wrote ten times as much code as you needed to solve the problem that you actually had.
Ten times code means ten times cost; the cost of writing it, the cost of documenting it, it the cost of storing it in memory, the cost of storing it on disk, the cost of compiling it, the cost of loading it, everything you do will be ten times as expensive as it needed to be. Actually worse than that because complexity increases exponentially.
This person did his own CAD software from scratch in order to make custom chips [2].
Have you ever looked at how useless Chuck Moore's stuff is? Like, the chip designs are of the type “384 independent Forth chips with tiny amounts of RAM and mediocre interconnects, and if you want to actually do anything with them, you'll need to use 128 of them to program your own DDR3 controller”. Or, he demonstrates how awesome Forth is by showing that you can do “the guts of a disk driver” in five lines, except that it's the dog-slow PIO mode.
It turns out that if you can just change the problem statement, then sure, you can write very simple things. But if you have a real problem to solve (and you can't just say “no, I want to solve a simpler problem”), the Chuck Moore way of thinking doesn't really produce, well, good solutions. It simply doesn't scale to anything large, and not everything can be made small.
I did some work with his Greenarrays chip as a student. It was very limited and awkward to use but it could also operate on absurdly low energy, with very low overhead for waking up or almost completely powering down. At some point, we had a demo running some simple signals processing code off a homemade bleach battery, and I wouldn't be surprised if you could make something work off a bunch of lemons too.
This was over a decade ago (yikes) and I don't remember the exact numbers, but I do remember it used substantially less power than a comparable MSP430 microcontroller.
That seems pretty useful and impressive to me, especially given it was created by a very small team with limited funding.
IMHO that is a failed example of "Chuck Moore's stuff".
He went down a rabbit hole to an extreme level because that's what he does.
His earlier CPU experiments like the 4016 and Shboom were excellent examples of ultra-RISC architectures.
The thing Chuck explored, related to abstraction, which I don't see much in conventional machines was reducing calling overhead. (1 cycle call and intrinsic return on many instructions ie: free return)
Some of the decisions we make today have a lot to do with what happens at the hardware level when we add abstraction. It just costs more than we are prepared to pay so it is avoided ... but for the wrong reason.
Harris RTX2000 is a 8Mhz machine with 16-bit data-path, paged program memory, and 1MB addressable memory. This is really an example of "small machine".
Philae had to choose this CPU because there are very few rad-hard low-power CPUs available (and it's not even that low power by modern standarts, 5 mA/MHz), but I am sure they'd choose something bigger if they could.
As for radio telescope, I am not sure which ones are you talking about, but those environments are not particularly challenging compared to spaceflight, so those run whatever hardware designers like. I am sure some of them used to run tiny 16-bit CPUs, but I'd be surprised to hear new designs run something that old.
> Like the RTX2000 which landed on a comet kind of useless?
Yeah, in 1983 he designed a chip that was further developed by others for space usage.
> Maybe the controlling radio telescopes kind of useless.
Yeah, which he did in 1970.
Note a pattern here? That this design paradigm holds up pretty well in a primitive computing world when things are simple and demands are low, and is thoroughly useless to keep on promoting today?
> That this design paradigm holds up pretty well in a primitive computing world when things are simple and demands are low, and is thoroughly useless to keep on promoting today?
Somehow all of those projects are… really old? A satellite control UI on top of Windows 7 from 2015 (somehow Chuck Moore's assertion that “If they are starting from the OS they have made the first mistake” did not extend to Windows here). A STM-16 (2 Mbit/sec!) multiplexer, very modern. A power plant control system from 1995. And yes, an aerospace project indeed, also from 1995. Notably none of these are using his CAD software, much less using any of his chips.
I'm sure they are? There are people interested in all sorts of things. (Well, at least some of them were in 2022, which is the last time someone bothered to add a news post.) That doesn't mean it is a useful design paradigm for the world at large. Remember, Chuck Moore's claim is that not following his ways means 10x the bugs, cost, etc. -- I don't really see anything supporting that claim.
I wrote a little chess engine in Python that plays at least elo 1400 using alpha-beta search with the goal of making it simple and pedagogical, trying to outdo Lisp. I am thinking about making it talk XBoard, removing the experimental stuff, then posting it to GitHub just as a nice example.
I think though if I want to get more into chess programming I'm going to switch to a faster language.
> Have you ever looked at how useless Chuck Moore's stuff is?
I've seen someone well known in the Forth community say something like that to Chuck. I think he said "what can I do with that?"
If GA144 is useless, it's only because it has not been put to good use, in my opinion. I think it was more-or-less his answer too, IIRC.
I work with system-on-chip or system-on-modules. You know, the ARM-based chips with tons of peripherals. I also worked with similar chips before the ARM era.
The complexity of these chips is as of today, absurd. I/O is multiplexed to make the chip usable for various things, but one has to configure all of them and watch out for conflicts. Then there's also zillions of configurable clocks in order to reduce the power consumption. Solutions to problems that spawn new problems, not in the "divide and conquer" style, unfortunately. This resulted in "device tree" configuration in Linux, a runtime configuration system, because CPU companies excrete a new variant every week.
Maybe I fool myself, but I can see a GA144 bit-banging IRDA, LCD, SPI, etc. in a much more efficient and flexible way.
> (2010) is a fairly interesting experience from someone on the outside trying to work in the same way. It… didn't work that well.
And here in 2025 there's still Forth-based companies alive, like MPE or Forth, Inc.
More specifically, this author writes,
[...] it was harder than we thought. Presumably that was partly the result of not reading "Stack Computers: the new wave", and not studying the chip designs of Forth's creator Chuck Moore, either. I have a feeling that knowledgable people would have sneered at this machine: it was trivial to compile Forth to it, but at the cost of complicating the hardware.
Implementing a stack-based processor without reading the literature on them is a bit foolish, don't you think? The rest is in the same vein; I can see why a person who was essentially a Forth newbie had a bad experience with this kind of project. If this article "debunks" Forth, it is by showing that just because something it is simple, doesn't mean it is easy. Because the world is not simple and simplifying is much harder than let Complexity loose.
The real time visual mixing console that produced many music videos that ran endlessly on MTV back when they actually played music videos, and special effects for blockbuster films like RoboCop and Total Recall, wasn't a "simple thing".
Coco Conn and Paul Rother wrote this up about what they did with FORTH at HOMER & Assoc, who made some really classic music videos including Atomic Dog, and hired Charles Moore himself! Here's what Coco Conn posted about it, and some discussion and links about it that I'm including with her permission:
>First shown at the 1989 Siggraph Electronic Theater to a rave response, this 3 minute humourous film went on to win several top computer graphic awards that same year including Niccograph of Japan.
>Coco: This was a show favorite at the SIGGRAPH film show that year. The year before the conference committee decided that showing demos wasn't the way to go anymore. Peter wrote Flying Logos as a way to sneak our demo reel into the show by turning it into a story. It worked and we made it into the film show.
>Don: I truly believe that in some other alternate dimension, there is a Flying Logo Heaven where the souls of dead flying logos go, where they dramatically promenade and swoop and spin around each other in pomp and pageantry to bombastic theme music. It would make a great screen saver, at least! Somewhere the Sun Logo and the SGI Logo are still dancing together.
----
Peter Conn and I [Coco Conn] had a company called HOMER & Assoc. which was located at the Sunset Gower Studios from 1977 until we closed shop in 1997. We made music videos, commercials & computer graphics/special effects for feature films. One cool note, we worked with Paul Verhoven on both RoboCop in 1986 and the x-ray scene for Total Recall in '89.
HOMER was actually a real time visual mixing console that our in-house engineer spent 1978 - 1981 designing and building, from scratch. The name HOMER stood for "Hybrid Optical Montage Electronically Reproduced." I helped as well, soldering the LEDs on the console and running cables. Peter built his own optical printer and three years into the build we also bought an early computer paint system. Our engineer finished building the console and promptly decided to move to England. We hadn’t used it because we still hadn’t found the right software to run the system. Luckily that’s when Paul Rother joined the company.
The joy stick on our console would bump you to the next line of code (being a command or sequence of events: fade, cut, dissolve, etc.) The console had touch sensitive fader pads. There were no dials. I think they were made by Allison? Each channel (which controlled either a slide projector or a film projector) was touch sensitive. After recording a sequence we could then tweek the current version using additional effects the channels offered such as momentary, additive, on/off, etc. For instance if you wanted to crossfade two images, you could either program it or perform it. Of course everything you did was recorded and would play back on the next round. You literally performed a sequence of visual effects with your hands. Peter would do countless passes until everything was perfect. This performance would then be played back to IP film on the optical printer. Each slide tray or film real would be individually run, one by one, to IP film. Sometimes there would be 10-15 or more passes to get all the elements transferred. Once that was done we would then convert the IP film to video and do additional video editing and effects. A totally nuts analogue system. But it worked.
---------------
HOMER Explained by Paul Rother, in-house programmer, (1982):
The photo is Paul sitting in front of the Optical Printer 7-bit Paint system, Homer and Associates, circa 1982. Homer and Associates was really one of a kind kinda of company. Founded by Peter Conn, originally I got hired to program Homer II, a visual realtime mixing console. Homer I is another whole story, but before my time. Homer II consisted of 16 slide projectors, 4 movie projectors, a 4 track tape recorder, 24 visual channels (each with its own Z80) touch sensitive sliders, a master Z80 S100 bus system and featuring "the joy stick bumper " control, which looked liked the gear shift right out of a 1964 mustang convertible.
The idea was that you would program a visual sequence, then play the sequence in sync with the sound track on the joystick, including cascades, bumps, cuts, etc. The whole thing would be recorded, and if you wanted to, like an audio mixer, go back and do over dubs, making corrections. Then once you had the perfect "hero" recording, you take the 8" floppy disc with the hero recording and the trays of slides to the optical printer, and record it to IP motion picture film, making multiple passes, one tray at a time. Now that I think about it, it was a crazy idea. We actually got the whole thing to work. And it worked great!
Forth & Charles Moore
We hired Forth, Inc. and got Charles Moore, the inventor of FORTH to program the console host computer. I learned FORTH and worked with Charles. I programmed the 2K byte EPROM in each visual channel. On the Master Z80 system we ran PolyForth a multi tasking system in 32K bytes. We had an extra 16K RAM for buffers and things. If I remember right, the system ran four tasks, but that was 20 years ago, my memory may be hazy.
Anyway, I learn not only FORTH from Charles Moore, but also how to factor code in to small reusable routines, WORDs they're called in FORTH. I learned Object Oriented Programming without knowing it. Also a lot of use of vectors. Its a cool language. Charles Moore was a great inspiration to me, and really taught me a great deal that they never taught me in computer programming school.
CAT-700
After we got the basic Homer II working and were able to record on the optical printer, Peter had another idea. He wanted to be able to see the movement of the optical printer, and see a prior frame compared to the current frame. We already had a video assist on the Fries Mitchell 35mm. What we needed was a Frame Buffer. We heard of S100 video board called the CAT-100, which was 1-bit frame buffer, good enough for what we needed. Somehow we never found a 1-bit version, but we found 7-bit version in the recycler!
We flew to Reno, rented a car and drove to a log cabin up in the hills of Truckie California. We got a demo of the thing. The guys were super secret and didn't want us to see the controlling program. It worked, so we bought it, and then flew onto Palo-Alto and met the French guy who designed it. They checked it out and it was OK. This was the days before computer designed boards, and all the traces on the board were curvy, kinda like a Van Gogh painting. We learned that it was 7-bit (CAT-700) because it would have been an 8-bit, but they could not get the 8th bit to work. We spent the night in Palo Alto with a Stanford friend of Peters working on a crazy secret Apple project, the Lisa. 32KByte Paint System
So I got the CAT-700 frame buffer to work, programmed in FORTH. So in that 32K we had an optical printer control system, and a paint system, all in one. (Also the OS, compiler, debugger, etc.) We later hooked up a Summigraphic Bitpad (before the Watcom tablet) and were able to draw on top of digitized frames. It got to the point where we needed TWO optical printers, one to digitize from film, and the other to record to film. Rube Goldberg is not strong enough descriptive to describe the system, with the filter wheels and all on stepper motors, it made music. The first use of the system was effects for Steve Miller Music Video, Abracadabra. I also remember using it on the George Clinton Video, Atomic Dog.
This photo was taken right after we got the system to work. I had hooked up an analog slider box, which controlled things like color. There were 4 color maps we could switch between instantly We did a lot of work in planes, using 2 planes for the original image to be rotoscoped, and the other 5 planes to draw onto. This photo was taken for an article in Millimeter Magazine. The photo ended up being a two page color spread, and I think Peter was pissed, cause I got premier exposure.
TTL logic
At Homer and Assoc. I also learned TTL logic and designed a number of computer boards for the S100 bus. One that controlled stepper motors with a timer chip (Motorola 6840). Another to control the Slide Projectors also using the same Motorola timer chip to control the lamp triacs. My favorite thing, about the system, was the use of the cassette storage interface as a cheap timecode reader/writer.
Although of course solving the abstract problem does not have to be 10 times as much code. The best solutions are often those, that recognize the more general problem, solve it with little and elegant code, then turn to the specific problem, expressing it in terms of the abstract problem and thereby solving it in just a few lines of code. Such an approach should usually be accompanied by some documentation.
To give a trivial example: Binary search or any other bog standard algorithm. You would want to have an implementation for the algorithm and named as such and then only apply it in the specific case you have. Sorting algorithms. You don't want to rewrite it all the time. Actually rewriting it all the time would be the thing that creates "10 times" the code.
Tell that to Richard Hamming: "Instead of attacking isolated problems, I made the resolution that I would never again solve an isolated problem except as characteristic of a class." [1]
I have seen this "premature abstraction" warning creep through our discourse lately, but I don't clearly understand it. I feel like I'm making calls all the time about when to introduce functions or classes that will save you time or effort in the future without forcing yourself to see the repetition before you do. Not only that but Hamming's advice has rung true in my career. Solving the general problem is often easier than solving a specific case and can be re-used for later instances, too.
> Solving the general problem is often easier than solving a specific case and can be re-used for later instances, too.
You and the other person are both correct. What you're saying makes sense and it is what everybody is trained to do. However, it leads to a lot of useless code exactly because you're applying an abstraction that is used only once. That's why most codebases are bloated and have a huge number of dependencies.
To your point, we use abstractions all the damn time. They're everywhere. Even programming languages are an abstraction (especially high level ones). You and I, and everybody else here doesn't pick a cylinder and a block to write to and tell the hard drive to move it's arm into place and record the magnetic data, no we all talk about inserting a row into the DB.
Abstractions are essential to productivity or you'll never get out of the "make it from scratch" trap
Yes, but there is a line to be drawn somewhere. Filesystems are sufficiently advanced such that there’s no meaningful gains to be had from manually allocating CHS for data, and they provide housekeeping. C abstracts a lot of architecture-specific information away, but still requires that you understand a modicum of memory management; if you understand it (and cache line access) well, you can get even more performance. Python abstracts that away as well, and gives you a huge standard library to accomplish many common tasks with ease.
You can quickly make a prototype in Python, but it won’t be as performant as C. You can spend time profiling it and moving computationally-heavy parts into C extensions (I do this for fun and learning), but you’ll likely spend more time and get worse results than if you just rewrote it.
Docker is an abstraction over already-existing technology like cgroups. It provides an easy-to-understand model, and a common language. This is quite valuable, but it does allow one to not know about what it’s hiding, which is problematic when troubleshooting – for example, naïvely assuming that querying /proc in a container shows the CPU resources allocated to the container, rather than the host.
That’s how I view abstractions. They can be incredibly useful, but they usually have trade-offs, and those should always be considered. Most importantly, you should at a minimum be aware of what you’re giving up by using them, even if you don’t fully understand it.
If we ignore 'buggy' part I think you projecting current good state back into not so good old times. I am pretty sure you will not replace radix sort that uses domain knowledge of reduced value range with qsort circa C++98.
Things became much better after relatively recent improvements in generalist sorting libraries: when pattern defeating quick sort variants became norm, when mostly sorted cases got covered ...
Tbh I do have a case from 2013-16 when I regret not doing it for one AAA project on ps4.
Well known data structures and algorithms are well know because they have been used more than three times. That said: if you are dealing with a situation where you have to implement them yourself, you may want to consider whether the rule of three applies. (Clearly this depends upon the situation.)
If the well known data structures and algorithms are not provide by your language get a better language.
There are exceptions. If you are implementing the language it is your job to write them. It is useful as a student to implement the basics from scratch. Your language may decide something is not allowed and thus not implement it (a doubly linked list is almost always a bad idea in the real world. Likewise you don't need to provide all the sorting algorithms)
I am wondering where the "generalize code after doing it for the 3rd time" rule of thumb comes from? I also subscribe to it, and read it somewhere 15/20 years ago.
But that's still twice the cost, for only a potential win. Better check first that it helps somebody write better/more correct/quicker code.
E.g. we wanted to change our text-screen video driver to allow a full-screen overlay (many years ago). The programmer assigned the task was changing the text-blit code to know about certain lines on the screen that were going to be the 'nested' window, which could be 'up' and masking a subset of the window below, or 'down' and the full screen should be displayed as normal.
He'd been hacking away, changing every case in the code to 'know about' the particular window that was planned. Pulling his hair out, getting nowhere.
I suggested a simple indirection buffer, where each line of a virtual display was pointed to indirectly. To put the subwindow 'up' or 'down' you just pointed those lines of the main screen to a buffer, and pointed the lines of the subwindow to the hardware store for video text.
All his changes folded into 'access display indirectly'. Trivial. Then the up/down code became a loop that changed a pointer and copied some text.
That was actually an abstraction, and actually much shorter/faster/simpler to understand and change.
Later we could change the window dimensions, have multiple windows etc with no change to the video driver.
While I generally agree with that, the problem is teams, mixes of skill levels, and future time constraints. There's no guarantee the person doing the second implementation knows about the first one, realizes they can be pulled into a better abstraction, or has the time to do it. On the flipside, it's possible the other person (or the first person with more experience) comes up with something better than the first possible version of abstraction. So it ends up being a trade-off you have to decide on at the start, rather than a simple rule, based on team experience and how big (or small [0]) the abstraction actually us.
imo more like the 50th time around, and by a computer scientist, and not on company time until after the abstraction POC is validated, for example both React and Angular were side projects before the firm decided to invest. Software development outcomes today are driven by: ignorance, narcissism, and self preservation
I saw this in my codebase firsthand. At first, we had to communicate with one device. Then two. For that second device, I subclassed the class that handled the communication with the first thing, and changed 3 methods in the subclass. Now it was possible to substitute that second device for the first, allowing a different product configuration. After a few years, we had to support a third device type, and now they wanted it to be possible to have more than one of them, and to mix and match them.
Supporting the third device was handed off to a junior dev. I pointed him at my subclass and said to just do that, we'd figure out the mixing and matching later. But he looked at my subclass like it was written in Greek. He ended up writing his own class that re-imagined the functionality of the superclass and supported the new device (but not the old ones). Integrating this new class into the rest of the codebase would've been nigh impossible, since he also re-implemented some message-handling code, but with only a subset of the original functionality, and what was there was incorrect.
His work came back to me, and I refactored that entire section of the code, and this is when the generalization occurred: Instead of a superclass, I took the stuff that had to be inherited and made that its own thing, having the same interface as before. The device communication part would be modeled as drivers, with a few simple functions that would perform the essential functions of the devices, implemented once per device type. I kept the junior dev's communication code for the new device, but deleted his attempt to re-imagine that superclass. Doing it this way also made it easy to mix and match the devices.
To put it in slightly simpler terms, abstractions are generally to separate the “what” from the “how”.
Functions are the fundamental mechanism for abstraction in computing, which demonstrate that very well, in their separation between interface and implementation. The function signature and associated interface contract represent the “what”, and the function’s implementation the “how”. If the effective (often non-explicit) interface contract relies on most aspects of the actual implementation, then the “what” is almost the same as the “how”, and there is little abstraction. The greater the difference between the “what” and the “how”, the more of an actual abstraction you have.
This relates to Ousterhout’s notion of “deep modules”, which are modules whose interface is much simpler than their implementation. In other words, the “what” is much simpler than the “how”, which makes for a good abstraction.
It’s true that often one has to first implement the “how”, possibly multiple times, to get a good notion of which aspects are also still important to the “what”, and which aren’t.
Note also that generalizations go two ways: The caller needs less knowledge about the implementation, but it also means that the caller can rely less on the properties of a concrete implementation. This is again well reflected in function types. A (pure) function g: A –> B is a generalization of a function f: C –> D only if A is a subtype (specialization) of C and B is a supertype (generalization) of D. Here D could expose properties of the function’s implementation (f) that g wants to hide, or abstract from.
IMO, I dislike that kind of mixing several concepts into the same name that software engineering is full of.
Abstraction means removing details from the thing you present. Generalization means making the same representation valid for several different things.
Are abstractions that don't generalize valuable? Well, maybe there is something better to be found, but those are the bread-and-butter of software engineering; they are what everybody spends almost all of their time writing.
Are generalizations that don't abstract valuable (or even possible)? Well, not if they don't abstract at all, but there are plenty of valuable generalizations that abstract very little. Hell, we have all those standardizing organizations that do nothing more than creating those.
Are the best interfaces the ones that achieve most of both of those? Honestly, I have no idea, there are other goals and if you optimize to extreme levels, they start to become contradictory.
I would warn against conflating the concept of an interface with an abstraction just as much as I would against conflating generalizations and abstractions.
An interface often accompanies an abstraction, sometimes even represents one. But if we get down to definitions, an interface is merely something that you interact with: a function, a class, a network endpoint etc. If you write in machine code, you might not think that you are working with any kind of an interface, and certainly it's not a high level one, but you are interfacing with the machine's native instruction set. You could then argue that the instruction is an abstraction that hides what the machine is capable of. But now we're splitting hairs and about to ask silly philosophical questions like whether a car's steering wheel qualifies as an abstraction that exists to hide the axles. I would argue not. The interface's primary responsibility is to provide a mechanism of interaction, rather than to reduce complexity (though a good interface is simple and intuitive).
Both you and the author of the article posit a similar definition of 'abstraction'. From the article:
> An abstraction is only as good as its ability to hide the complexity of what lies underneath.
And from your comment:
> Abstraction means removing details from the thing you present.
I would actually argue that both, while very close, are missing the mark.
An abstraction exists to reduce a concept down to its essentials.
This doesn't necessarily contradict the definitions offered, but I think there is a nuance here that, if missed, causes the offered definitions to become useless.
The nuance is in deciding what is essential or necessary. If, definitionally, you choose to dispense with acknowledging the essential... well then you get the problems that the author is writing about. You get shit abstractions because no one bothered to think in terms of what they INCLUDING rather than DISPENSING with.
Yes, obviously, we abstract in an attempt to simplify. But that simplification needs to come from a positive rather than a negative.
In other words: What is the thing for? What does it do?
"Hides complexity" is the shittiest answer that anyone could ever offer as a response when faced with a given problem. First, what is the complexity that we are trying to reduce? Secondly, why are we trying to reduce it? Thirdly, when we have achieved our primary goal of reducing the complexity, then what does the thing look like? What value does it offer over interfacing directly with the "thing" being abstracted?
Abstractions in computer science are extremely valuable. So valuable, I would offer, that any time we hear engineers decry abstractions or point to abstractions as the root of all evil we ought to ask a very pressing question: "who hurt you?"
A good engineering solution is a simple one, and an abstraction is intended to be a simpification mechanism. But it doesn't have to necessarily simplify the intuitive understanding of the problem at hand. This is a good goal if you can achieve it, don't get me wrong. But that abstraction might exist so you can swap vendors in the future if that's a business priority. Or because you've identified some other element / component in your system that will be difficult to change later. So you stick it behind an abstraction and "Program to Interfaces" rather than "implementations" so that you can simplify the process of change down the road, even if it comes at an more immediate cost of making the code a bit less intuitive today.
Everything in software is tradeoffs. And a good abstraction exists so that we can focus on requirements and programming to those requirements. This is a smiplification when done properly.But the focus ought to be on defining those requirements, of asking "what should simple look like?"
I agree with what you're saying, and I would phrase it as "write the abstractions that reflect the essential complexity". The whole program should minimally reflect the essential complexity of the problem. Of course actually doing that isn't easy, but the result is obviously a simple solution for a given problem. It becomes another challenge to maintain and refactor: the question of changing problem constraints and being able to minimally change a program to match.
Why are ADTs like stacks, queues, and hashmaps so popular? Why are languages like C or Forth so highly praised for a high ceiling for performance and efficiency? Because they are usually "about as good as it gets" to solve a problem, "what you would've more or less done anyways". Maybe on a GPU, a language like C isn't quite fit, because the problem has changed. Make tools (e.g. CUDA) that reflect that distinct complexity.
The best abstractions I've seen have always come into being only after a significant amount of particularized code had already been written. It is only then that you can identify the actual common properties and patterns of use.
Early Ruby On Rails comes to mind as a great generalization of web applications in the era of Web 2.0.
Well said! I have met a few of these codesmells where the actual functioning is hidden behind a bewildering maze of facades, shims, proxies and whatnot.
I guess some has had an irresistible itch to use as many patterns from the GoF book as possible.
To me it sounds like what I use Ruby to get away from.
It's rare to need facades, proxies, and shims in a dynamically typed language where the caller doesn't need to care about the type of the object they call.
In fact, most of the Gang of Four design patterns either make no sense in Ruby or are reduced to next to nothing.
Great point, and agree that generalization makes for the clearest wins from abstraction.
But there are also cases where there is no generalization, but the encapsulation / detail hiding is worthwhile. If you have a big function and in the middle of it you need to sort some numbers, you would probably implement a Sort routine to make the control flow much easier to understand - even if you only use the function once (let’s pretend there’s no sort functionality in standard library).
Curious if others agree, and what heuristics you use to decide when implementation encapsulation is worthwhile.
Generalisation is definitely a good approach, but it’s not the only one. Another is “conceptualisation” or reducing repetition: if you find that large numbers of functions are taking the same three parameters, especially if they’re using them in similar ways, that’s a good sign there’s a concept you can introduce that makes those observations explicit.
"10" is the representation of a data; "0xA" is another representation of the same data.
Not being able to touch something doesn't make it an abstraction. Light is not an abstraction, a contract is not an abstraction. 10 is not an abstraction, it is an ordinal [1].
"Isomorphism" is an abstraction. It doesn't name a particular data or value, but a class of functions that share common properties. A function template or functions written in a dynamically typed language can describe a particular group of isomorphism.
> Light is not an abstraction, a contract is not an abstraction. 10 is not an abstraction, it is an ordinal [1].
It seems to me that all of those things very much are abstractions. They are not the utmost level of abstraction, but they are abstractions!
(Actually "a contract," which at first I thought was the clearest win, I'm now not sure about. On reflection, it seems like a concretization, turning abstract ideas of trust and reliability into concrete conditions under which the contract has or has not been met.)
Unfortunately, I did. It is an attempt to approach complexity, cognitive load and high entropy in code, jumping to conclusions prematurely while suggesting a solution that is worse than the problem.
I would really like to convince you that you are missing something very important. Please try to make sense of the article, reading it again a trying to find cases where it makes sense for you.
I wish articles like this had more examples in them. In between “this thin wrapper adds no value but a lot of complexity”, and “this thin wrapper clarified the interface and demonstrably saved loads of work last time requirements changed” is an awful lot of grey area and nuance.
I did like the advice that if you peak under the abstraction a lot, it’s probably a bad one, tho even this I feel could use some nuance. I think if you need to change things in lots of places that’s a sign of a bad abstraction. If there is some tricky bit of complexity with changing requirements, you might find yourself “peeking under the hood” a lot. How could it be otherwise? But if you find yourself only debugging the one piece of code that handles the trickiness, and building up an isolated test for that bit of code, well, that sounds like you built a wonderful abstraction despite it being peaked at quite a bit.
The article did start off giving TCP as a good abstraction but then didn't follow up with examples of bad abstractions.
Dynamic typing is an example of an indirection masquerading as an abstraction. You end up carrying around an object and occasionally asking it whether it's an int64_t or a banana. You maybe think your type luggage will take you on exotic vacations when really in fact you take it on exotic vacations.
To me, it ties in with John Ousterhout's concept of "deep, small interfaces"
TCP is a good abstraction because it's essentially 4 operations (connect, disconnect, send, receive), but there's a lot going on inside to make these operations work. So are TLS, filesystems, optimizing compilers and JITs, modern CPUs, React (or rather the concept of "reactive UI" in general), autograd and so on.
Articles like this are a dime a dozen. Literally, there are 1000s of articles that all say the exact same thing using way too many words: "Bad abstractions are bad, good abstractions are good".
I second this, such posts are very generic, they are hard to disagree with, but also to agree with empathically as there are no clear examples of what is too much.
As someone who uses lots of layers and dependency injection I would like to be poked on where is that too much abstraction but I end up being no wiser.
Relying on mere “taste” is bad engineering. Engineers do need experience to make good decisions, yes. But surely we are able to come up with objective criteria of what makes a good abstraction vs. a bad abstraction. There will be trade-offs, as depending on context, some criteria will be more important than other (opposing) criteria. These are sometimes called “forces”. Experience is what leads an engineer in assessing and weighing the different present forces in the concrete situation.
That’s seems like it should be true, and it would be great if it was.
But in my many years of experience working with Jr engineers, I have found no substitute other then practice guided by someone more Sr (who has good taste).
There are just too many different situations and edge cases. Everything is situational. You can come up with lists of factors to consider (better versions of this post often have them), but no real firm rules.
I wouldn’t call that “taste”. It’s not a matter of taste which solution is better. If different engineers disagree about which solution to choose, then it’s fundamentally a different assessment of the relevant factors, and not about taste. Or at least, it shouldn’t be the latter.
I don’t know. We could look for some other word to encode “often sub-conscious though sometimes explicit heuristics developed by long periods of experiencing the consequences of specific trade-offs” but “taste” seems like a pretty good one because it’s quite intuitive.
There often - usually? - are more than one good solution and more than one path to success, and I don’t find calling different good engineers making different choices primarily because of their past experiences an egregious misuse of language.
I think you are going after something that is more an element of craftsmanship than engineering, and I agree it is a big part of real world software development. And, not everyone practices it the same way! It's more of a gestalt perception and thinking process, and that instinctual aspect is colored by culture and aesthetics.
In my career, I've always felt uncomfortable with people conflating software development with engineering. I think software has other humans as the audience more so than traditional engineered products. Partly this may be the complexity of software systems, but partly it is how software gets modified and reused. There isn't the same distinction between the design and the product as in other domains.
Other domains have instances of a design and often the design is tweaked and customized for each instance for larger, complex products. And, there is a limited service life during which that instance undergoes maintenance, possible refurbishing, etc. Software gets reused and reformed in ways that would make traditional engineers panic at all the uncertainties. E.g. they would rather scrap and rebuild, and rely on specialists to figure out how to safely recycle basic materials. They don't just add more and more complexity to an old building, bridge, airplane, etc.
Yes, this is what I mentioned in my original comment about experience being needed to weigh the trade-offs. That doesn’t mean that we can’t very concretely speak about the objective factors in play for any given decision. We can objectively say that x, y, z are good about this abstraction and a, b, c are bad, and then discuss which might outweigh the other in the specific context.
Needing experience to regularly make good decisions doesn’t mean that an article explaining the important factors in deciding about an abstraction is useless.
If we used the word "judgement", would that be a better option?
It seems that pretty much anyone can write code (even AI), but ultimately in software development, we get paid for judgement.
Taste amounts to a well trained neural net in the engineers skull. It should not be belittled. Articles like this attempt to describe taste systematically, which is worth attempting but impossible
Maybe not, but you can still move the needle one way or another based on reading an article. For those readers who recognize themselves as erring on the side of adding too many abstractions, they might move the needle a bit towards the other side.
try using LangChain and you'll get countless examples of bad abstractions
started working with it this week for a new project
gosh, it's so painful and unintuitive... I find myself digging deep into their code multiple times a day to understand how I'm supposed to use their interfaces
"I wish articles like this had more examples in them."
There is a class of things that don't fit in blogs very well, because any example that fits in a blog must be broken some other way to fit into a blog, and then you just get a whole bunch of comments about how the example isn't right because of this and that and the other thing.
It's also a problem because the utility of an abstraction depends on the context. Let me give an example. Let us suppose you have some bespoke appliance and you need to provide the ability for your customer to back things up off of it.
You can write a glorious backup framework capable of backing up multiple different kinds of things. It enforces validity checks, slots everything nicely into a .zip file, handles streaming out the backup so you don't have to generate everything on disk, has metadata for independent versions for all the components and the ability to declare how to "upgrade" old components (and maybe even downgrade them), support for independent testing of each component, and has every other bell and whistle you can think of. It's based on inheritance OO and so you subclass a template class to fill out the individual bit and it comes with a hierarchy pre-built for things like "execute this program and take the output as backup" and an entire branch for SQL stuff, and so on.
Is this a good abstraction?
To which the answer is, insufficient information.
If the appliance has two things to backup, like, a small SQL database and a few dozen kilobytes of some other files, such that the streaming is never useful because it never exceeds a couple of megabytes, this is an atrocious backup abstraction. If you have good reason to believe it's not likely to ever be much more than that, just write straight-line code that says what to do and does it. Jamming that into the aforementioned abstraction is a terrible thing, turning straight code into a maze of indirection and implicit resolution and a whole bunch of code that nobody is going to want to learn about or touch.
On the other hand, if you've got a dozen things to backup, and every few months another one is added, sometimes one is removed, you have meaningful version revs on the components, you're backing up a quantity of data that perhaps isn't practical to have entirely in memory or entirely on disk before shipping it out, if you're using all that capability... then it's a fantastic abstraction. Technically, it's still a lot of indirection and implicit resolution, but now, compared to "straight line" code that tries to do all of this in a hypothetical big pile of spaghetti, with redundancies, idiosyncracies of various implementations, etc., it's a huge net gain.
I don't know that there's a lot of abstractions in the world that are simply bad. Yeah, some, because not everything is good. But I think they are greatly outnumbered by places where people use rather powerful, massive abstractions meant to do dozens or hundreds of things, for two things. Or one thing. Or in the worst case, for no things at all, simply because it's "best practices" to put this particular abstraction in, or it came with the skeleton and was never removed, or something.
> Abstractions are also the enemy of simplicity. Each new abstraction is supposed to make things simpler—that’s the promise, right?
Not exactly, no. The purpose of abstraction is to hide implementation detail, and thereby insulate one part of the codebase/application/system from variations in another. Graphics APIs for example - yes your code may be simpler for not having to deal with the register-level minutiae of pushing individual triangles, but the core benefit is that the same code should work on multiple different hardware devices.
Good abstractions break a codebase up into compartments - if you drop a grenade in one (change the requirements for example), then the others are unaffected and the remedial work required is much less.
“The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.” — Edsger Dijkstra
But sometimes a new semantic level isn’t needed. Abstraction gets so much press when you might just need some good ol’ fashioned information hiding and separation of concerns.
This is such a great quote, and helps explain what is a good abstraction.
Because CRDTs have been in the zeitgeist a lot lately, I want to pick them as an example of a "good" abstraction.
CRDTs have mathematical properties which can be described and understood independently of a specific implementation. And importantly, you can judge whether an implementation is correct with reference to these abstract rules.
This means that when using a CRDT, you largely can treat it as a reliably-solved problem. Once you understand the concepts, and work out how to use the library you've picked, you don't have to think about the details. Though that doesn't mean sometimes the behaviour can be surprising:
TCP and HTTP are great examples too, though interestingly I don't know if they rely on mathematical definitions so much as just being extremely widespread to the point that reliable implementations are available anywhere you care to write code.
I like this article which also leans on the Dijkstra quote:
CRDTs are also an excellent example because of how their supporting infrastructure is impacted by their design, namely that Postgres’ method of dealing with updates makes for massive write amplification.
I wrote this [0] previously, but it still applies. IMO, as a dev, there are times where you really do need to think somewhat deeply about infrastructure choices. Unfortunately, knowing when you need to care practically requires you to already understand it.
It seems like almost everybody in these comments as well as OP are failing to differentiate between the narrow computer science / Dijkstra definition of abstraction versus how it's invoked colloquially, including in conversations of computer science such as here.
In that sense, we can correctly say that even encapsulation etc is a form of (colloquial) abstraction, but not Dijkstra's use of the word abstraction.
Maybe that’s the problem. Everyone knows that abstractions are desirable, and encapsulates as much as they can, and ends up with redirection hell without any of the advantages of true abstractions. If people understood what the word actually means, we might not be in this mess.
On practice, there exist platonic abstractions that hide mechanisms and can be this way, but you also need to abstract parts of your requirements. And you will never be able to achieve that while abstracting your requirements.
Dijkstra was an academic after all, and academics usually don't care about complex requirements.
While this is how the term is often used, I think it's cavalier use of language and confuses abstraction for modularity, and this linguistic confusion is one of the reasons a lot of programmers write bad "abstractions".
Organizing your code into components that are as independent as possible is a good practice and is the pursuit of modularity. A proper abstraction on the other hand, is a generalization that simplifies a conceptual layer in your code base.
Abstraction often enables greater modularity as a consequence, but they are not the same thing. For example, in the problem of text editing, people eventually realized that the manipulation of text is typically line oriented. Thinking of a text file as a collection of lines may seem like an obvious and modest abstraction, but it works well. This abstraction, in turn, leads to other couplings (e.g. line oriented motion is highly dependent on the line abstraction), but it also leads to potential modularity (e.g. printer code may no longer need to understand exactly how a display renders each character of text in a grid, instead, it too can work on "lines"). Good abstractions support modularity to the extent that they establish a shared domain of objects to communicate about and across systems, but they do not necessarily produce modularity in themselves.
Yes, abstraction and modularity are coincident but distinct topics, and my last paragraph implies that I am equating those two things. Good point.
I disagree that considering a text file as a set of lines really qualifies as an abstraction; it's just a different representation of the data - you could instead choose to use a list of words or a tree or whatever. An abstraction is a general interface to a concrete thing that allows you to substitute different but similar concrete things without changing the consumer of those things. The Linux filesystem is a great example - I can 'cat' basically anything that has a path, and some driver will pull data from the associated endpoint and display the data on screen. "File" is the abstraction, and consumers of files need not care about the specifics of communicating with the underlying devices. Such a system is also modular, but it's hard not to be when your abstraction is well conceived.
Perhaps we're saying the same thing with different words?
When you write "an abstraction is a general interface to a concrete thing that allows you to substitute different but similar concrete things without changing the consumer of those things", I feel you are actually describing a module created by making use of abstractions.
Abstractions almost always represent data differently than as it is in its most concrete form (e.g. lists and trees are abstractions both often built of vectors and pointers), but that is not the purpose of abstraction. To continue with the original example, the representation of text as a sequence of lines is a sort of abstraction, but rarely the most useful one: words, sentences, paragraphs and/or syntax trees are often what you want, and they are useful because they give us a representation of the raw text that is aligned with its purpose, not because we can implement them in multiple ways.
Furthermore, while we want to do some information hiding when implementing these abstractions (think particularly of syntax trees), that is not the purpose of having these abstractions - what we are hiding is the infrastructure to make it work, not any information conveyed by the text itself.
I don't like the mental image of layers. It's not wrong, but it's terribly inclusive of everything that is bad about bad abstractions. Even the worst abstractions will often look kind of nice from the layers angle.
I prefer the concept of orthogonality: you have an entire plane where your domain requirements can move in, but the way you make persistence reliable is completely orthogonal to that plane. It should be implemented in a way that does not change when, say, your customer account becomes a composite of of multiple identities. And neither should the way you organize UI components change, or your transportation tooling change. That entire transportation layer stack, from ethernet all the way up to your domain model generator of choice.
Did I say layers? Yeah, I did. You'll often see the orthogonal aspects themselves consisting of layers. But that's an implementation detail and those layers tend to be more pragmatic than dogmatic, as in the way TCP goes out of its way to avoid hiding any of the addressing of the underlying IP. TCP absolutely is a layer, but it has zero ambition to perhaps run on top of something other than IP, or hide differences between IPv4 and IPv6 from higher layers. It focuses on one thing, implementing streams on top of packets, again a problem nicely orthogonal to the problem of routing those packets (what IP does) or anything built on top of those streams.
To add on to your great comment, I've read a few papers that justify your remarks, but some have paywalls and I'll try to condense them anyways.
Parnas' criteria for when to split code into modules[0] is very compelling: modules should enforce information hiding, where the information of each module is how a key decision is implemented. Key decisions should be orthogonal: e.g., encryption shouldn't inform filesystem access and vice versa. This fits in nicely with practices for determining solutions for a given problem: nail down key decisions (or uncertainties around them) early on and encode them through modules and interfaces.
A logical application of Parnas' module criteria is, naturally, in network stacks. According to [1], although coarse program flow generally follows the OSI layers, the fine control flow may hop around quite a bit. For some time people tried to keep the OSI layers as inspiration and use hacks like upcalls to selectively violate layering, which is a sure sign that the layers are unhelpful. Instead, modules should reflect functionality like encryption, NIC interfacing, and routing. Aim for a flatter, horizontal architecture like a B-tree, not a contrived vertical tower with hacks to cross layers "incorrectly". There will be hierachies and layers, but they should be obviously necessary now. Layering is not the end or even the means but merely how things are.
A program's capability (functionality) is determined by the capabilities of its dependencies, and splitting up functionalities well is important. I use this language to relate it to capability-based security[2]. After all, "security" is basically just a stricter form of program "correctness", and just as a terminal emulator's input text handling capability shouldn't have much to do with its window event handling capability because malicious input shouldn't cause dangerous graphics bugs, there isn't much logical reason to have overlap regardless. They govern separate devices, protocols, and purposes. Capabilities along the lines of capability-based security reify the interfaces and provide strict enforcement.
Lastly, flexibility around protocols should be added or discard as given by the precise problem at hand. Don't preemptively choose some extreme, but instead be informed by what is actually desired. It will yield substantial implementation complexity and performance detriments[3] to over-generalize, and of course over-specializing is just inadequate.
> The purpose of abstraction is to hide implementation detail
Technically, that's encapsulation, though the sentiment is close, I think.
I rather view it as a matter of semantics. At one low level, you have operations that deal with some concrete interface or API, etc. You bundle those operations up behind an abstraction, providing methods whose names involve your application domain. Perhaps they are still at a technical level, and you bundle those up behind another abstraction, whose method names involve your business domain.
Yes, the lower level details are hidden from the higher levels, but the hiding is not the point. The point is to be able to write code that readily corresponds to the problem you are trying to solve.
> In object-oriented programming languages, and other related fields, encapsulation refers to one of two related but distinct notions, and sometimes to the combination thereof:[5][6]
> - A language mechanism for restricting direct access to some of the object's components.[7][8]
> - A language construct that facilitates the bundling of data with the methods (or other functions) operating on those data
The hiding is absolutely the point of abstraction, in the sense that caller can manipulate the subsystem without knowing about (important) inner details of it. As I said, graphics drivers are a great example - I just want to put a triangle on the screen, I don't want to care about the registers I have to set on an NVidia card vs those on an AMD card, or what their memory maps look like, and I don't want to have to rewrite my code when they release a new generation of hardware. Drawing a triangle is an abstraction, hiding away the details of how a specific graphics card achieves that.
Think about what the word 'abstract' means - the idea of a 'human' is an abstraction for bundles of molecules conforming to a general pattern with broadly similar qualities but almost infinite variability. The word conveniently hides a wealth of detail that we usually don't need to think about; if I tell you there are three humans in a room, you can make use of that information without knowing anything about their exact physical attributes or who they are.
I would refer to what you're describing as 'modelling' - these are related topics but I don't see them as the same thing.
>> but the core benefit is that the same code should work on multiple different hardware devices.
I came here to say this. Attractions act as a bridge between things which allow those things to change independently.
Using an ORM allows my program to easily work against multiple sql databases. Using a compiler allows me to target different hardware. Using standard protocols allows me to communicate with different programs. Using libraries to do say email hides me from those protocols and allows me to adapt to service providers Using APIs not protocols.
In other words the abstraction is designed to hide a layer (which can change) from a program not interested in that level of change.
The key is to stop abstracting when the program cares. By all means encapsulate rules and processes, but they're a direct implementation of those rules and processes.
One can argue my "calculateLeaveForEmployee" function is an "abstraction" - but that would be a misnomer. Since there's only one set of rules in play (the set that matters now) its an implementation. An abstraction supports (at least) two things at the same time.
> Using an ORM allows my program to easily work against multiple sql databases.
A curious example, since most developers who've worked on any project with significant amount of data in a database would likely disagree.
IMO, ORMs mostly allow using programming-language-of-choice as a syntax for relational queries instead of constructing it by hand on top of serialization and deserialization of objects into rows and vice versa.
Indeed, given that most sql dialects have subtle differences that make them noncompatible with one another, and most ORMs have support for dipping into raw sql, sufficiently large systems tend to end up coupled with a particular database anyway. That's not to mention that the BAs will want raw sql access for report writing and switching systems breaks all of their scripts too.
Yes, ORMs came to mind for me as an example of indirection without abstraction. If you accept OP’s litmus test of “how often do I have to peek under the hood” I think ORMs generally don’t score particularly well.
If enabling the ablity to think of the database as the rich native data structures provided by the host programming language, and not the tables (rows and columns) that the underlying database service wants to think of the data as is not an abstraction, what is?
Fair, it’s probably incorrect to say there’s “no abstraction” in an ORM, maybe more precise to say it’s quite a leaky one. I’ve never seen an ORM used in the wild where you could actually avoid thinking about tables and columns and rows pretty frequently. But that’s admittedly armchair anecdotal.
In the small-to-medium scale cases I’ve seen, the main re-use you get for a particular mapping is when an object is used as a child node elsewhere. If it’s a one-off you’re essentially writing a bespoke shorthand wrapper for some custom SQL. Which may be a convenient shortcut or prettifier, but isn’t really functioning as an abstraction. Net net it seems like more overhead. Or like the real benefit is constraining database design to closely align with object design (which could be valuable but isn’t a function of abstraction).
This is all assuming code ownership across the stack. With a separate team managing data representation and handing reliable object contracts “over the wall” I could imagine a situation where the consumer gets to think purely in objects. I just haven’t observed that in practice. E.g. What if the contact has two address records? What if the number of purchase records exceeds the maximum result size? If things like that come up regularly enough, it’s worse when you’re waiting on a separate team to fix the leaks.
> I’ve never seen an ORM used in the wild where you could actually avoid thinking about tables and columns and rows pretty frequently.
There is a compelling argument that absent the n+1 problem, it could be a leak-free abstraction. And, in practice, SQLite doesn't suffer from the n+1 problem... But that is definitely true in a lot of cases, particularly where the n+1 problem is unavoidable.
> If it’s a one-off you’re essentially writing a bespoke shorthand wrapper for some custom SQL.
While the original comment also conflated query building and ORM, I am not sure that we should. The intersection of query building and ORM was historically referred to as the active record pattern. I suspect the ActiveRecord library, from Rails, calling itself an ORM is where things started to become confused. Regardless, whatever you want to call the data mapping and query building, they are undeniably distinct operations.
It may be fair to say that query builders are no more than an indirection. That said, I posit that they only exist as an abstraction to model composability on top of SQL. If SQL naturally composed through basic string concatenation, I expect the use of query builders would quickly disappear.
as the base of the query, with all of these being concatenated together.
But instead of this, an ORM usually provides you with a syntax (that will pass syntax checks and have highlighting) that matches the language, which is all nice and good because dealing with arbitrary strings does suck.
I've seen ORMs being used for query composition way before Rails even existed: I believe Hibernate had HQL with their first release in 2001, just like SQLObject did in 2002. I am sure neither of those "invented" the pattern.
Note that fetching objects using an ORM library, filtering and such is what I also consider query composition (it just happens behind the scenes).
> But SQL is "naturally composable" using string concatenation.
It is not. Not even in a simple case. Consider:
base = "SELECT * FROM table LIMIT 10"
cond1 = "WHERE foo = 1"
cond2 = "WHERE bar = 2"
base + cond1 + cond2 or any similar combination will not produce a valid query. It could if SQL had some thought put into it. There are many languages that have no problem with such things. But that irrational fear of moving beyond the 1970s when it comes to SQL...
The only realistic way to assemble queries is to prepare an AST-like structure to figure out where the pieces fit, and then write that out to a final query string. In practice, that means either first parsing the partial queries into that structure (hard) or providing a somewhat SQL looking API in the host language that builds the structure (easy). Unsurprisingly, most people choose what is easy.
> I am sure neither of those "invented" the pattern.
None of these invented the pattern. But the invention point is irrelevant anyway. You must have misread something?
ORM, as the name suggests, is an "adaptation" layer ("mapping" or "mapper").
It does not really abstract away relational data, it instead maps it to object/class hierarchy. And every single ORM I've used also provides an interface to map function/method/expression syntax to SQL queries to get back those rows (adapted into objects).
Now, in a very broad sense, mapping would also be an abstraction — just like an "indirection" would be — but considering the narrower definition from the OP (where indirection is not an abstraction), I think it's fair to say that ORM is also not an abstraction.
It's interesting that some modern ORMs are dropping trying to abstract out the dB engine. Specifically thinking of Drizzle for js. And rather just focusing on the programming interface
>> How many times have you required that your program runs against different sql databases without modification?
Our main commercial product currently supports 2 database engines, and we'll be offering a 3rd next year. For enterprise offerings it's pretty common for the client to prefer, or outright specify, the engine.
Commodotizing your complementary technology is a good way to not become dependent on any specific database, and hence can pivot quickly when required.
Fortunately I don't have a plane, so I don't have code to fly in any atmosphere.
The rest of us go with Postgres, or SQL server, or Oracle, but definitely don't have to prepare our systems to run in PostgreSQL on Mondays, on SQL server on Tuesdays and so on.
In short: good abstractions simplify by centralizing operational logic. But it's not until a certain scale where that option is more efficient from bespoke implementations.
A heuristic we use at work is to not introduce an abstraction layer until there are at least two different implementations required.
That is if you think you'll probably need multiple implications, delay introducing an abstraction until you actually do.
Also, there are different ways of providing abstraction.
Perhaps you don't need to abstract the entire implementation but, as an example, rather change one parameter from passing a value to passing a function returning a value.
I got a piece of advice writing UI code a long time ago: Don't marry your display code to your business logic.
I'd like to say this has served me well. It's the reason I never went for JSX or other frameworks that put logical code into templates or things like that. That is one abstraction I found unhelpful.
However, I've come around to not taking that advice as literally as I used to. Looking back over 25 years of code, I can see a lot of times I tried to abstract away display code in ways that made it exceedingly difficult to detect why it only failed on certain pieces of data. Sometimes this was sheep shaving tightly bound code into generic routines, and sometimes it was planned that way. This is another type of abstraction that adds cognitive load: One where instead of writing wrappers for a specific use case, you try to generalize everything you write to account for all possible use cases in advance.
There's some sort of balance that has to be struck between these two poles. The older I get, though, the more I suspect that whatever balance I strike today I'll find unsatisfactory if I have to revisit the code in ten years.
I learned that lesson building an utility with JavaFX. I've done a few years with React and the usual pattern was to move almost everything out of the components. Unless it's an event handler, a data transformer for the view, or something that manipulates the view, it has no business belonging to this layer.
I don't try to generalize it, I just try to make the separation clear using functions/methods/classes. Instead of having the post button's handler directly send the request, I create a function inside the `api` module that does it. It does have the effect on putting names on code patterns inside the project.
When you say "components" do you mean that in the mixed React sense where a component could contain HTML? In my own usual cases, I call the Javascript a component, and the HTML a template. I usually take a handlebars approach on the template content and something like "data-role" to identify the template tags to the JS, and beyond that don't mix them. However, my client-facing JS components themselves are totally bound to the templates they load up - they expect the presence of certain fields to work with. I'm talking more about not mixing any business logic into those JS components: Let's say, in a form component, not anticipating that a dropdown menu will have any particular shape or size of dropdown item, which means those items need to be specified separately. This leads to JS components relying on lots of other components, when sometimes you just need one type of dropdown item for a particular component, and having dropdown items be a 20-headed beast makes everything upstream need to define them first.
Sometimes you just need a form to do what it says on the label.
I don't think I'm conflating things. Any time you insert sugar into your HTML that makes your end user see output that's been inserted by how your framework interprets that sugar, you may be binding your business logic to your display code. Possibly in subtle ways you don't realize until later. A common case is rounding decimals to a user that aren't rounded in the system.
Letting any control flow into your HTMX/JSX templates is begging for trouble. It boggles the mind that people abandoned PHP - where at least the output was static and checkable - for something exactly like mixed HTML/code where some processing was done on the front end and everything was supposed to be dehydrated. Only to pivot again to hydration on the back-end.
JSX and React came on the scene 10 years after I first realized using the client for anything logical was an anti-pattern. Back then, I remember people would write things like:
and put a bunch of validation on the client side, too. Clients are dumb terminals. That's it. I'm not confusing the use of logic-in-JSX with business-logic-in-display. One only has to look at every basic JSX example on the web that's taught to newbies to see a million ways it can go wrong.
As someone who has little experience in this topic but has come to a similar conclusion, I think the main downside of this strict separation you're recommending is performance/efficiency. Have you noticed that to be a problem in practice? It's not always clear whether the simplest solution can actually be feasible, or perhaps that is just a reflection of still untapped understanding of the problem domain.
Generally there isn't a huge tradeoff in business software. In games where every CPU cycle counts it's another story. And yes, I have been guilty of writing display code divorced from game logic that was way beyond the scope of what needed to be actually displayed on the screen for a particular situation, and having that over-generalization lead to unacceptable performance drops. So you make a good point.
I think your principle is good, but this is going too far to throw out react.
You really just want to split those things out of the UI component and keep minimal code in the component file. But having it be code free is too much for real client side apps.
Modern apps have very interactive UI that needs code. Things like animation management(stop/start/cancel) etc… have subtle interactions. Canvas zoom and pan, canvas render layouts that relate to UI, etc… lots of timing and control code that is app state dependent and transient and belongs in components.
I apply the simple principle and move any app state management and business logic out of the component UI code. But still lean into logical code in UI.
I started with a templating system that had a very limited logic and I'm still quite fond of this approach.
Basically arguments for a template had a form of a tree prepared by the controller function. The template could only display values from the tree (possibly processed by some library function, for example date formatter), hide or show fragments of html dependaning on the presence or absence of some value or branch in the tree, descend into a branch of a tree and iterate over an array from the tree, while descending into one iteration at a time. Also could include subtemplate feeding it a branch of the tree. This was enough to build any UI out of components and kept their html simple, separate and 100% html. Even the template logic had a form of html comments. One could open such template in any html editor including visual ones. It was all before advent of client side frameworks.
You could mimic this in React by preparing all data in render method as a jsonlike tree before outputting any JSX tag and limit yourself inside JSX part to just if and map(it =>{}) and single value {}
Yeah. I did this too. Mine's driven off a database with a structure of
1. pages, each of which may or may not have a parent pageID; the entire tree is read and parsed to generate the menu system and links/sub-links
2. modules which reference HTML templates to load through a parser that replaces areas with {{handlebar}} variables written by the clients through their own API
3. something like page_modules which locate the modules on the page,
4. a renderer which uses the above stuff to make the menu system, figure out what URL people are trying to get to, load the page, load the modules in the page, and fill in the crap the client wrote inside the {{handlebars}}
This has worked so well for so long that I can basically spin up a website in 15 minutes for a new client, let them fill in anything I want them to by themselves, throw some art on it and call it a day.
It worked so well that I ended up writing a newer version that uses a bunch of Javascript to accomplish basically the same thing with smoother transitions in a single-page app that doesn't look or act like an SPA, but it was basically pointless.
TCP is great. Long chains of one-line functions that just permute the arguments really suck. These both get called abstraction, and yet they're quite different.
But then you hear people describe abstraction ahem abstractly. "Abstraction lets you think at a higher level," "abstraction hides implementation detail," and it's clear that neither of those things are really abstractions.
As the OP mentions, we have a great term for those long chains of one-line functions: indirection. But what is TCP? TCP is a protocol. It is not just giving a higher-level way to think about the levels underneath it in the 7-layer networking model. It is not just something that hides the implementations of the IP or Ethernet protocols. It is its own implementation of a new thing. TCP has its own interface and its own promises made to consumers. It is implemented using lower-level protocols, yes, but it adds something that was fundamentally not there before.
I think things like TCP, the idea of a file, and the idea of a thread are best put into another category. They are not simply higher level lenses to the network, the hard drive, or the preemptive interrupt feature of a processor. They are concepts, as described in Daniel Jackson's book "The Essence of Software," by far the best software design book I've read.
There is something else that does match the way people talk about abstraction. When you say "This function changes this library from the uninitialized state to the initialized state," you have collapsed the exponentially-large number of settings of bits it could actually be in down to two abstract states, "uninitialized" and "initialized," while claiming that this simpler description provides a useful model for describing the behavior of that and other functions. That's the thing that fulfills Dijkstra's famous edict about abstraction, that it "create[s] a new semantic level in which one can be absolutely precise." And it's not part of the code itself, but rather a tool that can be used to describe code.
It takes a lot more to explain true abstraction, but I've already written this up (cf.: https://news.ycombinator.com/item?id=30840873 ). And I encourage anyone who still wants to understand abstraction more deeply to go to the primary sources and try to understand abstract interpretation in program analysis or abstraction refinement in formal verification and program derivation.
Hey Jimmy, I've read your comment and also your article in the past with great interest. This topic is absolutely fascinating to me.
I just re-read your article but unfortunately I still struggle to really understand it. I believe you have a lot of experience in this, so I'd love to read a more dumbed down version of it with less math and references to PL concepts and more practical examples. Like, this piece of code does not contain an abstraction, because X, and this piece of code does, because Y.
I'll have to muse about what the more dumbed down version would look like (as this version is already quite dumbed down compared to the primary sources). It wouldn't be quite a matter of saying "This code contains an abstraction, this other code doesn't," because (and this is quite important) abstraction is a pattern imposed on code, and not part of the code itself.
We do have a document with a number of examples of true abstraction — written in English, rather than code, in accordance with the above. It's normally reserved for our paying students, but, if you E-mail me, I'll send it to you anyway — my contact information easy to find.
For example, in the "TV -> serial number" abstraction, if I were to define only one operation (checking whether two TV's are the same), would it make it a good abstraction, as now it is both sound and precise?
And what are the practical benefits of using this definition of abstraction? Even if I were to accept this definition, my colleagues might not necessarily do the same, nor would the general programming community
> if I were to define only one operation (checking whether two TV's are the same), would it make it a good abstraction, as now it is both sound and precise?
It would!
> And what are the practical benefits of using this definition of abstraction?
Uhhh...that it's actually a coherent definition, and it's hard to think or speak clearly without coherent definitions?
If you're talking about using it in communication, then yeah, if you can't educate your coworkers, you have to find a common language. They should understand all the words for things that aren't abstraction except maybe "concept," and when they use the word "abstraction," you'll have to deduce or ask which of the many things they may be referring to.
If you're talking about using it for programming: you kinda can't not use it. It is impossible to reason or write about code without employing abstraction somewhere. What you can do is get better about finding good abstractions, and more consistent about making behavior well defined on abstract states. If you're able to write in a comment "If this function returns success, then a table has been reserved for the requesting user in the requested time slot," and the data structures do not organize the information in that way, and yet you can comment this and other functions in terms of those concepts and have them behave predictably, then you are programming with true abstraction.
In this case, not programming with true abstraction would mean one of two things:
1. You instead write "If this function returns success, then a new entry has been created in the RESERVATIONS table that....", or
2. You have another function that says "Precondition: A table has been reserved for the user in this timeslot," and yet it doesn't work in all cases where the first function returns success
I think it's pretty clear that both ways to not use true abstraction make for a sadder programming life.
This is a great point. Most modern software is riddled with unnecessary complexity which adds mental load, forces you to learn new concepts that are equally complex or more complex than the logic which they claim to abstract away from.
I find myself saying this over and over again; if the abstraction does not bring the code closer to the business domain; if it does not make it easier for you to explain the code to a non-technical person, then it's a poor abstraction.
Inventing technical constructs which simply shift the focus away from other technical constructs adds no value at all. Usually such reframing of logic only serves the person who wrote 'the abstraction' to navigate their own biased mental models, it doesn't simplify the logic from the perspective of anyone else.
> Think of a thin wrapper over a function, one that adds no behavior but adds an extra layer to navigate. You've surely encountered these—classes, methods, or interfaces that merely pass data around, making the system more difficult to trace, debug, and understand. These aren't abstractions; they're just layers of indirection.
“No added behaviour” wrapper functions add a lot of value, when done right.
First off, they’re a good name away from separating what you’re doing from how you’re doing it.
Second, they’re often part of a set. E.g. using a vector for a stack, push(x) can be just a call to append(x), but pop() needs to both read and delete the end of the vector. Push in isolation looks like useless indirection, but push/pop as a pair are a useful abstraction.
A consequence of adding these two points together is that, if you have a good abstraction, and you have a good implementation that maps well to the abstraction, it looks like useless indirection.
Another consequence is that those pass-through wrapper functions tell you how I think the implementation maps to the domain logic. In the presence of a bug, it helps you determine whether I got the sequence of steps wrong, or got the implementation wrong for one of the steps.
Ultimately, the two aren’t completely independent —indirection is one of the tools we have available to us to build abstractions with. Yes, people misuse it, and abuse it, and we should be more careful with it in general. But it’s still a damned useful tool.
I have seen tons of ”abstractions” in recently created code bases from ”senior developers” which in actual fact is only titanic-grade mess of complicated ”indirection”. Many people nowadays are unfortunately not fit to work in software development.
> a bad one turns every small bug into an excavation.
I find that I need to debug my abstractions frequently, while I’m first writing my code, then I never need to dig into them, ever again, or they do their job, and let me deal with adding/removing functionality, in the future, while not touching most of the code.
That’s why I use them. That’s what they are supposed to do.
Because they are abstractions, this initial debugging is often a lot harder than it might be for “straight-through” code, but is made easier, because the code architecture is still fresh in my mind; where it would be quite challenging, coming at it without that knowledge.
If I decide it’s a “bad abstraction,” because of that initial debugging, and destroy or perforate it, then what happens after, is my own fault.
I’ve been using layers, modules, and abstractions, for decades.
Just today, I released an update to a shipping app, that adds some huge changes, while barely affecting the user experience (except maybe, making it better).
I had to spend a great deal of time testing (and addressing small issues, far above the abstractions), but implementing the major changes was insanely easy. I swapped out an entire server SDK for the “killer feature” of the app.
is not a nice one. When was the last time you had the data in a buffer, wanted to send it over to the peer at the other side, but didn't mind that it's not sent in it's entirety ? So you have to write a loop over it. Also, is the call blocking or not ? (well, you'll have to read the code that created it to know, so that's no fun neither).
However, it does prove the point the author is trying to make: good abstractions are hard to find!
Anyway, I tried to think of a better example of a good abstraction and found the "Sequence" that's available in plenty of programming languages. You don't need to now what the exact implementation is (is it a list, a tree, don't care!) to be able to use it. Other example I found were Monoid and Monad, but that's tied to the functional paradigm so you lose most of the audience.
Just thinking on my feet as to how I separate abstractions from indirections and it seems to me that there's a relatively decent rule of thumb to distinguish them: When layer A of code wraps layer B, then there are a few cases:
1) If A is functionally identical to B, then A is a layer of indirection
2) If A is functionally distinct from B, then A is likely an abstraction
3) If A is functionally distinct from B, but B must be considered when
handling A, then A is a leaky abstraction.
The idea is that we try to identify layers of indirection by the fact that they don't provide any functional "value".
Pretty easy to give generic advice without examples.
“Write more tests, but not too many”
“Use good abstractions where appropriate?”
“The next time you reach for an abstraction, ask yourself: Is this truly simplifying the system? Or is it just another layer of indirection?”
It’s easy to create a strawman here (the FactoryAdaptorMapper or whatever) but in reality this kind of generic advice doesn’t help anyone.
Of course people want to use good abstractions.
That’s not the problem.
The problem is being able to tell the difference between generic arbitrary advice (like this post) and how your specific code base needs to use abstractions.
…and bluntly, the only way to know, is to either a) get experience in the code base or b) read the code that others have left there before you.
If it’s a new project, and you’re not familiar with the domain you’ll do it wrong.
Every. Single. Time.
So, picking “good” abstractions is a fools game.
You’ll pick the wrong ones. You’ll have to refactor.
That’s the skill; the advice to take away; how to peel back the wrong abstraction and replace it with your next best guess at a good one. How to read what’s there and understand what the smart folk before did and why.
…so, I find this kind of article sort of arrogant.
Oh, you want to be a great programmer?
Just program good code.
Use good abstractions. Don’t leave any technical debt. Job done!
…a few concrete examples would go a long way here…
I can see the value of examples, but in this case I appreciate the post largely for its universality and lack of examples. On reading it, examples from past and present experience spring immediately to mind, and I'm tucking this away as a succinct description of the problem. Maybe I can share it with others when more concrete examples come up in future code review.
A principle takes skill the apply, but it's still worth stating and pondering.
> examples from past and present experience spring immediately to mind
Examples of what?
Picking the wrong abstraction? Regretting your mistakes?
I can certainly think of many examples of that.
How you unwrapped an abstraction and made things better by removing it?
I have dozens of battle stories.
Choosing not to use an abstraction because it was indirection?
Which is what the article says to do?
I’m skeptical.
I suspect you’ll find most examples of that are extremely open to debate.
After all, you didn't use the abstraction so you don’t know if it was good or not, and you can only speculate that the decision you made was actually a good one.
So, sharing that experience with others would be armchair architecture wouldn't it?
That’s why this article is arrogant; because it says to make decisions based on gut feel without actually justifying it.
“Is this truly simplifying the system?”
Well, is it?
It’s an enormously difficult question to answer.
Did it simplify the system after doing it is a much easier one, and again that should be the advice to people;
Not: magically do the right thing somehow.
Rather: here is how to undo a mistake.
…because fixing things is a more important skill and (always) magically doing the right thing from the start is impossible; so it’s meaningless advice.
That’s the problem with universal advice; it’s impossible to apply.
It might not give enough examples to show you how to do something right, but I think it's enough to warn against doing something wrong, namely introducing an indirection that doesn't provide any of the benefits of abstraction.
My colleagues invariably refer to indirections as abstractions, and it's a frustrating sort of name-squatting, because you can't usefully discuss the tradeoffs of abstractions if they're actually talking about indirections.
That said, the article does drop the ball by seeming to use the terms interchangeably.
> That’s the sign of a great abstraction. It allows us to operate as if the underlying complexity simply doesn't exist.
While I generally agree with the sentiment that current day software development is too indirection heavy, I'm not sure I agree with that point. All abstractions are leaky and sure good abstractions allow you to treat it like a black box, but at some point you'd benefit from knowing how the sauce is made, and in others you'll be stuck with some intractable problem if you lack knowledge of the underlying layers.
I'll try to explain, but I'm not sure my English is good enough for that task. But lets try.
When the author says "great abstraction" they mean "ideal abstraction". You can see this for example in this quote: " The less often you need to break the illusion, the better the abstraction." They say even the phrase "all abstractions leak", which is the main point of yours.
So, if they mean an "ideal abstraction", what does it mean? What it means to be ideal? It means to be an imagined entity with all sharp corners removed. The idea of "ideal" I believe is an invention of Ancient Greeks, and all their art and philosophy were built around them. Their geometry was an ideal thing, that doesn't really exist anywhere except the brains of a mathematician. Any ideal thing is not real by the definition.
Why to invent ideals? To simplify thinking about real entities and talking about them. They allow us to ignore a lot of complicating details to concentrate on the essence. So it is like an abstraction, just not for programming but for thinking, isn't it?
And now we come to the recursion. The author used abstraction over abstraction to talk about abstractions, and you used built-in deficiency of all abstractions (they are not real) to attack the abstraction over abstraction.
Somehow it not as funny as I felt first, but still...
While I too like to marvel at the TCP/IP stack as an example of abstraction done right, it would be unwise to think an abstraction is only "good" if you get it right first time.
The real point of abstraction is to enable software that is adaptable. If you are ever sure you can write a program the first time and get it perfect then you don't need to bother with any of this thinking. We do that all the time when writing ad hoc scripts to do particular tasks. They do their job and that's that.
But if you ever think software will continue to be used then you can almost guarantee that it will need to change at some point. If it is just a tiny script it's no problem to write it again, but that's not going to be acceptable for larger programs.
So this necessarily means that some layer or layers of your well-architected application will have to change. That does not mean it was a bad abstraction.
Abstraction is not about hiding things, it's about building higher levels of language. It enables you to work on individual layers or components without breaking the rest of the system. It very much should not be hiding things, because those things are likely to need to change. The bits that really don't change much, like TCP, are rarely written into application code.
I forget which programming talk I watched which pointed this out, but one extremely common example of this in Java is recreating subsets of the Collections API. I've done this before, heck even the Java standard library is guilty of this problem. When a class has a full set of get/put/has/remove methods, it is often not actually hiding the complexity of its component data structures.
Good example of a bad abstraction. If you're speaking the language (or "abstraction") of sets, you should see certain terminology: union, intersection, disjunction. These words are not part of the Java Set interface.
I would actually argue that the Collections API itself is a pretty good abstraction. It is often the case that conceptually I just want to work with multiple things and properties like order, duplicates, random access, etc. are not particularly important (in fact, requiring them adds inherent complexity). It's very useful that a vast amount of the standard library data structures conform to this interface or can create data views that conform to it.
How did "abstraction" and "hiding complexity" become perceived as such fundamental virtues in software development? There are actual virtues in that ballpark - reusable, reliable, flexible - but creating abstractions and hiding complexity does not necessarily lead to these virtues. Abstraction sounds no more virtuous to me than indirection.
>How did "abstraction" and "hiding complexity" become perceived as such fundamental virtues in software development?
They aren't just fundamental virtues in software development, they're the fundamental basis of all cognition. If I twiddled with every bit in my computer I'd never write a hello world program, if I wrestled with every atom in my coffee cup I'd never drink a sip of coffee.
Abstraction and information hiding is the only way we ever accomplish anything because the amount of information fitting in our heads is astonishingly small compared to the systems we build. Without systems of abstraction we would literally get nothing meaningful done.
Decades ago, when The Structure and Interpretation of Computer Programs taught us that programmers fundamentally do two things: abstraction and combination; and we are interested in programming languages insofar as they provide means to those two ends.
The two classic "hard problems" of computer science - cache invalidation and naming things - are both aspects of abstraction. Cache invalidation is a special case of making sure the abstraction does what it's supposed to, and naming is the most important part of causing the abstraction to have meaning.
Not parent, but I have a similar impression. Design patterns, clean code, and several of these well known tools were particularly useful during C++ and early Java eras, where footguns were abundant, and we had very little discussion about them - the Internet was a much smaller place back then. Most of the developer work was around building and maintaining huge code bases, be it desktop or server, monoliths were mostly the only game. And many initiatives grew trying to tame the inherent hazard.
I think that microservices (or at least, smaller services) and modern languages allow the code to stay more manageable, to the point where Java devs now are able to dismiss Spring and go for a much simpler Quarkus.
What languages do you work in? Would you be happier or more productive if you had to be aware of the quirks of every ISA and interrupt controller your code might run on?
Abstraction is good, in the way that leverage is good in the physical world: it is not always necessary, but people who are aware of the tool are vastly more capable than those who are not.
Part of it has to be that finding and defining abstractions is fun like a puzzle so programmers like doing it and finding ways to justify it after the fact
I disagree. There are base abstractions you can’t avoid, of course, like the machine code of your computer, or the syscalls presented by it. Using these is not abstraction, unless you choose to build up interfaces of reusable pieces. Abstraction is structure. You still have to actually write some code that can be organized into structure. You could write code using just those base abstractions if you wanted, or as many do, choose libc as your base abstraction. Watching a program through strace gives you basically this view, regardless of the abstractions the program actually used to achieve the result.
Some abstractions are so ingrained you don't even think of them as abstractions. A file is an abstraction. A socket is an abstraction. The modern terminal is an abstraction.
A network close call at a high level closes a network socket. But at the tcp level there's a difference between close and reset. Which do you want? Your api has removed that choice from you, and if you look you will have no idea if rhe close api does a close or a reset.
Is the difference important? If depends. If you have a bunch of half open sockets and run out of file descriptors then it becomes very important.
Another example: you call read() on a file, and read 10k bytes. Did you know your library was reading 1 byte at a time unbuffered? This abstraction will/can cause massive performance problems.
My favorite one is when a programmer iterates over an ORM-enabled array. Yes, let's do 50,000 queries instead of one because databases are too complicated to learn.
Just like any tool, abstraction has costs and benefits. The problem is that lots of people ignore the cost, and assume the benefit.
I think an important distinction is hiding details from other parts of the program, and having details being hidden from you.
99.9% of the time I don't care what the tcp socket closes with as long as it isn't leaking a resource.
And if I did care, then I picked the wrong level of network abstraction to engage with. I should've used something more raw.
Regarding ORM arrays. I have myself recently debugged such a case. I chortled a bit at the amateur who wrote the code (me last year) and the schmuck who accidentally wrapped it in a loop (me 3 weeks ago). Then I changed it slightly to avoid the N queries and went on with my day. No need to lambast the tooling or the programmers. Just write something maintainable that works. No need to throw the entire ORM away just because we accidentally made a web page kinda slow that one time.
And don't worry, I too lament when web pages I don't control are slow. You may rest uneasily knowing that that page would be slow regardless of whether ORMs existed because it is not slow because of ORMs, but because there is no incentive for the business to care enough to make it faster.
>> But at the tcp level there's a difference between close and reset. Which do you want? Your api has removed that choice from you, and if you look you will have no idea if rhe close api does a close or a reset.
If you are doing a „TCP application” of course it makes no sense to abstract the TCP layer. Is not about cost. Now if you have an application that has to communicate somehow with other system, and you want to not depend on specific protocols, then the communication part should abstract away that part.
How to deal with your example? Well, if you can say “I will always want X, you can make a configuration option “TCP.close” or “TCP.reset”. If “it depends” then you have to build the logic for the selection in the abstraction layer, which keeps hidden.
I found this to be a pretty poor article. The article lacks concrete examples and speaks in generalities to explain its core thesis. But without specificity, a reader can nod along, sure in the knowledge that they already are wise and follow this advice and it's everyone else who's out there mucking up code bases with layers upon layers of garbage. I mean,
> The next time you reach for an abstraction, ask yourself: Is this truly simplifying the system? Or is it just another layer of indirection?
Is anyone reading this truly going to alter their behavior? If I could recognize that my abstraction was "just another layer of indirection" as simple as that, obviously I wouldn't have added it in the first place!
I don't know. Sure, examples could be nice. But an article can be imperfect and still be interesting or useful. Also, it's easy to say come up with examples, but I find it hard sometimes.
In any case, I find it a nice article. Will it change how I write code? Maybe, maybe not. But it will change how I review code and talk about abstraction.
Let's not forget about a particularly frustrating kind of "level of abstraction": the bespoke interface to a part of the code that has side effect, and that has exactly two implementation : one in the production code, and one in the tests.
If I were to create a language tomorrow, that's the one aspect where I would try something that I have not yet found elsewhere : can you make it so that you can "plug" test doubles only for test, but keep the production path completely devoid of indirection or late binding.
(I'm curious if you know of a language that already does that. I suppose you can hack something in C with #ifdef, of course...)
While I wholeheartedly agree with the premise, the article doesn't really say anything other than "TCP good, your abstraction bad, don't use abstraction".
Reminds me of an old Java Android project I encountered.
EVERY class implemented an interface.
98% of interfaces had one implementation.
Every programmer was applying a different programming pattern.
A lot of abstractions seemed incomplete and did not work.
Proguard (mostly used for code obfuscation for Android apps) definitions were collected in the top module even though the project had multiple modules.
Half of the definitions were no longer needed and the code was badly obfuscated.
Problems were solved by continuesly adding classes and checking what sticks.
The UI was controlled by a stateful machine. State transitions were scatter everywhere in the code with lots of conditions in unforeseen places.
Legacy code was everywhere because no one wanted to risk a very long debugging session of an unforseen change.
No API definitions. Just Maps that get send via REST to URLs.
By biggest mistake was to not directly rewrite this project when I entered the team.
We did after one year.
The article seems to go with the premise that abstractions are most often carelessly introduced when there is an obvious alternative that is simpler and more performant.
Yes, abstractions have a cost that will accumulate as they are layered.
But simple elegant solutions are not free. They are hard to come with, so they often need large amount of dedication ahead of any coding. And as long as we don't deliver anything, we have no clue what actual requirements we miss in our assumptions.
The road to reach the nice simple solutions is more often than not to go through some some clunky ugly solutions.
So rather than to conclude with "before running to abstraction think wisely", I would rather recommend "run, and once you'll have some idea of what was the uncharted territory like, think about how to make it more practical for future walks."
But don't forget that the territory will change underneath your feet. This is especially true if you write business software. A tectonic shift in the business can make the assumptions you made a year ago completely invalid.
This is complicated further by the fact that good software will drive the business. So you will always be creating new problems because you're driving the business into new areas and capabilities that simply weren't possible before.
So this makes it doubly important to make sure your software can change in small ways over time. It's not just trying to navigate the moors at night with a torch, it's like trying to navigate the desert at night with a torch. The sand will move under your feet.
Interesting read, although I don't agree with everything. I like the distinction between different qualities of abstractions, made in the beginning. The following bashing of abstractions is too generalized for my taste.
The best part comes close to the end:
> Asymmetry of abstraction costs
> There’s also a certain asymmetry to abstraction. The author of an abstraction enjoys its benefits immediately—it makes their code look cleaner, easier to write, more elegant, or perhaps more flexible. But the cost of maintaining that abstraction often falls on others: future developers, maintainers, and performance engineers who have to work with the code. They’re the ones who have to peel back the layers, trace the indirections, and make sense of how things fit together. They’re the ones paying the real cost of unnecessary abstraction.
While I think it's good the pendulum is swinging toward a more restrictive approach to abstractions, we've (and I've) certainly been leaning a bit too much toward just solving every problems by adding a layer of indirection around it, and such onion-layered designs tend to (as the metaphor implies) cause a lot of tears when you cut through them. That said, it's not like abstraction itself is bad.
A big part of the problem is arguably that IDEs make code navigation easier, which has us adding all these indirections and discover only when it's too late what a horrible maze we've built. Being more judicious about adding indirection really does help force better designs.
I suggest there are three types of layer that one passes through
Abstraction - this thing of rare beauty
Decision - often confused for abstraction and wrapper, this is best thought of as a case statement in a function. They are wildly better in my opinion than lots of classes
Wrapper - either fluff like getters and setters or placeholders for later decisions (acceptable) or weird classes and instances that the language affords but tend to be confusing - what is called indirection in the article
Tools, utils, libraries - these are I classify as handles / affordances for other code to use - maybe they add layers but they add a single way in to the nice abstraction above.
This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. -Doug McIlroy
I can't help but wonder whether the problem is subjective, to each person and what they need to accomplish at the time. What is cognitive load and indirection to one at one time is a simple abstraction to another at another time.
And so I wonder if a solution to this is for editors to be able to represent the code differently, depending on what the person's needs are at the time.
Funnily enough, logical deductions or formal theorem proofs can be seen as a set of transformative steps from the initial premises to the conclusion, where no new information is added in the process. Which makes the conclusion (at a stretch) a bit like "just" rephrasing the initial premises.
Not really. Usually during the deduction you have some steps that pull surprising knowledge from the rest of math to support the reasoning or create and prove interesting lemmas.
If you can prove a theorem without any of that, that's a little boring theorem to prove.
I agree with the spirit of your comment, but not the literal fact of it. Sure, interesting proofs require pulling out some interesting knowledge in the reasoning, but notions like "surprising" or "interesting" are about human subjectivity and don't really exist as a property of a deduction. Surprising or interesting knowledge is not somehow new knowledge that wasn't there before, it's just that we didn't see it previously.
Sure, but it's humans that do the deduction. So "surprising" and "interesting" still matters. Especially when you are treating deduction as a parallel to a piece of prose that one might reasonably hope to be surprising and interesting not just repeated rephrasing of main thesis.
I'm not sure what it's called (abstraction vs. indirection) but I dislike when everything needs a class/object with some odd combination of curried functions. Some programming languages force this on you more than others I think? As a contrived example "StringManager.SlicingManager.sliceStringMaker(0)(24)(myStr)", I've seen code that reminds me of this and wonder why anyone uses a language where this not only an acceptable idiom, but a preferred one.
Indirection serves a purpose as well. One that is related to, but not the same as abstractions. When you add a layer of indirection, you make it easier to, say, delete or change every item X instead of iterating through everything.
Unnecessary or redundant levels of indirection are bad, just like unnecessary or wrong abstractions are. But when applied correctly, they are useful.
Shameless plug: a while ago I made a video explaining the idea of abstraction in computer science, and it seems to be helpful for beginners: https://youtu.be/_y-5nZAbgt4
GangBang-Abstractions would be a good name. Abusive, damaging the respective micro-biome ( parts of the code/system ) almost irreversibly and passing around the data for brutal telemetric exploitation ...
A good abstraction shouldn't make its usage shorter, it should make the proof that the usage is correct shorter.
This usually means the total amount of assumptions needed to prove everything correct is decreased.
(when the code is not actually formally verified, think of "proof length" as mental capacity needed to check that some unit of code behaves as intended)
The author misses the whole point of abstractions, and that is a layer B that covers layer A so completely that no one using layer B needs to know how layer A works at all, except for those working on the layer A/B bridge.
For example, binary logic is a perfect abstraction over semiconductor physics. No one doing computer science needs to understand anymore the complexities of voltages and transistors and whatever. TCP is a perfect abstraction over IP. Memory as a big array of bytes is a perfect abstraction over the intricacies of timing DRAM refreshes.
And that's about it. No one reading this post has written an abstraction ever. (a leaky abstraction is not an abstraction). So yes, actually, abstractions are free, and they don't leak. That's the whole point. The problem is that what you call an abstraction isn't an abstraction.
Computer science's complete failure to create a new abstraction since like TCP intrigues me. Why don't we have a system X that abstracts over memory accesses so well that no one needs to know anymore how caches or memory locality works? Why aren't there entire subfields studying this?
It's a huge ecosystem which turns 30 in May. Yes, it accumulated lots stuff that was cool back then and isn't so cool now but overall is doing pretty well (COBOL was 36 when Java got public).
Without more details his position rubs me the wrong way.
As somebody who has done a huge amount of "fix this bug" and "add this feature" on existing code bases I think excessive use of cut and paste is the worst problem in the industry. Cut-and-paste is the devil's own "design pattern" as it is a practice that gets repeated throughout a code base to solve various problems.
When it comes to bugs repetition means a bug might have 13 copies throughout the code and you could easily get the ticket sent back 2 or 3 times because you didn't find all the copies at first.
Repetition (together with poorly chosen abstractions) also causes features that should add in complexity to multiply, as if I have 3 versions of a function and now something can vary 5 ways I now have 15 functions. In a good design you might pass one of 8 functions to a 9th function. Repeat this a few times and one guy has 98 functions and the other guy would have had 13200 if he'd been able to get that far.
Granted the speed demon won't like all that function calling, right now I am thinking about writing a big switch statement for a CPU emulator, I get it, for you there is code generation.
It is also healthy to have "fear of framework planets", a horrible example is react-router which has gotten up to incompatible version 7 because (i) it's the kind of thing you can write in an afternoon (it would take more time to write good documentation but... take a look at that documentation) and (ii) the authors never liked any of the frameworks they created. More than once I have dug into a busted app written by a fresher where there was a copy of react-router and there were some use's from it in use but they had bypassed react-router and parsed document.location directly to figure out what to display. The very existence of a bad framework creates a kind of helplessness.
Those folks will say various versions of react-router support React Native, SSR, etc. We don't use any of those where I work, I don't care.
It is a very good prop bet that you can dramatically speed up so-and-so's program by switching from AoS to SoA.
(If it's a Java program, you eliminate the overhead of N objects to start with)
but it's tricky to implement arbitrary algorithms, my mental model to do it is to build programs out of relational operators (even in my head or on paper) SQL is one of the greatest abstractions of all time as I can write a SQL query and have it be run AoS or SoA or some hybrid as well as take advantage of SIMD, SMT, GPU and MP parallelism. 10 years ago I would have said I could have beat any SQL engine with hand-optimized code, today products like
I don't know how one read an article that talks specifically about how great TCP is as an abstraction and come away with the conclusion that it's arguing that all abstractions are bad.
>There’s a well-known saying: "All abstractions leak." It’s true. No matter how good the abstraction, eventually, you’ll run into situations where you need to understand the underlying implementation details
This is false. One can read up on Theorem's for Free by Wadler to see that not all abstractions are leaky.
Theorems for Free tells you that some abstractions satisfy some mathematical properties (for free!) under some circumstances.
If you write down a function with signature
{T : Type} -> T -> T
then it must be the identity function, if you do not use "malicious" extensions of the type system.
But what is the performance of the identity function?
Here is an identity function:
lambda T, lambda t, if (2 + 2 = 4) then t else t
In other words: I can hide pretty much arbitrary computation in my identity function.
Users of my identity functuon will notice that it is wicked slow (in reality, I let my identity function compute Busy Beaver 5, before doing nothing). Their complaints are evidence of leaky abstraction.
Now you might have a smart optimizing compiler that knows about Thm4Free... But that's another story.
The point of the abstractions I'm talking about is not to abstract away the computation part of a program, it's to abstract away the types and complexities of the semantics.
I don't think you can have such compiler, at least for a general case, without solving the halting problem first. After all, you can encode arbitrary computations at the type level.
This has such potential to be an interesting thread of conversation but all I get is a reference to a book that I haven't read and am unlikely to.
What examples of non-leaky abstractions do you have?
I could imagine something like "newtonian physics" but that leaks into my daily life every time I fire up google maps and get a GPS fix.
The OP's example of TCP seems close to the mark, but to be totally honest I'm not convinced. Every time I have to run ping to check whether my hung connection is due to connectivity, I'm breaking the abstraction. And I've had to debug network issues with Ethereal (yes, I'm dating myself). TCP does leak, it's just that most people don't know what to do with it when it does.
In computing, we emphasize the communicational (i.e. interface) aspects of our code, and, in this respect, tend to focus on an "abstraction"'s role in hiding information. But a good abstraction does more than simply hide detail, it generalizes particulars into a new kind of "object" that is easier to reason about.
If you keep this in mind, you'll realize that having a lot of particulars to identify shared properties that you can abstract away is a prerequisite. The best abstractions I've seen have always come into being only after a significant amount of particularized code had already been written. It is only then that you can identify the actual common properties and patterns of use. Contrarily, abstractions that are built upfront to try and do little more than hide details or to account for potential similarities or complexity, instead of actual already existent complexity are typically far more confusing and poorly designed.
reply