The basic point is right, but I think a lot of the details are off. The core idea is composition. A system is compositional if you can build it from small reusable parts. It's one of the key ideas in functional programming, and it's somewhat core to object-oriented programming. (OO is too big to be able to claim their is a consistent philosophy behind it.)
But ... firstly, let's stop with the Unix worship. They didn't deliver, even in the domain of the terminal. My copy of `ls` has at least 40 flags (where I stopped counting). Secondly, people have tried composition on the desktop. That's what CORBA, COM, and OLE were all about. They all kinda sucked for various reasons. Finally, the place at which the author arises has been around for as long as Unix: it's Emacs.
Let's just be careful what we call Unix though. In SVR4, I don't think "ls" had that many options. It didn't even have colour (not that I would know; my Wyse 60 could only have Green, Orange or White text... and the colour was set in the factory).
It's the same for many programs that started small but grew loads of knobs with age. You can't claim that this was the Unix vision, because what we call Unix today isn't even technically Unix.
I'm not saying Linux is bad or anything; I'm just pointing out that it's much more organic than Unix ever was, and that the resulting system is not necessarily carefully orchestrated.
IMO the problem with the Unix model is not that it's been corrupted over time, but that it was fundamentally flawed from the start. I'm hardly the first to point this out, but sending around unstructured text makes doing some things inordinately hard (imagine writing a separate program to color ls's output, instead of having that as a flag) + going beyond pipes leads to the awful experience of programming in shell.
The Unix command line tools were designed to be usable in a lot of ad hoc situations. In an ad hoc situation, unstructured text is often what you have. So, to be usable for that, the command line tools had to be able to operate on unstructured text as input. That left two options - either also accept something structured as input, or accept only unstructured text as input.
"Usable in an ad hoc situation" gets us back to composability. If you can compose existing parts to meet a novel situation, that's better than having to carefully architect a solution. And the key to being able to do that is common interfaces. Unix did exactly that. And, for all the criticism, it works pretty well.
Can a well-designed interface do better? Sure. Can a well-designed interface deal with something outside the design parameters as well? Probably not.
I mean people still use `ls` everyday so it in fact did win. We have super rich GUIs, REPLs, the start menu, live programming environments and notebooks, web browser file listing capability, the ability to build your own web app to list files visually anyway you want.
Depends on how you define "functionality". You could say that ls's functionality is "providing a text listing of files in a directory. You can't get that from a GUI, certainly not without extra steps.
Why would anyone care about that difference? If I'm reading the listing, I don't care. My eyeballs don't care if the photons came from a GUI or text in a shell. But if I'm trying to pipe the results to another program, then I care. GUIs don't pipe well.
And people like me pipe the results of ls all the time.
How those bytes are interpreted is up to the endpoints of the streams. Sure, a lot of endpoints interpret byte streams as text. That's not Unix, that's a particular endpoint.
You will notice that Unix does not even offer 'text' files (like Windows does). Or ISAM files, or random access files. or Fortran carriage control. Just binary files. Because the Unix way is streams of bytes, and nothing more.
Further if you wanted to go with JSON or something you can, there's nothing preventing it. Pipes are based on unstructured text so you have that flexibility. If you want to add JSON ls and filter that through JSON sed to JSON less you can.
Finally 'the Unix way' is a cultural thing. It isn't set in stone. It can, and probably has evolved. So while you could argue that it meant that to some people at some time, that doesn't make it true today or in the future. The down side is that we have these disagreements over what 'the Unix way' is. But any philosophy that's been around for 40 years, in tech no less, has to be flexible, that's why it's stay relevant.
I disagree, the unix way has structure: lines of text each consisting of space-separated fields. The real issue is that sometimes your data has spaces, or even newlines, so the structure quickly isn't enough. And that "sometimes" happens xay more often than it has to, and the escape hatch is a mess.
> lines of text each consisting of space-separated fields
I wish! Back in the day, tools like iostat and vmstat printed tabular data using fixed width fields with no spaces between. This worked fine until the computers got bigger and faster, then the larger numbers filled the fields so they ran together making the output unreadable. A surprising number of vendors had this problem.
It's bizarre that this is the case given that ascii contains characters for this very purpose. There's record separator, group separator... Now if your data contains those, that's bad. But that's much rarer than containing ordinary whitespace.
I don't think so, this idea of JSON is a more modern interpretation and I'd say is counter to the UNIX philosophy, not an extension to it (it just so happens to squeeze into existing primitives a little better than e.g. a binary serialisation protocol).
The thing is, we can see even from the 1970s 'ls' how the Unix model doesn't meet the goal "to chain these simple programs together to create complex behaviors".
There is no option to escape or NUL terminate a filename, making it possible to construct a filename containing a newline which makes the output look like two file entries.
The option for that was added later.
There's also the issue that embedded terminal codes will be interpreted by the terminal.
It's still a failing of the Unix philosophy that people found it easier to add all these flags to ls than to string together a few "simple&composable" commands.
It's for the very simple reason that flags are more easily discoverable (with simply a couple of tab key presses I can iterate and see the list of most flags of most of the cli apps I use) than "whichever program amongst the 7362 executables with obscure names I apparently have in /usr/bin will do the thing I want"
I guess they could have implemented some of the flags as aliases or rather bash functions to a preset chain of invocations. The merging of flags would be difficult though.
There are many flags that change the behavior of the file system calls that would be made...such as getting additional information.
To support all the formatting commands, the piped output of `ls` would need to be very verbose and structured too...like json or something.
They had to draw boundaries of functionality at some point.
So the unix idea of "one thing" was probably defined a bit more vaguely or more end-user use case related. Convenience seemed to be king.
It would have be nicer if they could have added a shell-like pipe syntax to C (and designed the code in a more modular and stream-like way).
In the `ls` [source code][1] you can see simple blocks of steps like:
For example the `file_ignored` function in `ls` would have made a nice reusable library (with standardized configuration params) and a cli tool (with standardized flags).
Missed opportunity to unify the shell and C, but I guess SmallTalk was kicking around at that time which went a whole lot further.
It's funny that we are still very much lacking this unification...
We don't have a high-level interpreted language that can also perform well as a system's language.
New systems languages are popping up all the time, but they are more and more hardcore (looking at you Rust). They don't embrace any aspect of scriptability.
That is what people outside Windows keep thinking.
After WinDev managed to botch Longhorn efforts, its .NET ideas were redone in COM for Vista, and it has been like that ever since.
WinRT/UAP/UWP, is basically COM with a new base interface IInspectable, .NET metadata instead of TLB files, and app boxing.
And it is the foundation of WinAppSDK, whose goal is to port UWP subsystem into plain Win32, although their current execution leaves a lot to be desired.
Microsoft ported OLE 1 to Macintosh, but never OLE Version 2, because Apple guidelines forbid programes to start other programs. Because of that Photoshop had a not supported version of OLE. I don't know if it is still available in up to date Photoshop.
Yes, ls does have a lot of flags, but how did it get them, and what are the alternatives? It got them to satisfy real needs, and using flags, especially when combined with having man at your fingertips, was (and is) a highly effective interface for interactive work. If there is anything to be learned from the proliferation of command-line flags, it might be that one should not attribute too much significance to a vague aphorism such as "do one thing, and do it well."
I agree that composition is the core idea and I was disappointed that the author neglected to summarize the idea correctly. Composition is a structured way of describing a unique task using a few common nouns and verbs.
But composition needs to be understandable and controllable to be useful.
For it to be understandable, it needs to match the user’s anticipated mental model. Now, this isn’t an appeal to an intuitive notion uninformed by user education. But it does need to make logical sense to connect two things together temporarily to solve one task. And that logic needs to come from the user and not the developer. In other words, the developer needs to provide concepts that conform to a user’s algebra. For a photo editor, that’s hard to provide to the average user. For some CLI tools like wc and cat, it is easy.
For composition to be controllable, the primary method of configuration needs to be in the contextual relationships among the tools and not in the tools themselves. That is, using cat and wc to count the number of words in a text file is primarily enabled by the filesystem and shell features, none of which require user configuration during the operation. Configuring wc to count instead the number of lines doesn’t distract too much, but imagine counting the number of lines emitted from a streaming serial device: you want it unbuffered, but only kinda because of other issues that arise, like multiline error conditions. In that case, the value is disrupted by edge cases and you have to configure the nouns (the serial stream at least) by looking it up on Stack Overflow. It’s not just easier to consider building a monolithic tool, but arguably the right thing to do because you need to configure everything so much. Plus, this is too much to expect the user to handle through a simple composition.
In my opinion, desktop composition doesn’t work because the tasks are often beyond the limits of composition. And without a culture of composition among users, it’s harder and harder for remaining desktop applications to do it. Especially as younger users are unfamiliar with concepts like the filesystem, system keyboard shortcuts, “saving” files, offline operation (i.e. you have to reason through a problem and cannot lookup the answer), etc. Better to go with the monolithic app model, which can still use plugins under the hood or multiple processes for developer productivity. And I’m sad about that.
This is a version of a No True Scotsman argument: the True Unix did not have these flags and therefore these flags are not part of the Unix Way. The extra flags were added over time because the One True Unix did not provide the functionality that people needed in real world use. They couldn't be added in a composable way because of failures in the shell programming model (unstructured text + sh is terrible). The Unix shell model was broken from the start.
I won't deny the useful additions GNU and BSD made to the barebones UNIX tools, but just by skimming the ls man page I can list some flags whose addition wasn't caused by the Unix Philosophy's limitation:
-1 list one file per line
-C list entries by columns
-t sort by time, newest first; see --time
-r, --reverse reverse order while sorting
-S sort by file size, largest first
--sort=WORD sort by WORD instead of name: none (-U), size (-
S), time (-t), version (-v), extension (-X), width
I think you’re actually making the point for him here. You are correct that GNU is not Unix. And what version of ls has the most installations? I betcha it’s the ls from GNU Coreutils. I do agree that GNU tools are sometimes too maximalist. But they are also more useful than the minimalist alternatives.
I don't believe GNU's triumph over the BSD derivates was for the most part technical, they just happened to be at the right moment at the right time to ride along Linux's success.
A consequence of the proliferation of arguments in the GNU tools is imho that it made the GNU manpages too verbose and information dense to the detriment of their usefulness.
That's exactly why RMS is so pissed at Linux being mentioned without GNU. The funny thing is that it is the license he shipped his code under that allowed them to do this, he could have made the license say 'if you use this code you are required to prefix the name of any project with 'GNU/'' and that would have been that. If you give people rights and freedom they will use them.
That is because of the "lawsuite", unless you are referring to the 80s.
Back then, Companies and Colleges were the ones who got UNIX. Companies went with AT&T, and IIRC Colleges when with BSD. So AT&T won out due to better financial backing and maybe a possible threat of AT&T going after BSD.
To me, the point is you don't need many flags for ls to be useful. You will learn a few idioms early, such as ls -a and ls -lR but after that, the rest of the flags are nice to haves: if you wish there was a certain feature, you can look it up on the man pages etc. "I wish I could sort by time - oh there's -t"
In many cases, yes! On my Mac, which has a userland derived from FreeBSD, I have voluntarily installed stuff like GNU Findutils and GNU sed to get that extra functionality.
> A system is compositional if you can build it from small reusable parts.
Take big monolith. Refactor into 1001 pieces. Glue together again.
If glue is inflexible, then result is monolith+glue.
If glue is thick, then result is mostly glue + little bits of monolith.
If glue is superb (well designed & flexible), the whole works the same as monolith, but pieces can easily be rearranged as needed (and individually tested!).
You have to hand it to them though, that shell commands are still the least-typing way to do something compared to every programming language out there.
Like if you gave the user an always-on JavaScript or Python repl, shell commands are still going to win every time.
There are so many JS functions I have that I wish would just get automatic cli interfaces...instead of having to go through the ceremony, or use a repl, or something like that.
Here’s the actual good advice: Build your system as a set of composable parts.
The article doesn’t really quite get there, but is close. Almost everything else in there is wrong and should be ignored. Let’s see… *nix command line never did follow DOT, that’s the definition of shitty, not enshitification, the graphs are pulled out of thin air and represent the author’s feeling, not data, microservices don’t solve problems in large projects that various other approaches don’t solve (microservices tends to combine an certain approach to architecture, deployment, organizing dev teams and assigning responsibility to them within a larger organization, organizing code, change management, etc… the relevant things here aren’t specific to microservices).
BTW, the microservices graph shown near the end is a lot like spaghetti architecture, where everything depends on everything else. (Not that you have to do it like that, but you’ll need some higher-level organization to manage it.)
You don't need a damn network between two pieces of code just to "do one thing and do it well", for God's sake, I'm so sick of this rampant cluelessness in the industry.
Do you saturate the resources of one machine and need to split things off? Do you have multiple teams each taking care of their own stuff? Do multiple services, it's fine in those cases. For almost any other reason you are just adding complexity, boilerplate and additional failure conditions.
If you think that microservices solve the "spaghetti code problem", well, good luck to you, you'll need it.
Who's hiring grugs, the lazy programmers that try to do the most efficient thing in the least amount of work? I'm not smart enough to set up a whole georeplicated k8s cluster for your blog. I'll use Apache, maybe put Varnish if you get on the frontpage of HN, ok?
(Work smart, not hard. DevOps today is the exact opposite of that)
When a company measures developer productivity in code added and feature delivered, it creates a perverse incentive that attracts people that really love to write a lot of code and building mountains out of molehills.
The best engineer (and not only, see that famous quote by Kurt von Hammerstein-Equord) often is the lazy one. It was a known notion that somehow disappeared around the turn of the millennium.
> If you think that microservices solve the "spaghetti code problem", well, good luck to you, you'll need it.
That is a very good point. In fact, good monolithic code layout is a precursor for microservices, as only once you have identified your boundaries and isolated your concerns can you begin splitting them out into their own services.
I will say this however - microservices might not solve the "spaghetti code problem", but it definitely helps isolate it. When we get consultants in to speedboat a new system, we give them their own separate service. Saves a lot of time and de-risks our beautiful monolith.
Yep and people you do not trust should not have access to CI/CD workflows and repo settings. Your CI/CD workflows should prevent everything else you mentioned.
What specifically do they not solve? Because in this thread we're, very specifically, discussing isolating the well maintained monolith from risky product produced by consultants, which microservices absolutely do solve.
Well, speaking for myself, I was against them from the start. Maybe the rejection of distributed objects (when it came) was a hot trend also but it didn't make it wrong.
If you have a problem with bad consultants then solve that. Trying to solve it by jumping into microservices you will just have bad consultants writing your microservices equally badly, or even worse.
You wont know how bad the product produced by consultants will be until it already had effect.
And on pricing specifically, both shitty and good consultancies charge about the same. The ones that charge well below market are an outlier that will guarantee bad product - those are to be avoided.
When we get consultants in to speedboat a new system, we give them their own separate service. Saves a lot of time and de-risks our beautiful monolith.
Can your "facade" "isolate" interns making change in your service as well so that the whole service which powers the system doesn't go down. Microservices can.
I don’t understand how micro services prevent a system from “going down”. I pushed an infinite loop to the auth service, oops. Now nothing works. How did micro services prevent this?
Totally missed the point. Even with Code reviews, Unit/Integration Testing, Automated CI/CD and more can a person broke the prod.
One intern was told to deploy the app and I guess he made a mistake and updated k8 configs which deleted all the pods. Tell me how code reviews are supposed to catch it.
If you think "why not restrict config actions" etc, it's because the dev can go there and tweak those actions too. And each step added means less velocity of development. It's a game of cat and mouse. With isolation, you can limit the blast radius and loss of revenue.
> One intern was told to deploy the app and I guess he made a mistake and updated k8 configs which deleted all the pods. Tell me how code reviews are supposed to catch it.
So you don’t have a staging environment to check config changes before deploying to prod? Or you don't put k8s manifests (or helm charts or whatever)in source control and just update prod through CLI? And you let interns do it alone?
Your problem isn't interns, it's a terrible deployment processs with weak controls. Even senior engineers are going to break prod regularly in conditions like that.
> One intern was told to deploy the app and I guess he made a mistake and updated k8 configs which deleted all the pods. Tell me how code reviews are supposed to catch it.
These sound contradictory to me. Part of having automated CI/CD, is not having to manually deal with k8s to do a regular deploy.
Sure with proper CI/CD in place which you need regardless of app being a monolith or a set of micorservices. It's also infinitely simpler to setup proper CI/CD for a monolith.
This. So much this. We are 5 years into a microservices wank-fest. So far the net ROI is negative, the user experience sucks more than ever, the complexity is so high that people can't get things done, nothing works properly any more and no one owns anything because they have washed their hands of it all. But this is still promoted as a success because no one wants to be accountable for the fuck up.
Our team spend most of the time designing fucked up messes that run over poorly designed APIs, slowly, that impact customers. If it's not that it's upgrading 100 services worth of dependencies constantly, debugging contract violations and weirdness or performance issues.
There are companies that are very successful and have split their service architecture into domain entities, called microservices.
Are they successful because or despite this decision?
Is both true in some sense?
A more natural way to split up a server architecture is to use computational boundaries. These are found by thinking of how data is processed and flows through the system as opposed to separation of high level domain concerns.
But this requires a computational design and not a domain/feature centric one.
There are successful companies that this model applies to. But the problem is that they are outliers.
The biggest problem with the whole IT industry is seeing outliers promoted as the successful path and people taking on faith arguments for technical decisions instead of rational decision making processes.
The code reflects the organisation, that's one of the laws of computing (Fred Brook's law maybe?). Microservices work well in a sufficiently large org, with autonomous entities working on different parts of the system, extremely well-defined interfaces between the services, and someone high up having a bird's eye view of the system.
If your organisation doesn't look like that, you can only fail at microservices. Either the company restructures around the code, or the code looks like the company. Trying to make pasta from mash potatoes is doomed to fail.
This is one reason I am an opponent of microservices generally. Every organisation I've worked for is made of shit and straw bound together with a solution of piss. ergo a monolithic crap pile is were they should aim.
Plus it's easier to run a hundred monoliths than it is to run 100 services. This is on the basis that running 100 instances of something the same with no inherent complexity in each instance is much easier to automate, build and manage than 100 things that are different and have to talk to each other.
> If it's not that it's upgrading 100 services worth of dependencies constantly, debugging contract violations and weirdness or performance issues.
Of all the reasons microservices are bad this isn't it. Your first point can be handled mostly automatically with half decent SRE tooling. Lock files on any modern language (including Python via poetry) fix dependency weirdness. Contract violations are a problem with your developers not the style of programming. As for performance I've been on both sides and being able to independently scale microservices via kubernetes is a god send. With proper tooling microservices work great.
The real problem with microservices is what you mentioned before that IMO. Complexity for complexity's sake. Some things are better as monoliths and some things are better as microservices. It's the same problem you see with normalization vs denormalization. Microservices are unambiguously faster in almost all cases. "Do one thing" means that one thing can be optimized to death, in isolation, without fear of interrupting other services. It also means team structures are easier to manage. Microservices don't mean pull in the entire universe of RESTful libraries and use exactly one feature for 1 or 2 endpoints.
But this introduces another problem:
A second managerial problem is that microservices are often used in scrappy startups with < 200 engineers. On any complicated platform this means one engineer might need to know how 10 services work. That's not feasible. You need to split it so a team owns a couple services and "does one thing right" (those services) and nothing else. It's the overapplication of microservices to every problem that is the source of most woes.
Startups will religiously apply microservices. The thought, of course, being that they can hire/fire faster and an outsourced engineer can probably pick up the necessary work quicker. Of course, this never happens, because instead of 10 microservices you have 300 and no one even knows where half of them are. 99% of the time it's not the pattern that is the problem. It's cargo culting. Just like everything else in the industry (looking at you electron, leetcode, functional programming, agile, etc).
It's over applied in almost every professional context. I've watched entire codebases go from reasonably tolerable OOP to FP and the subsequent fallout. Usually, it's because some unhinged programmer starts adding stuff they like to a language it doesn't fit with. Then, you bring new developers on and no one wants to wade through the several layers of map/flatten/reduce/etc. Worse, the bolted-on FP is not optimized well by the compiler leading to worse than usual benchmarks. It's almost always a lose-lose except when it's done with a functional-first language.
It has it's purpose. It's almost universally over-applied owing to the cargo cult and holier-than-thou attitude of a lot of functional programmers. I see it happen in Python where comprehensions are often replaced with maps. While it is nice, the comprehension is more idiomatic and far more widely understood.
> Do you saturate the resources of one machine and need to split things off?
No, of course not. We divide a single machine in an uncountable number of virtual ones; write some code to make sure they don't talk to each other; write some code to make them able to talk to each other; write some code make more or fewer divisions on the run, automatically; and write some code so we can set them up automatically too every time we get a new split.
That's how you use software scalability and create a simple, predictable ops environment.
And what if a certain permutation of these hundreds of services goes down or slows down? What if one service starts generating tons of data and floods another service? Do you have backpressure figured out? Distributed systems introduce a whole new level of complexity and edge cases, but most companies never even hit the scale at which it matters. It's just a pointless, wasteful exercise in "how they do it over at Google".
Nothing that can't be solved with more code and more splitting! That will make things simpler! The more problems your code causes, the more code, the more solutions!
(Well, ok, I'm having a hard time solving the first one with more code, but did hear this claim applied to it more than once, so I'm keeping it general.)
There is a huge amount of people that sees nothing wrong with my rationale up there. I don't get it either.
I think you are misunderstanding the whole topic. Obviously it's about software at scale. Picking the right solution for a problem is kind of the whole job of software development.
I have gone through the migration of monoliths to microservices (yes, at scale with multiple teams and requirements to scale individual components etc.) and it solved a lot of problems.
The benefits outweigh the costs in my opinion (and drastically so).
When people say "you can write well separated components in a monolith..." I can only say: of course you could, but you are not going to (and certainly not everybody at your giant ass company is going to).
The problem is that most of people don't work in a giant ass company and so don't have the same problems you've had have, and thus they can achieve the same easier, faster and cheaper with monoliths - but they don't because of the hype.
I also don't get the comparison to UNIX philosophy into the domain of service development. These are totally different domains with their own patterns of resource usage and interaction. Piping "fairly" simple input -> output code that is sharing machine resources and releasing them at the end of execution is total different than running a micro service architecture across multiple containers/hosts. Even if some of them share the same host there is still extra resource overhead from their allocations that will have a constant baseline.
Unix has a lot of services that listen to 'localhost'. It has had elements of a service oriented architecture since the first daemon was launched, even though the IPC endpoint wasn't always something listening to a socket.
I can’t even get people to make new web pages in a web app past year 1. Everyone wants to just cram new features into whichever page makes the most sense, and so average page load time just gets worse and worse because we still want ### milliseconds but now the page is doing twice as much.
The way in which microservices, unix programs and extensible editors in this article are compared as if they are all attempts or at least examples of how to somehow "solve software architecture" is really rubbing me the wrong way. The author is speaking as if "How to organize software" is the nail and these things are all attempting to be hammers when they are really not.
Would I want to build a world wide streaming service, I would probably need some custom microservices and I will not need some kind of plugin architecture for them. Would I want to build a slick editor that non-programmers can use for knowledge organization I will not wire something together with awk and gnuplot. If I want to quickly search some log files and sort and count the errors in them I will not look for an obsidian extension nor build a microservice. The question of how to architect such software is still an open one.
I was happy about the clear thesis statement in the byline of the article:
> Extensible programs like Obsidian have achieved a Holy Grail of software architecture, after decades of failed attempts
However, by focusing on categories like "Applets", "Unix programs" or "microservices" the article in effect did barely touch the topic of software architecture and offered no evidence to support that byline.
I have recently had the thought that this is mostly what software architecture is...at some level.
It's about where we draw the boundaries around code to make it easy for humans to understand.
Imagine taking an existing repo, and extracting every block of code into a function, and then moving them into their own separate file in one big folder. You could extend it to all the libraries of your code, and of those services we communicate with over the network too.
This would look a bit like a debugger symbol table. It's closer to how the computer understands the program.
The app still runs. It's just hard for people to understand what is going on.
Software architecture is simply the grouping of these functions to some extent.
It's not a foolproof analogy, because there are some decisions to be made about implementation details of things, but at some level of abstraction you would find the same operations need to be run.
Just an interesting thought.
> The question of how to architect such software is still an open one.
I find a problem we face with architecture discussions is that its always about tradeoffs that are not immediately apparent.
And it takes a lot of mental effort to remember why something is a bad idea.
The way we discuss it is limited by plain text. It's hard to demonstrate a system evolving over time in a concise manner.
Architecture should be evaluated by throwing a spec at it, and then changing every part of the spec (including adding perf requirements) and seeing how long it takes to make the changes, and how many bugs it has.
I was there, in the 1990's, as a working professional not a hobbyist. The popular architecture was, for better or worse, structured as layers, not spaghetti.
Some software from the 80s might meet that definition, but I think it's uncharitable (almost to the point of insulting) to assert that until the 2000s most programmers didn't know how to write anything but spaghetti code.
Since Wirth nobody had any excuse. Structured programming was the key to breaking through the few-thousand-lines glass ceiling. Case in point: I tried writing a program to be able to edit sheetmusic and to play it back using the sound hardware in my computer. Endless re-tries of doing this, it would start off just fine and grow and grow and then I would get bogged down. With every iteration it was clear that it was getting better but I never could see it through to the finish. Then I read Wirths book https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures... and even though the book was geared towards Pascal it was like someone had turned on the light. Everything fell into place and suddenly I could write code until I ran out of memory and keep it all organized. Massive change. This was in 1982 or so.
Proper spaghetti code, in my mind, has always been pre-structured code. Which generally means assembly.
C is rough to refactor sometimes, but it's nowhere near the kind of "push a couple return addresses and jump into the middle of a function" that used to exist.
I wish people would stop applying the "only do one thing and do it well" to all software.
Kernighan/Pike were explicitly talking about software tools in the Unix environment, not about all software in general. It is an idealist, you might even say elitist view of how software should work, which they then realized in Plan9. And as impressive as Plan9 is, there's a reason it never got widespread adoption, and that is not just because of evil Microsoft/IBM/whatever. It simply did not solve the problems people had, because it did not run the software the people (which are mostly non-programmers) actually needed to do their daily work. Yes, you can use these systems to write shell primitives, yet another build system, a window manager, a Wiki system, a basic text editor - mostly tools for programmers. It's like people with 3D printers mostly printing stuff to improve their 3D printer.
I love the cognitive dissonance of people saying "do one thing and do it well" when talking about Unix/Unix-like systems shipping with tools like sed, awk, and Perl. The idea that gets elided or list entirely is building composable tools.
Composability lends itself to focusing a tool on a task but doesn't necessarily require it. If you build tools focused on composability i.e. output that is regular and easily parsed and properly using input and output channels, you'll get a system that is very extensible and responsive to end user needs.
If you instead focus just on the "single purpose" tools you'll often end up getting ones that are too limited. They'll end up tightly coupled with other programs/service since their functionality alone doesn't do anything all that useful.
Came looking for insights, found none.
Instead I got platitudes like "sometimes microservices work well, sometimes not" and some copied together cartoon graphs, with a topping of meme pics.
The conclusion is: "There are no easy answers". I wish I was kidding.
Correct. Even in this thread, it’s just a bag of opinions. Most are elaborate and well articulated, all correct, all wrong nonetheless. No wonder this topic is always such a pile of patchwork. It’s as diverse as developers are.
I think the core thesis is that "do one thing and do it well" is about small components that are connected together.
The classic example is shell pipelines, but they are actually quite shit because they only deal with unstructured data (there's finally some work to fix that in Powershell and Nushell but it took many many years).
A better modern example of doing one thing and doing it well is apps that support plugins.
----
I think the whole "do one thing and do it well" is terrible advice. A "thing" is not well defined. It's equivalent to "don't have too many features" which of course leads to "how many is too many" and you're on your own.
It's one of those bits of advice like "premature optimisation" that is more often used to excuse thoughtless design than to motivate good design.
Yup, but who knows, that graph also doesn't even bother with a description on thy y-axis, maybe it is supposed to go from 0 "maximum bugs" (zero code non-buggy) all the way up to "no bugs" (all lines of code bug free). Or maybe the author just wanted something colorful for us to look at.
> Big monolithic apps have large codebases that slow down development velocity. They’re slower to compile, harder to test, and full of dark corners where bugs can lurk and multiply. A bad change to one part of a codebase can cause headaches for an entire building’s worth of developers, tanking productivity for hours or days.
This has NOTHING to do with monoliths. If you cannot architect a monolith, you’re going to have a hell of a time with micro services. This just comes down to being an awful software engineer.
I don’t think this is true. I’ve seen the same 30-odd engineering team build a horrible monolith and a pretty decent microservice architecture. The difference between the two was that it was very easy to cheat in the monolith world—it was expedient to patch in some feature by giving access to some thing which ought to be private in one component and that shortcut was often commanded by management (often with promises that We Will Definitely Prioritize Fixing Properly Next Sprint). Microservices make this sort of cheating more difficult than doing the proper thing, so management and their yes-men don’t have an incentive to cheat in that particular way.
Another reason things were cleaner was because we could write components in the language that was best suited for it. In our monolith version, everything had to be Python because we essentially needed its data science ecosystem, but that meant every other component in the system was fighting against Python’s package management, performance, and reliability problems for no material gain.
I think you can get similar rails in a monolith world via strict anti-shortcut culture, but technical controls are a lot easier than political controls IMHO. Similarly, you could probably build some Frankenstein FFI regime in a monolith to support multiple languages, but that seems strictly worse from a maintainability perspective.
All that said, there are definitely costs to building with microservices—I’m not saying they’re a panacea or even better than monoliths in the general case, I just don’t buy the “if you can’t do it in monolith you won’t be able to do it with microservices” line.
It has nothing to do with individuals not being great, organizations of perfectly talented well intentioned people still end up with a mess eventually.
Those same engineers will still be there in the microservices world. You can argue that the damage they can do is reduced, but I don’t think that’s always true.
Good article, but to me it seems to be co-opting the term enshitification in a strange way:
> Large codebases will eventually reach an “enshittification point” — the point at which bugs are introduced faster than they can reasonably be fixed.
I think of enshitification primarily as an organisational/business phenomenon rather than a technical one.
It doesn’t necessarily emerge “bottom up” as the result of technical debt but from top down as a result of misalignment of values between the brutally commercial goals of the business and the values/goals if its customers. Tech debt and development velocity do play into this but I don’t think it’s the primary driver of it.
The term is a useful one at that level of abstraction imho because we already have quite a lot of language to describe things like this at the lower, more technical level.
> I think of enshitification primarily as an organisational/business phenomenon rather than a technical one.
The other two important criteria implied by this new term are that it is deliberate and that it is taken for the benefit of the “author” (company) at the expense of the user.
Bugs are inadvertent and have no intent so don’t match either criterion.
Google’s so-called “integrity” system is DRM for the benefit of advertisers (and thus Google), not users. Intel’s on-chip security manager isn’t for the user’s benefit and adds security risks.
Encrusting photoshop or Word with a million features or back-compatibility with a mistake made decades ago may make the program worse, but are in service of giving customers what they want. It may make the application worse in some ways but the motivation is pro-customer, regardless of the ultimate result. (Note: I chose these two programs bc I don’t even like either)
> The other two important criteria implied by this new term are that it is deliberate and that it is taken for the benefit of the “author” (company) at the expense of the user.
> Bugs are inadvertent and have no intent so don’t match either criterion.
Yes, good distinctions. Very different from inadvertent bugs or routine technical debt.
That caught my attention too. The misuse is especially strange because the author both cites Doctorow and links to a definition in line with Doctorow's usage, before then using it in a completely different sense
I'm not a prescriptivist when it comes to language; if a meaning evolves that's just dandy.
I do think it's odd to link to a definition of a word as the author does, and then use it differently without explanation (potentially depriving the reader of some interesting insight into the phenomena under discussion, if the reasons are good).
Secondly, doesn't 'misuse'really just mean 'used differently from how most people use it'? In which case, I think the descriptor probably applies, given the word's recent coinage and limited usage
And you know what is meant by "enshittification" in the context being used, you just don't agree with the usage.
Let's not pretend this word has decades of deep cultural and etymological history behind it. It's not a technical term or a term of art, it's a meme. It's something Cory Doctorow came up with in a blog post last year that only a few tech people care about. It's a hipster nerd poop joke.
And because it fits cleanly into a description of anything hipster nerds consider to be "turning into shit," it inevitably will, until people get tired of the meme and move on.
To me enshittification is when the overall usefulness of a piece of software starts going down as more money is attempted to be squeezed out of its users. Agree that it's not a technical phenomenon
This is why I tend to go for the "open" option (eg. open source software) wherever possible.
Not necessarily because it's better. But because the incentive to enshittify it is lacking.
Closed / proprietary ecosystems tend to be nice for a while. And then disappear without warning. Or turn into crap, like a big sinking ship taking all its users down with it.
FOSS / open platforms tend to fork and/or evolve. And (if they survive) improve over time. That I can deal with.
> I think of enshitification primarily as an organisational/business phenomenon rather than a technical one.
I will second this. I can't count how many projects were started with a clean design and clean early releases (what worked well), to be later messed up with business changes because the original (business) idea didn't work. And those sudden changes are expected to be implemented usually "yesterday", which adds additional momentum to the project's downward spiral.
> to be later messed up with business changes because the original (business) idea didn't work
If the point of a project was to run a business, I would argue the project was messed up from the start if it could not adapt to the changes. Very little software survives contact with the real world.
I think software should always be written with a known half-life. Code I write for some one-off script is incredibly different than code I write when I expect the code to outlive me (eg in an opensource library that people depend on).
Scrappy prototype code is written with developer velocity as the most important property. I’ll throw everything in one file. Stick to tools I know well. Copy from other projects. Use crap variable names and generally make a bit of a mess. Something made fast can be remade fast if it’s wrong.
Code you expect to last a long time should have tests, CI and documentation. If I’m serious, I’ll often rewrite the code several times before deciding on a final design.
A good engineer should be able to switch between styles based on the needs of the project. And a good company will make those needs explicit to the engineering team and they’ll be transparent and trustworthy when expectations change. (Eg if a prototype is going to stay in production long term, they need to tell the engineering team).
It’s fine for businesses to pivot. And it’s fine for engineers to make quick prototypes to test a business idea. But everyone needs to be on the same page the whole way along.
If you buy apples only later to realize you need oranges the apple project wasn't a failure. Part of running a business is trying what works. Now if you were unsuccessful finding apples than the apple project failed. The overall business might fail but the apple project went off smoothly
I've once worked on a large and old Scala-cats codebase (i.e. Haskel-style, the whole program as an IO monad). I'd say we (the team) hardly introduced any bugs. The language made making whole classes of errors less likely, and the team's disciplined approach handled the rest.
Am I the only one who thinks that the term sounds far too general given the apparently narrow interpretation that's intended by it? When first seeing the word, it literally just sounded like it meant "the process of becoming shitty", and I never would have guessed that it was intended to be so specific. It seems unfortunate that the term was picked primarily for how evocative it is rather than for its clarity.
The En prefix means “causes to be”. Things can become shitty but “en” changes it to a deliberate act. Thus shitification is process of becoming shitty but enshitification is the act of making things shitty.
So, it’s not as general a term as you are implying and definitely the usage in this article doesn’t make sense.
Why does "causes to be" necessarily imply intention? I don't think that's nearly as definitive as you're claiming.
> It’s not as general a term as you are implying
My point is that even if the term isn't "supposed" to be general it definitely sounds like that to a non-trivial number of people, which is why these sort of discussions crop up almost every time the term gets mentioned in comment threads on this site.
Sure, you can make up any new term and define it how you want, but people will carry expectations based on the word you pick; if I had been the first one to coin the term "enshittification" and defined it as "the process of making something better", pretty much everyone would be confused, and for good reason. The mismatch here isn't as severe, but almost every time I've seen the term mentioned in comments on this site there's a discussion in the replies litigating whether the usage of the term was correct, which is a tell-tale sign of there being inconsistent expectations around the meaning of a term.
As an aside, it's interesting that the phenomenon of expectations around the meaning of terms goes beyond obvious relationships to existing known words but even to the way a word sounds: https://en.wikipedia.org/wiki/Bouba/kiki_effect
> I think of enshitification primarily as an organisational/business phenomenon rather than a technical one.
Codebases can enshitified, this is a deliberate action. Having a zombie corpse codebase where bugs grow faster than features is not necessarily enshitification.
An enshitified codebase is created when a product person or a TL spends an undo amount of technical debt on new features or even better anti-features.
Yes, the bug accumulation problem is better described as 'degradation'. 'Enshittification' implies deliberately introducing features that make user experience worse while benefitting only the owners of the platform.
Is someone keeping a list of features that qualify for the label? For example browsers like Safari make it very difficult to export bookmarks to another browser like Firefox, even though a simple json file would suffice.
The article could have used the term technical debt. To me technical debt happens not only when new code gets added in a sloppy way, but also when the developer experience gets worse.
It nicely captures a lot of the usually "fuzzier" things of technical debt, like how code even with no changes and no changing requirement somehow accrues technical debt (you get better, your idealized effort reduces).
These tools are not composable once you venture outside of a shell, and if I have to write anything beyond 5 lines, I'm not using one. There needs to be a way to make these applications usable as software libraries, and no, invoking them as a separate process from the main application does not count. This is the reason the functionality of these tools have been reinvented many times over and why they get extra cruft added is because they're actually not composable everywhere that matters.
We need more standard special purpose interfaces. Not unstructured RPC protocols, but very specific "Every plugin of this type has this function that does exactly this" kinds of things.
Linux people seem to tolerate bash partly because they DO have a special purpose modular system. They algorithmically manipulate text in batch rather than realtime mode to produce an output on a regular basis(I'm not sure for what, besides compiling and building, but they all say they have use cases). Apparently it works great if you process lots of text files.
Perhaps a GitHub awesome list. I've been meaning to start one for very common standards in general(18650 batteries, tripod thread, etc).
We have LADSPA and LV2 for audio, shell commands for text, etc, What else do we have, and what should we have?
On a modern Unix-like OS, these tools aren’t as simple as they used to be in the 1970s. The man page for ls alone lists tens of options, corner cases and historical cruft. If those tools were really composable, they wouldn’t need so many options hidden in their man pages.
I don't agree with this conclusion on the number of flags for the GNU core utils relating to it's composability; the flags are natural advanced extensions of what the tool is meant to do, and even without the flags the tools are straight forward enough that the default execution likely will meet the requirements of even new users.
From my own experience teaching newbies about bash, the only hard part about all the flags was getting newbies comfortable with ignoring them at first and just trusting that `ls` lists files in the current directory, `sed` let's you manipulate text in a smart way, `grep` finds things, and so on. Once they got that in their head and understood that the flags just allow very specific operations that you previously needed to write a lot more code for, most people got it pretty fast and could punch out quick shell scripts to help with their work. But the base commands without a slew of flags was still quite useful for these newbies.
Again though, that's just my experience and it was fairly narrow in scope for accomplishing specific troubleshooting tasks, so that may factor into why I saw success here; I don't know if the people I taught further evolved their scripting skills past basic troubleshooting, though I am fairly confident these persons understand the "kitchen tools" nature of the GNU core utils as many ended up writing scripts on their own for situations we never discussed, and the scripts were pretty okay.
Edit: Typo, changed Exceptions to Extensions in the first paragraph
> invoking them as a separate process from the main application does not count
Why not? What is your worry? That starting a process is too slow? That data transfer is too slow? Something else?
I agree that process startup may be meaningful overhead in the rare cases when the actual computation is very fast. But sharing uncompressed data between programs is essentially zero-cost.
Separate processes look particularly elegant and modular to me. They are much easier to debug and profile than bolted-in dependencies, since the running time and input/output of each process are trivial to isolate. Of course, having both a library and a cli interface for the same computational brick is always better, but the unix-style tool is an essential thing to have.
As someone who writes software that does a lot of this:
Process management gets very annoying, very quickly. When you run a helper program soon you have to consider process management. What if it crashes? What if it gets stuck? What if you crash and it remains running? What if the user uses the same program elsewhere? None of this gets work done.
A text stream is an awful API. Many times the called program doesn't intend to provide an API, or doesn't commit to stability. Your code breaks because version 5.0.2 fixed a typo, and you relied on the typo for your parsing.
A text stream is an awful API. Instead of a nice protocol you get to parse lots of text, deal with escaping and quoting. You better hope the called program does it competently. It may well not, then you have a problem.
A text stream is an awful API. No normal program will dump or read a JPEG over stdin/stdout, so if you need to communicate some sort of binary data now you need another communication channel. That may involve commandline arguments, killing, restarting the program, and reestablishing the state. More process management fun.
A text stream is an awful API. You'll find yourself doing things like assembling multiple lines of output into a single coherent concept, and trying to detect where something ends when the program doesn't necessarily provide a clear indication.
A text stream is an awful API. Sometimes programs print stuff before it took effect, and may not ever give you a clear indication of "now it's been applied". You may need to somehow test for it, retry operations, insert wait states.
99% of the effort invested in this doesn't get work done. You're spending it on management that wouldn't exist if you were using a sane API like DBus or similar, where you don't deal with process management, where things are broken down into nice fields, and where the API is intended as an API.
I agree 100% with using processes as the natural unit of isolation.
I also believe we could solve many of the integration issues between multiple tools, by having them all optionally spit out a universal, easy / easier to parse format -- even JSON would be good here. So you could do
ls --foo --bar --json | jq ...
and be able to process the output to your heart's content, without having to do any kind of white space parsing; as a free bonus, you get automatic support for file names with embedded white space...
> Separate processes look particularly elegant and modular to me. They are much easier to debug and profile than bolted-in dependencies
Debugging async rocesses is definitely not an easy task. As is managing them in systems that have no management and supervision capabilities (aka nearly all of them).
What happens when your process errors out? Gets kille dby OS? Gets stuck? Fork bombs?
They are also quite expensive to start in OS terms.
Performance is not what concerns me, it's design. To what end is it useful to have distinctions between libraries and applications? If I have a JSON parsing library, it might be nice to run one-off scripts that grab a certain value and pipe it somewhere else. It's also useful to deserialise in complex manners inside an app of my design. My belief is that programs whether they are intended to or not usually end up being composed with other programs so on that basis, having the distinction of bin/lib doesn't serve a useful end to me at least.
I agree. The reason small tools are simple and have few bugs is that they don't do the whole job! They say "here are your modules! now you do the last bit of programming to integrate them and do all the integration testing please. good luck!"
What the article says about the hub and spoke model, singing praises of Obsidian, couldn't these points just as easily apply to Eclipse (the open source editor)?
Eclipse isn't universally beloved. I haven't been following Eclipse development very closely of late, but there were versions released in the early 2010s that were universally derided.
This is one example of cherry-picking I noticed in the article.
Also linear relationships in Unix - ever heard of heard of tee?
Also "enshittification" is turning into a meme at this point, I believe mistakenly applied here.
"Enshittification" was a meme - in the original, non-lolcat sense of the word - from the very beginning. Perhaps what you meant to say is that it is being overused even where it does not make sense. :)
" The model [Unix Philosophy] never successfully made the jump to desktop operating systems. So popular modern programs like Photoshop and Word are about as “encrusted with dubious features” "
You export your phtoshop design as a .png file and you import it into word. There you go.
Windowed desktop programs like Photoshop and Word do offer interoperability, in the form of files. (Which by the way is another way programs communicate in UNIX, pipes are just a way for that communication to happen on memory).
> You export your phtoshop design as a .png file and you import it into word. There you go.
> Windowed desktop programs like Photoshop and Word do offer interoperability, in the form of files.
I think you are comparing apples and oranges here.
Maybe if Photoshop and MS Word used the same dedicated underlying binary/application to read and process the png, then maybe it would follow the UNIX philosophy a bit closer?
"Maybe if Photoshop and MS Word used the same dedicated underlying binary/application to read and process the png"
The unix philosophy is to do one thing and do it well, it doesn't need to conform with expectations about each binary achieving low-level tasks.
What even would be "reading and processing pngs"? Sounds like a dumb technical way to divide responsibilities. I rather have responsibilities of apps be user-defined like "editing images" and "writing documents".
The unix philosophy precedes the development of the UNIX ecosystem, it's anachronistic to understand an OS based on how applications ended up interpreting it.
It's like conflating a constitution of a country for its laws.
To count the number of functions in a rust file you could run:
cat main.rs | grep "^\s*fn\s" | wc -l
This both promotes the "one thing well" model, while revealing its inherent limitations.
How many functions are in this rust file?
/* confuse things
fn fo fp
fn fm fl
*/
fn
main() {
println!("Hello, world!");
}
The above grep reports two, when there is only one.
It can be made to work if everyone agrees to follow conventions, like always formatting the Rust code, and indenting comments which look like Rust code.
But that means pushing complexity onto people, rather than into the tools. Which you could do at the beginning, when complexity is low, but it gets increasingly more difficult over time.
This sort of heuristic bugs me too. We should be able to write language server-adjacent tools which could do this sort of thing as easily as grep. You could do a lot of useful stuff with that, for example to implement custom linting rules:
- Count non-comment tokens within each scope, to look for complex scopes: `foo --tokens --exclude-comments --group-by=scope`
- Count expressions (as opposed to just SLOC) per function: `foo --expressions --group-by=function` piped to `uniq`
- Get the functions and their argument count: `foo --arguments --count --group-by=function`
- Find very short or long names: `foo --names --sort-by=length` piped to `head`/`tail`
Most languages are open source, it should not be difficult to extract their parser and do this stuff. As far as I know, only Babel ( JavaScript ) and Java have a plugin ecosystem, but only related to compiling.
But then Kernigan contributed to awk which does regexes, count lines, and enough other things that there are 500 page books on how to use the tool. Maybe stringing together grep and wc wasn’t always enough.
Of course it wasn't always enough. It was enough to be useful in many situations, and not enough to be useful in all situations. So there's awk. And it's useful in some situations, but not in all. So there's C. (And some will say it's not useful in all situations...)
I’ve been learning PowerShell recently and as I was writing down the good things about learning about a shell scripting language I came to this conclusion.
I normally use Python for things like this, but I have gotten burned more times than I care to admit with simple scripts having dependency issues.
I realized that shell languages are not only smaller languages, but are more limited in scope. This means there are less dependency issues to handle.
On top of that PowerShell scripts are easy to stitch together with pipes, so doing things like parallelizing and sending a job to the background as a task, then checking on it later is a lot easier.
Furthermore the stitching together of shell feels very functional, and because the cmdlets only do one thing getting it to be parallelized is infinitely easier. To this day PowerShell is the only code I’ve ever parallelized on purpose.
I then realized this was all possible due to the self contained nature of the cmdlets, which goes back to the Unix philosophy invented in 1969, about doing one thing and one thing well
The article is pretty average, and goes to great lengths to say very little, with some questionable analogies and examples.
But near the end, there is a gem, which is the introduction of the Hub and Spoke design.
Now Plugin architectures aren't new by any means. But they've not been discussed as much nowadays, and I think the idea of "Hub and Spoke" describes it very well.
Taking Obsidian and VsCode as examples of the unix/microservices philosophy is very bizarre.
VsCode is in many ways a philosophical successor to Emacs and sits on the opposite end. Yes, it has plugins but it is designed from the top down to work in a particular way, and everything integrates with it. VsCode Extensions don't function on their own, they're not composable tools, they don't expose any agnostic interfaces (except for the LSP). They're designed to work well within the existing ecosystem of VsCode.
It's essentially a lesson in systems thinking. Paraphrasing Russ Ackoff, the complexity of a system is the consequence of the interaction of its parts, not the parts themselves. Microservices and unix tools always ignored this, which is why they fell out of favor. I can split one complex thing into a dozen simple things, but that doesn't make my life easier, because then all the complexity is in glueing them back together. And even worse improving one part doesn't mean you improve the system you care about.
Ackoff always used to give the example of a car. You take the best part of all cars in the world an put them together, you don't have a great car but a pile of junk, the parts don't fit. This is also the issue of unix tooling and microservice architectures, people optimize for the wrong thing.
To me, Wirth’s Oberon system has the same philosophy of having small composable tools and being extensible. There’s no distinction between system language and scripting language. Text is also a common way of passing data, but you can use whatever data structure you want. I think of it less as an OS than as an application shell that you shape to your needs.
Could you elaborate? I think your comment is meant in jest, but as both the Eclipse and Obsidian have marketing departments, I don't know how to interpret it.
I think this is an oversimplification. Having something that does one thing and does it well works for simple things that can be composed. Sometimes this is not the case, as the composition would be too cumbersome or too slow.
The example of Obsidian (in my eyes) confirms this. Markdown is a very simple and elegant way to produce simple documentation, but as soon as one needs more it just crumbles.
I agree that the big tools (Word, excel, photoshop, etc) are bloated. These applications have crossed the bridge into 'they do everything about X' long time ago and they work well in their very wide problem space. Just taking excel (or Calc) as an example, I cannot see how using simpler tools I could create pivot tables and aggregate values across multiple sheets.
Its an insightful read and the final graph analogy seems very apposite. But here is the rub: if microservices is a general graph pattern, it includes all the other patterns (likear, hub and spoke etc) as these are all just particular graphs.
In other words, you could implememt unix pipe like or plugin like microservices by applying certain constraints.
Is that relevant, though? An undirected cyclic graph is a superset of all other data structures[1], yet we still differentiate between them and choose the most suitable.
[1] And can be used to build all other data structures. A tree is a directed acyclic graph, a list is a tree with no more than one child per node, etc
The proof would be in eating the pudding, but the thrust of the argument would be to indeed recognize that "microservices" is too general a pattern and one might benefit and make them more usable by developing tailored patterns (imposing constraints adapted to the data exchanges that dominate the application).
Its just a thought sparked by that interesting diagram at the end of the post and motivated by the fact that the REST architecture is also specified as a set of constraints...
Right, but implementing a restricted domain (pipes) with tools designed for a more general domain (graphs), gives up the efficiencies and reduced complexity of the restricted domain.
The efficiency argument depends a bit on the scope one considers. If you only ever need pipes it makes sense to optimize and learn how to build pipes and your toolkit will reflect that. But if you need a variety of patterns you need to weigh in the inefficiency and cost of switching. The analogy would be a multi-purpose tool versus many special purpose ones (and the knowledge of operating each).
This is all theoretical ofcourse, but my sense is that the debate about microservices is colored first of all by the extra complexity of networked computation (which raises the bar for a succesful architecture) and maybe indeed the absence of well defined constraints that would guide people on different possible best practices for segmenting monoliths.
This is why I'm disappointed that more OSes didn't steal the contact list from Windows Phone.
The WP contact list acted as a shell for all person-to-person communication programs, showing a unified history feed of all communications you can read from a given person. Like RSS but for humans.
It seems like the obvious way to do "lots of small programs" on the phone -- one single place for managing your communication with a person, whether it's Twitter, email, StarCraft, phonecall, chess, WhatsApp, teledildonics, etc.
I think it is hard to reconcile this with the reality that every big billionaire dollar app does the complete opposite. Excel, Word, Jira, Salesforce, Facebook, Oracle ERP, etc.
I don't think there is a real contradiction. Once success gives you access to billions you no longer need to work the same way as you had to when you were much, much smaller.
That's why as a small team it is wrong to adopt 'The Spotify model'. And yet, many companies will do just that. At every level of the corporate ladder there are different appropriate sets of tools and ways of working that will allow you to achieve particular results. Once you reach the Microsoft of the 1990's era level size (or Oracle, or SAP) you can ignore some of the truisms from the leaner days and do different things that would have killed a younger version of your company.
Just like an adult can run and jump further and faster than a toddler.
> Once you reach the Microsoft of the 1990's era level size (or Oracle, or SAP) you can ignore some of the truisms from the leaner days and do different things that would have killed a younger version of your company.
I think there are two sides of this.
There’s of course a much stronger need to integrate with established systems, no matter the unnecessary complexity, for those companies. They need to provide stronger compatibility guarantees as well as mitigate risk.
The other side is that once you have that many resources, it becomes less necessary to strive for simplicity and uniformity. The value for those companies shifted a long time ago from engineering to business processes.
If small, focused programs are good, so is concise, focused writing. This piece seems to have a central point that small composable code units are better than large monolithic ones. But it should not take a "12 minute read" to make that point.
Honestly, every paradigm and tool the industry has come up with has been great at identifying and attempting to solve either or one or many issues with software development and maintenance. The UNIX guys did this for C, where when UNIX systems are viewed as a C dev environment, the tools and paradigm are fine. The issue, IMHO, is that people tend to get… religious in their dedication to a paradigm and/toolset. I think the functional and OOP communities have done well too. The issue today is, I think, speed. People move very quickly now and there’s not adequate time for teams to document, bug fix, and grok a code base. If they took enough time for this, they’d be late to market and miss a giant amount of revenue. There’s also just the problem of accretion. All human organizational and informational systems suffer this, and software is no exception. At some point, the size of the code base and the number of developers eclipses any attempt and understanding and control. The extremes of the UNIX paradigm are neither sufficient nor necessary to arrest this, personal discipline is more the requirement.
> "The issue today is, I think, speed. People move very quickly ..."
Companies make people move quickly.
> "If they took enough time for this, they’d be late to market and miss a giant amount of revenue."
They arent worried about markets or revenue, the shareholders are.
> "At some point, the size of the code base and the number of developers eclipses any attempt and understanding and control. "
This is the result of corporate software development. Its software is shit because no one is allowed to sit down and think about problems. Just close the jira ticket and move on to the next one.
To seek refuge, find a software community that cares and participate there.
As a developer I can sort of see the appeal. I often have tasks where I can compose programs.
But as a user I just want the software that solves my problem from start to finish even if it does a worse job and costs more. Even if there are N pieces of software that solves my problem in N steps, that composition isn’t something I generally can or want to do. I just want someone to make a big ball of mud app that does exactly what I need. It’ll try to do whatever everyone else needs too, which is why it is a buggy ball of mud. But it’s still a lot more attractive than having N separate steps and doing any sort of composition myself. Because that invariably requires knowing about files, data formats compatibility between components .
The Unix principle works well in Unix because the people using it are computer people, perhaps even software developers. In the age of everyone using computers (so 1990 and onwards) this breaks down horribly. The big ball of buggy mud do-it-all Software wins hands down every time.
> But as a user I just want the software that solves my problem from start to finish even if it does a worse job and costs more.
Well, if you're willing to pay, there's a lot more options.
You correctly identify the unix principal only working because the people doing it are computer people. It's designed for a situation where you are the one that has to solve the problem, not one where you can just pay your way out of it.
And while that might break down post 1990 once you start getting non-computer people using computers, I think your analogy breaks down post 2009 or so in the age of ads and everyone expecting everything to be free.
May be that is the key to the problem. We should have a programmer os vs a user os, sort of like macOS vs the underlying command interface. Not like dos to windows. The unix or linux is designed by programmers for programmers. User should pay their way out by paying the programmer.
Obviously there is some in-between cases… occasional use is ok.
I feel like this is misusing the term "enshittification" that Doctorow coined, which mostly described the phase where a popular product has to start generating value for its investors and starts exploiting its users.
The difficulty with the plugin model is that it tends to be brittle with regards to change. You need to have very well-abstracted, succinct and stable interfaces that don’t prohibit changes in the main application too much. It’s also difficult to prevent plugins from relying on various implementation details that aren’t part of the interface contract. If plugins have to be regularly updated for new application versions, that only leads to constant churn and abandoned plugins. It means the architecture effectively isn’t truly modular.
I've been having a delightful time these last couple years working with Jax (the machine learning thing), which takes the DOTADIW stance. The philosophy is to get rid of frameworks, and replace them with libraries developed with DOTADIW in mind. This leads to better separation of concerns (amongst the various libraries), fewer leaky abstractions, sane-r, better thought-out APIs, and so on. All kinds of good.
At my last company, we had developed years ago an API layer that allowed outside developers to build plugins to our web app. We also created a market place in which these developers could host these plugins.
It was an absolute constant headache trying to tease out the bugs that these plugins would introduce. They often slowed the app down to a halt, and because they’re so opaque, users would blame us for the slow experience. I suspect it was because of the fact that their performance impact was so difficult to hold accountable, plug-in developers were never incentivized to make their plugins performant and just hacked them together.
And this is universal. Every app I’ve used with a sufficient amount of capability for extensibility will invariably suffer this problem. Slack and Figma also suffer from this issue after you’ve installed a bunch of plugins. They get really slow and bug out.
It is inevitable: the more code you add to something, the worse it gets.
The point about "do one thing" is that it takes a lot of rewrites to work out what one thing to do, and where the boundaries are.
Unix tools have developed in tandem in the unix ecosystem over decades of rewrites. They rubbed the rough edges off of each other.
The major failing of most "systems design", architecture and companies is a reluctance to replace what is there with something better. No one likes a rewrite.
And this affects microservices too. Once a microservice exists it's boundaries are defined. sure you can rewrite it to be more peformant but you cannot chnage its interfaces or it's scope - something is depending on that.
The ability to reform those interfaces is what makes it possible to grind down and find the one thing to do well.
And that takes lucking into the right fundamentals and being willing and able to make large scale changes.
"Move fast and break things" might actually be excellent advice
This hub-and-spoke architecture is the core of acme, the editor of plan 9 aka "unix if it had evolved using its own mantras and didn't choose pragmatism all the time". That editor has extremely basic functions built-in, and more complex operations are done by external processes plugged to acme and accessing all buffers directly, with a good, mandatory usage of the mouse. Unfortunately it doesn't have all batteries included by default, so it's not very famous today.
There's irony in using K'nex as an analogy. I hated those toys as a kid; sure, they're flexible in all directions, but all that flexibility means there's no structural integrity. To build something that doesn't twist and collapse, you have to overbuild all the connections and consider all the possible vectors of movement (gravity or applied) that need to be guarded against.
There is no contradiction between monolith and DOTADIW. The code unit is the function, so each function does one thing and does it well.
I don't get why microservices is the opposite of large code base : having the code splitted in different files or in different repo, changes nothing about the sum of all lines of code, no ?
Can't say I love Obsidian being held up as the best we can do for program design. It's closed source. It could be an elaborate system of duct tape and string holding the thing together.
I mean it likely is quite good based on their velocity and quality. But if we're to learn anything I'd really like to see its source code.
It’s okey to prefer open source over proprietary applications. But obviously we can still learn lessons about interface design from proprietary systems. We don’t need the source code in order to observe those aspects of the application.
(And if we need understand how to application operates under the hood, it’s entirely possible to use tools like IDA and Ghidra.)
Then there is something I miss in the article maybe. What is the lesson there? What is indeed so special about Obsidian, a platform I'm completely unfamiliar with, that is different from any other platform with a plug-in architecture?
With plugins and extensions we've gotten to the point where "Create New Document" in Photoshop is an extension.
Now that electron apps spawn their own browser process, then act like the central GUI hub and "do that right" - it seems to be quite the opposite from "do one thing and do it well" from my point of view.
But ... firstly, let's stop with the Unix worship. They didn't deliver, even in the domain of the terminal. My copy of `ls` has at least 40 flags (where I stopped counting). Secondly, people have tried composition on the desktop. That's what CORBA, COM, and OLE were all about. They all kinda sucked for various reasons. Finally, the place at which the author arises has been around for as long as Unix: it's Emacs.