Systemd pissed off many people. For me and most users, the transition was fully imperceptible. With the difference that systemctl tools seem to work better for what it does than the many sparse different tools it replaced.
This is the comment I came to upvote and agree with.
I am considered a Linux expert in my field but the truth is that I really don't know how systemd internals work well enough to read that review and fully understand it. The review did teach me some things about what Units really are, more than just services and timers, but other than that it was too deep for me.
And yet I work full time with Linux systems, building services on Linux systems for major clients and government contracts.
From my perspective as a user and operator of Linux, systemd has been nothing but good things.
A bunch of very large bash scripts that were entirely different in each distro, had tons of races and were much slower to boot (one of the reasons why systemd became so popular and there was popular pressure to switch to it is that it would speed up booting by several seconds).
Also upstart, which was actually pretty good, but systemd was more ambitious.
I agree with most of this but have to disagree about these shell scripts being slow. On Debian most of them were written for dash instead of bash and they were executed in parallell based on magic comments which defined the dependencies. It was still a mess but a mess which was fast, not as fast as systemd but fast enough for most users.
Additionally Upstart was not good. It has some good ideas but its approach to dependencies was generally regarded as unintuitive. Writing services in systemd is much more obvious than writing for Upstart.
For the init system, the most common predecessor was a combination of sysvnit and "initscripts" (a shell script for each daemon on your system). Writing the init scripts was simple for the most basic cases but quite complex for complete, bug free case coverage. A big appeal of systemd for packagers and sysadmins was the relative simplicity of unit files. Now we just have to fill out a simple boilerplate INI style file for most services.
There is strange false history that systemd replaced sysvinit, which occurred only on Debian, because Debian was the last major system to discard sysvinit.
Gentoo had replaced sysvinit with OpenRC in 2006; Fedora and Ubuntu were using Upstart in 2005. systemd primarily replaced Upstart and systems that used Runit or OpenRC tended to have stayed with them.
Frankness be, this false history seems to be crafted in order to compare it with very outdated technology to make it look good. Obviously systemd is better than sysvinit, technology that almost all systems but Debian had discarded long before systemd was even started. The issue is whether it compares favorably to Upstart, runit, and OpenRC, and to the first I would say yes, because Upstart was full of very ugly hacks and didn't deliver it's ambition. In the second two cases, their mode of operation is so different that it's hard to compare. runit was inspired by daemontools and is very simple and effective and works in a very different way from most models of service management; OpenRC is backwards compatible with sysvinit and is essentially a large library so that init scripts typically need nothing more than something such as:
Or something similar to work. I forgot the exact syntax but it works similar to simply assigning some variables and it handles the rest, but it still gives one full access to the shell to use whatever shell functionality one wants.
Yeah OpenRC is pretty great and much lighter than systemd. It's way better than sysvinit. Less daemons running than systemd and it doesn't try to be more than just an init system. I use alpine on all my servers for this reason. And because it's so minimalistic as a distribution, I can just install what I need and nothing else at all. On a desktop system with the OS being suspended, USB devices coming and going systemd makes more sense I think.
But on the desktop I use FreeBSD which has a different system again.
By the way alpine is also working on a full service manager init system (like systemd) but more minimalistic, based on s6 service manager. Looking forward to seeing how that pans out.
When something comes from the alpine team I have a much higher confidence that I'll like it than from RedHat :) I don't care for Gnome either as it has become so opinionated.
Misleading in that Ubuntu and Fedora did use Upstart but not in its native mode, they used it in its sysvinit compatibility mode. This is almost identical to sysvinit, so from the point of view of a user or a package maintainer it was functionally barely distinguishable from sysvinit.
OpenRC was considered by Debian, but 7 out of 8 members of the technical committee preferred both systemd and Upstart to OpenRC (Ian Jackson preferred sysvinit over OpenRC and "further discussion" over every other option).
There was also uselessd, which was the systemd init system without the rest. https://bitbucket.org/Tarnyko/uselessd/src/linux-devel/ It was a joke, but nonetheless if you like systemd's init system them you are probably going to like uselessd, because that's what it is - systemd's init system without the rest.
At 82k loc it's huge for what it does, but nonetheless is order(s?) of magnitude smaller than systemd. If it was active I'd trust it more that systemd on that basis alone.
I can't say I'm a fan of systemd's event driven approach. A system whose state depends on it's configuration plus past state is usually a nightmare to debug and control compared to a system who state is dependent on it's configuration alone - and that's how it has worked out for me and systemd.
The simple cases work fine of course but systemd's power over what came before it is it's dependency relationships it allows you setup. But they are the very things that break in weird and wonderful ways when the events it's reacting to fire in an unexpected order. If you could set them up reliably without much thinking it would be a big win - but as it is, not so much.
> Incorrect in that openSUSE and Arch are major distros and migrated from sysvinit to systemd.
OpenSUSE also had upstart in the meanwhile but continued to support sysvinit as an alternative.
Arch Linux had sysvinit as it's pid1, but it's big sell initially was that, similar to Crux, it had a custom BSD-style init script setup and did not use sysvinit's booting and rc mechanism.
OpenRC systems typically also continue to use sysvinit's pid1 to this day, which is not the complaint people have with it but the impossible to understand script sand lack of dependency management of it's service management because it was never meant to do service management which was a bit of a hack.
> Misleading in that Ubuntu and Fedora did use Upstart but not in its native mode, they used it in its sysvinit compatibility mode. This is almost identical to sysvinit, so from the point of view of a user or a package maintainer it was functionally barely distinguishable from sysvinit.
There is no such compatibility mode as far as I know. A big sell of Upstart and OpenRC was always that it was backwards compatible with old-style sysvinit-style scripts even though new ones were written in the new style. There is no global mode as far as I know but no doubt many of the olds scripts remained.
> OpenRC was considered by Debian, but 7 out of 8 members of the technical committee preferred both systemd and Upstart to OpenRC (Ian Jackson preferred sysvinit over OpenRC and "further discussion" over every other option).
That would not surprise me. It's not what Debian is looking for.
openSUSE 12.1 defaults to systemd (see previous comment), with sysvinit as non-default option.
So Upstart was a short-lived non-default option.
> A big sell of Upstart and OpenRC was always that it was backwards compatible with old-style sysvinit-style scripts even though new ones were written in the new style.
You are right that there are no distinct modes as such, my memory is quite fuzzy as it's been some years, and I wouldn't rely on anything I say. And it looks like Ubuntu (unlike Fedora) did replace a lot of init.d scripts with native Upstart jobs.
I can't remember what the problem was with mixing init.d scripts and native Upstart jobs, or what impact it had, maybe it was not important in practice.
The Debian wiki has this item, but, well, it's a Wiki, so perhaps it's not the most reliable source.
Can't set proper dependencies until everything has converted from SysV init files to Upstart jobs.
And we can excuse Debian for that, because it was an OS of choice for those who plan to run things for a decade or two. If Debian included modern web frameworks into the distro, most of them wouldn't make it into stable.
> The issue is whether it compares favorably to Upstart, runit, and OpenRC
I would add S6 to the list, as this would be the better challenger to systemd.
That said, I use OpenRC daily on Alpine servers with little to complain about.
is that 2006 number correct re: gentoo? Although i wouldn't start by comparing against openrc, but rather the previous uhmm "baselayout" init system. (sorry, i haven't used gentoo in a long time).
The previous gentoo init system was already better than sysvinit and initscripts.
No, it is actually incorrect, all numbers are incorrect by one year it seems. Gentoo switched in 2007, and Ubuntu and Fedora used Upstart since 2006.
And why wouldn't you compare it with the systems it actually competes against and replaces rather than against something that was barely used any more before systemd even stated development? The comparison with sysvinit, suggesting that those that do not use systemd advocate sysvinit instead, is simply a flagrant straw-man designed to make systemd look good.
OpenRC was a C rewrite of what was previously a bunch of bash scripts, but they still implemented the same rich dependency and function-based init logic; it was miles ahead of Debian's init.d scripts before it became OpenRC. OpenRC largely made it faster, and over the years the configuration became (optionally) more declarative and gained features. The actual OpenRC transition was largely seamless for users, because the init script syntax didn't change.
So the year you're looking for isn't when OpenRC was introduced, but rather when Gentoo started using the OpenRC style of init script, which happened much earlier - it's definitely been like that since I started using Gentoo, at least since 2003 or so.
It's actually a misconception that sysvinit is the whole init system; sysvinit is PID 1. Gentoo still uses sysvinit for PID 1 by default. What matters is what your service manager is - that's OpenRC here.
>And why wouldn't you compare it with the systems it actually competes against and replaces rather than against something that was barely used any more before systemd even stated development?
I would say that's a meaningless comparison and those other systems were never considered as real replacements. At that time I don't remember OpenRC getting any traction outside of Gentoo, and the distros using Upstart were already unhappy with it. Adding to that, systemd explicitly was targeting distros that used a lot of sysvinit scripts and was intended to be used as an upgrade for them. It's largely succeeded at that.
Take a look at OpenBSD. They managed to sidestep all this drama, and still ship a sane, well-documented init.
If you spend an afternoon installing it and configuring a few network services, you’ll already know most of what there is to know about how init used to work.
It used to be extremely elegant and robust.
The main flaw in the design was that the “service” files were written in a Turing complete language, which meant that distribution maintainers could ship things that were arbitrarily terrible, and many chose to do so.
(Edit: Also, the organizations that somehow couldn’t figure out how to use and maintain traditional init systems were the ones that produced the current generation replacements and pushed them so hard.)
> Also, the organizations that somehow couldn’t figure out how to use and maintain traditional init systems were the ones that produced the current generation replacements and pushed them so hard.
This is revisionist. The organizations that built the current generation were those who had to support those who couldn't figure out how to use and maintain "traditional" init systems. Distro init scripts were (usually, and at great effort) decent, whether from Red Hat, SuSE, Debian, etc. (just don't ask too hard about why they were different). But the third-party (often, but not exclusively, proprietary) software vendors looking to support even a single distro were godawful. Eventually the support burden for e.g. Red Hat got large enough it made more sense to replace it wholesale than keep mentoring the latest batch of e.g. Oracle interns in how to write half-decent shell scripts.
I guess you've never had a reason to check what processes are doing what on your systems. I've never been comfortable with the mess you get upon opening something like top on a systemd managed OS. It isn't just the aesthetics of it... it is difficult to reason through the pages of tasks and it creates a lot of low level performance noise (preemption, task switching overhead, etc).
For example, one of my Void flavored machines is using runit: 11 tasks; but a freshly booted headless Debian using systemd: 95 tasks... and that is with a massively slimmed down, purpose built, kernel. Abstractions atop abstraction... the amount of waste in the pursuit of generality has to stop - if this silicon shortage doesn't do it nothing will.
A normal user will never really link the features they see or do not see back to the init system. I'll draw a parallel with the worst subsystem being cut out of modern Linux - X Windows.
As far as I can tell Linux is missing good macro systems that can record actions taken by a user, then replay them. For loops for non programmers, basically. This isn't because the idea is radical, but because implementing the vision historically needed a programmer to understand X - which is too hard. There is a hidden link between the subsystem being awful and programs not existing.
Similarly, the monoculture created by systemd is influencing how the future of Linux evolves by making some things easier and others harder. End users won't be able to detect those influences though, and unlike X's problems systemd is likely a net positive. Systemd easily improves on the old init scripts.
Such a thing doesn't happen if the thing offered isn't an improvement over what was before.
As a matter of fact, this doesn't even happen if the thing offered is only a small improvement. We have seen this time and time again in Programmig languages: Exciting, clever new things that offer real advantages vanishing into obscurity because while they were better, they were not better by a large enough margin.
Diversity is great if it offers advantages. Different kinds of RDBMS are a good example. Different kinds of shells. Rust being there as a C alternative, Go being there as a Java alternative, Julia as a possible Python alternative, WASM instead of JS. All great.
But keeping something mainstream just because its different, adds no value. People can and do still use other init systems, but I am very glad I no longer have to wrestle 400-line sh init scripts on my production servers.
> Such a thing doesn't happen if the thing offered isn't an improvement over what was before.
It does happen sometimes. GNOME3 and GTK3 didn't offer much to Linux desktop users, yet they were adopted.
And improvement is subjective. Maybe systemd made maintainers' lives better. But it also introduced big nasty software full of bugs that discourages hacking by users. Many users consider that a deterioration of the Linux ecosystem.
I'll take wrestling a bug in a 400-line shell script to wrestling a bug in the systemd C code and compiling my fixed version.
> But it also introduced big nasty software full of bugs
If a bug happens in this code, everything is affected, everyone screams at the same time, it is obvious there is a bug, and once its fixed, its fixed everywhere and for every service.
If there is a bug in one of the old init scripts, it may go unnoticed for several years, until suddenly someone needs service Bar depend on Service Foo, but Bar cannot figure out the state of Foo correctly, and good luck to whoever has to solve this: unless the combination of services is a popular one, whoever has this problem, is likely the first to encounter this, there is no general outcry because nothing else is affected by it. A fix may not be forthcoming because the behavior may not even be considered a bug (the maintainers of Foo and Bar may simply disagree how some service state is to be communicated), and usually someone has to trawl through a pile of sh scripts (because backward compatible all the way to the 90s it has to be!) and implements a fix, which then gets destroyed by an update when the guy who fixed it is on holiday.
As someone who has been that exact guy, I have to say sorry but no, I don't want to do this any more.
Yes, but often only for major bugs that are deemed worthy of developer's attention. In practice it's not always so great. Read systemd's bug tracker, not all user-reported bugs get fixed to those users satisfaction, neither by redhat nor by systemd. Systemd is too big and developers refuse to accomodate everybody.
Shell scripts are tunable to the specific system. If a rare bug is in a shell script, I can usually find and fix it, unless the script is insane, in which case I can write a sane one.
If a bug is in systemd, I can't easily fix it myself. I now have to fight upstream to recognize my issue and fix it for me. I don't have a good experience with this. Nowadays I'd rather write a small script to work around that bug if possible. Much faster and much more effective.
>If a bug is in systemd, I can't easily fix it myself. I now have to fight upstream to recognize my issue and fix it for me.
That's not true. You can do as you would with the shell script and patch it in your local copy. If this is too much hassle then that shell script workflow shouldn't be broken by systemd and you can always fall back to it. It's still possible to run shell scripts as systemd units. You could also run another service manager as a systemd unit as another workaround.
Also, in my experience, if you're fixing an actual crash, those patches are really likely to get accepted. Upstream appreciates that, I haven't seen them fight anyone over an easily verified issue. If you're submitting giant 10,000 line patches that change the public interfaces and cause regressions, that's a different story.
>GNOME3 and GTK3 didn't offer much to Linux desktop users, yet they were adopted.
Are we thinking of the same thing? GTK3 had a much better theming system, new widgets, support for high DPI displays, improved rendering, support for animations, and a lot of other things. GNOME 3 had a much more streamlined design too. You may not have liked them but there were plenty of new features to offer.
>I'll take wrestling a bug in a 400-line shell script to wrestling a bug in the systemd C code and compiling my fixed version.
I wouldn't, once you fix the bug in the C code it's fixed for everything. The bugs in the shell scripts keep needing to be fixed over and over again, because copying code around in shell scripts is really common. There is also a line to be drawn where low-level code should be written in C and I think this is well past that line. Service script code is usually privileged and runs as root so the utility of making it hackable seems pretty small. I expect you'd agree running bash scripts inside the kernel is a bad idea too.
That was mostly feature expansion. I meant improvement of existing workflows. Maybe high DPI would count. I didn't have high DPI monitor back then, but those who did maybe were pleased.
Fixing a bug in shell script, even if 30 times, is still much easier than downloading, understanding modifying and compiling and deploying 4000 files and 1.6 million lines of systemd C code. There is an obvious cost/benefit factor here.
If systemd fixed just init and basic service management in correct and minimalistic way, maybe fixing its C code here and there by power users would be rational. Not with the ever-expanding pile of prototype-level software we got. I'm not reading that code, not even for money.
>That was mostly feature expansion. I meant improvement of existing workflows
For desktop app developers? There were a lot of API refinements and enhancements to the widgets too, I didn't mention them all because there's far too many.
>4000 files and 1.6 million lines of systemd C
Just FYI these figures are incorrect, you're likely including the tests, documentation and data files. Running cloc on the current systemd code base I get around 2100 .c and .h files and 520,000 lines of C. That also includes all the other components too, not just the service manager. The service manager is a small fraction of that, it's only around 150 files including the headers.
>There is an obvious cost/benefit factor here.
From the perspective of a distribution I think it's much more cost effective for them to fix this once in C. To you what looks like just 30 shell scripts can quickly multiply into the thousands when you have to consider this is across all users of that distribution who might all have the same buggy set of 30 shell scripts but with different ad-hoc fixes applied.
>If systemd fixed just init and basic service management in correct and minimalistic way, maybe fixing its C code here and there by power users would be rational.
I think if you actually look at it, you'll find this is already what it does. That's what I saw when I actually took a few hours to familiarize myself with the code.
> systemd became the de-facto default init sys because Distro Maintainers have chosen it.
Because they were forced to choose it due to vendorlock and many other projects became dependent on systemd. Unsystemding everything would be very expensive.
> Snap and flatpak are essentially just package managers, I can't see why you would expect those to be worse than any other package manager.
Those are even worse than tarball bundles. Main purpose of both overengineered projects is to tie users to vendor's services (snapcraft, flathub, e.t.c.). Centralized delivery is outmoded, future is decentralized distributed distribution.
Both package managers deprive me of control over my system.
I don't like the aggressiveness with which these projects are enforced. When Let's Encrypt switched to Spat as the only official distribution of their tool, I had to switch to alternative tool acme.sh instead of installing bloated snapd on my server.
Under some circumstances (not all), GNOME has a dependency on logind. If you're in one of those circumstances, you can use elogind as a drop-in replacement. That's far from having a dependency on systemd.
>Main purpose of both overengineered projects is to tie users to vendor's services (snapcraft, flathub, e.t.c.). Both package managers deprive me of control over my system.
> Similarly, the monoculture created by systemd is influencing how the future of Linux evolves by making some things easier and others harder.
While I agree with the broader point about (any) monoculture, I don't think that this can be blamed on systemd.
Systemd did not itself create the monoculture. When it came up, it was one of a multiculture. Distro maintainers then chose it en masse so that it became a de facto monoculture.
I'm not saying that necessarily makes it the best choice, though.
On a slightly different note, I find such comments funny in a way, because one of the reasons given for Linux's lack of market share on "the desktop" and, broadly, as a platform for "enterprise" software (think Adobe, etc.) is because the ecosystem is so fragmented. So, essentially, the platform isn't considered because it's not a monoculture (as would be Windows and Mac).
it's kind of like locked bootloaders and pentalobe screwdrivers for organizing a unix system where most users were used to easy to inspect and modify scripts and single purpose programs .
yes, it's still open source, but modifications and inspection are far more complicated creating a much higher barrier to entry.
in the earlier days when it was still buggy, even linus himself was screaming about not being able to easily take the case off of pid 1 when it misbehaved.
I have to hard disagree on this one. As someone who had half a foot in sysvinit, half in upstart, and half in systemd (don't try to do math with those numbers), sysvinit was always the hardest to use by wide margin. It was ridiculously complicated to get anything nontrivial to work. It was error prone, and the number of corner cases you had to think about was extreme. And the worst part was I kept having to relearn it each time because I did it infrequently enough to forget what I'd learned the last time.
Personally, I was more of a fan of upstart then systemd, but both were a massive improvement over sysvinit in terms of offering an actual interface with abstractions for the things you wanted to do. This is again from the perspective of a user, but that's where (in my opinion) most of the cost/benefit was anyway.
It's not like systemd is written in deliberately obfuscated C or something inscrutable altogether like Brainfuck.
A monorepo containing well-organized boring C, developed on github of all places, is the polar opposite of raising the barrier to contributing vs. what was replaced.
Of course, but I would not like to have to patch it to modify something that on other systems can be modified with either a configuration option, or a shell script.
To me, the most poisonous example that contrasts the traditional design with Freedesktop's design is acpid contrasting logind. In the former case, an acpi event is sent, and acpid simply executes a file such as `/etc/acpi/action/lid_down.sh`. That file can contain anything the system administrator wills, from a simple script that does nothing more than `systemctl sleep` to whatever complex task he could desire.
The last time I checked the situation, logind had replaced this powerful, and simple mechanism with a configuration that offers nine options to execute when the lid is pressed, none of which involve executing a simple file. Of course I can edit the source code and recompile to allow it to do what I want it to, but I would far rather simply quickly alter a simple shell script to do so.
And luckily I can, because I do not use logind but acpid, but that does not change the ire I experience when dealing with many Redhat and Freedesktop people who make it very clear that I should be using logind, that it's better, and that they create artificial dependencies on it to as Poettering calls it “gently puh” me in that direction.
And this means we can’t do things like force disconnect an eGPU or whatever on acpi events.
I think I figured out my issue with systemd. It does not apply Chesterton Fence when redeveloping things… and how could it, when it redevelops so much.
Maybe it’s fine if it’s all you’ve ever know (or you never hacked on your system) but those that did have lost something.
The situation is that such hackers and tweakers rarely use systemd, for these reasons.
They are simply annoyed by A) it being pushed on them; B) people often acting that they are irrational for not liking it while it does not what they expect and need; and C) discussions with Freedesktop developers that, frankness be, reveal they live inside of a very strange bubble and have no idea of the use cases outside of it.
This is software brought to you by the minds who brought you. “I have no idea what XFCE is or does, sorry.” or that libinput did not need a way to disable mouse acceleration because everyone wants mouse acceleration.
> libinput did not need a way to disable mouse acceleration because everyone wants mouse acceleration.
You mean you want a flat acceleration curve? You want to accelerate the mouse, otherwise you need a gaming mouse with the DPI set very high and then keep going into the mouse firmware to adjust the DPI whenever you play a game or switch between different (scaled) resolutions. Libinput has different acceleration profile defaults for different devices, but you can just change it to the flat profile which I prefer as well.
I'm an open source hacker and I like systemd. Once I took the time to learn its APIs I found it does everything I expect.
Your issues with XFCE and libinput don't seem to be related to systemd. I can't see how airing grievances against other unrelated projects would be a constructive place to take this discussion. Complaining about open source projects only supporting some use cases is also confusing to me. A lot of these projects are very open about the fact that they exist only to "scratch their own itch".
I have no issues with XFCE. I personally would never use i, but it also leaves me alone and it's developers don't exhibit this mentality.
I have a problem with Jon McCann's famous quote, who was at the time a lead developed of GTK+, who did not know what XFCE, one of the biggest consumers of the library, even was when he made a change that broke about anything outside of GNOME. — What his language suggested was that he lived in a bubble thinking that only KDE and GNOME existed, and that since KDE wasn't using GTK+ it was fine to do this.
The ticket is about GNOME 3 removing support for status icons from the GNOME panel. GTK3 still has the status icon API, although they're deprecated and don't work on Wayland. That shouldn't have any effect on XFCE or any other shell with an old panel that supports the old X11 status icons. I think you can also restore the functionality to GNOME with an extension.
Also, if you check further down the issue you'll see this clarification:
>There should be no change in behavior for non-GNOME platforms.
This is a peculiarity of the GTK developers, not anything to do with systemd or fd.o. I maintained a popular GTK-but-explicitly-not-GNOME application long before the days of systemd and they had the same attitude.
I believe it is common to pretty much all Red Hat and Freedesktop developers.
Poettering has come with very similar claims that betray that he does not understand what people outside of his small circle want and need, the same can be said for Wayland developers, DBus, NetworkManager, PulseAudio and all such other infamous projects.
It's difficult to understand what your complaint is, that describes every software project I've ever seen. You pick a target set of users and then develop for those users. Is there some other way to develop software?
I don't think it is common to find memorable quotes akin to “I have no idea what XFCE is or does, sorry.” outside of this circle. This is a vintage and often quoted Red Hat-ism that betrays their mentality.
QT developers will not tell you “I have no idea what LXQt is or does.”; they do not generally break things that compromise 90% of their consumer base and they do not remove theming because they fear it's existence will hurt the “brand identity” of KDE.
It's dishonest to say similar problems occur outside of the Red Hat circle and that circle alone is commonly criticized on these policies.
If you're trying to prove a point, you could at least mention something that actually caused a real problem instead of taking this one out of context. I have no idea why anyone would keep mentioning this quote or find it memorable, it seems completely insignificant to me. You also seem to be ignoring all the positive interactions this engineer (or any other Red Hat engineer) might have had. I can personally name a lot of instances where Red Hat engineers have fixed upstream bugs that were affecting me. And just to make it clear I'm not saying this to single them out for praise, a lot of other companies fix upstream bugs too. That's how open source is supposed to work.
And you actually could go and search around on old mailing lists to find similarly questionable 11-year-old quotes from Qt developers if you really were interested in digging up more old drama. These things happen everywhere that people go because people don't agree on everything. I suspect you also think that's ultimately futile though, so why keep flogging this particular dead horse? This is still way, way outside the scope of discussion for systemd anyway.
If your problem is with the Red Hat developers, and the fd.o developers, and the GTK developers, and the Debian and Arch and SuSE and Ubuntu developers, and the DBus developers - maybe your problem is not actually with any of these developers but a mistaken idea of how open-source development ever worked?
Or what, is everyone except you under thrall to Lennart Poettering, master wizard?
I'm replying to a post in which you specifically called out Wayland and D-Bus, and your criticism of GTK and fd.o is all over this thread. Your lack of criticism of SuSE developers appears to be because you misremembered when/how they switched to systemd - because they did, relatively quickly.
But that raises the real question: If the problem is systemd, and the problem is so bad, why do you only blame Red Hat and not the every other distribution that switched to it? Do you think they were all victims of Tricksy Lennart, or?
> I'm replying to a post in which you specifically called out Wayland and D-Bus, and your criticism of GTK and fd.o is all over this thread. Your lack of criticism of SuSE developers appears to be because you misremembered when/how they switched to systemd - because they did, relatively quickly.
And you also brought in SuSE, Arch, and KDE, which I never criticized to make your argument that I simply hate everything work, which is quite disingenuous.
> But that raises the real question: If the problem is systemd, and the problem is so bad, why do you only blame Red Hat and not the every other distribution that switched to it? Do you think they were all victims of Tricksy Lennart, or?
I do not, and have never blamed distributions for switching to systemd.
They can use what they want and it's neither their fault nor problem that systemd and other Red Hat projects are known to create dependencies upon one another for ill technical merit.
I have criticized Red Hat projects for creating dependencies on other Red Hat projects for political, rather than technical reasons; the rest is simply your putting words into my mouth to make a straw-man argument function.
I haven't seen any of these supposed Red Hat projects that have dependencies for political reasons. If that were really true then it would be trivial for anyone to remove those dependencies, and there wouldn't be anything for anyone to complain about. Any way you slice it doesn't seem like a cause for alarm.
acpid vs logind is another illustration of the trade offs involved when moving towards GUIfication of the bazar that is (was?) unix.
Basically the proposition is akin to a weapon exchange program: hand over your powerful but dangerous Turing complete configuration language in exchange for nicer looking desktops.
It's what you consider nicer. I consider dialog windows to be a slow and frustrating way to configure anything and on top of that in this case it loses me the ability for my computer to do what I want it to when I close my lid.
And if it simply remained at that, but left me alone I wouldn't be so irate, but I don't feel left alone and “Who would not prefer a better looking desktop?” already borders on this “My way, of the highway.” philosophy of Freedesktop, whose developers all too often tell one that one's subjective likings are wrong and that one is somehow “wrong” or “outdated” for favoring a machine that is fast to configure and does what one wants.
It also doesn't help much that on a purely visual level there design choices look ugly to me, and that these are also the same people that remove theme options lest the holy “brand identity” be compromised, as well as that their design blasts too much bright white light into my face when I'm already tiring my eyes out trying to learn to read a language that's not written in a script I'm accustomed to.
I believe this issue of Turing complete config language vs simpler declarative configuration is one of the basic thing for which there exist no good universal solution.
I face the very same issue in my day job where I'm struggling to cut the right balance between a featureful data manipulation language with which the customer can implement whatever he needs, as long as he is OK with a bit of programming, or a more limited one for which we have a GUI... and that GUI itself has to be tested, and I'm also currently reviewing QA tools to help testing GUIs, and they basically fall in two camps: those that require one to spend a week learning how to program them, and those that are much more limited but come with a straightforward GUIs...
Before systemd the same conflict happened with network-manager: my network setups were usually custom and out of reach from network-manager that I despised and purged of my desktops first thing after a fresh install. Yet with time network-manager supported more and more use cases and I'm now happy to use it. Maybe one day systemd will also make it possible to add the bits of hackery here and there and we will all be satisfied with it.
For now, if so many users have it easier thanks to systemd, which seems to indeed be the case, then it's still something.
> I believe this issue of Turing complete config language vs simpler declarative configuration is one of the basic thing for which there exist no good universal solution.
Indeed there isn't. And people that want simpler declarative configuration files can have them as far as I'm concerned.
My ire is the mentality common of Red Hat employees that I am wrong for wanting traditional Unix-style flexible configuration and their dependency politics to try to get others to switch over. If they just designed their software how they want to and otherwise let it be, such as KDE and XFCE are doing, then they would not be drawing the same criticism, but they have a tendency of coming with a dismissive attitude to everyone who wants it different than they do and insist their way is the future and when people point out that their way does not offer functionality they need either for convenience, or their livelihood they dismiss it.
> For now, if so many users have it easier thanks to systemd, which seems to indeed be the case, then it's still something.
If various Red Hat projects didn't find a way to create dependencies on systemd to encourage it's adoption and systemd was simply there for the people that wanted it but left those that didn't alone, then it wouldn't be so controversial.
What I quoted was “Who wouldn't want [...]”; that's the Red Hat mentality, that everyone who has subjective ideals than they either does not exist, or is wrong.
>If they just designed their software how they want to and otherwise let it be
In my experience, that is what they're doing. Their software just happens to be adopted by more users because in a lot of cases it is better and does represent the future. You have that advantage when you support a lot of commercial users over a long period of time. Red Hat also has a policy of contributing a lot to upstream so that also adds to why they have success with other projects.
>If various Red Hat projects didn't find a way to create dependencies on systemd to encourage it's adoption
I think this is an exaggeration. I haven't seen any projects that have anything more than a trivial dependency on systemd to enable some optional functionality.
Although I like the acpid approach, the logind way is probably safer to abstract behind a user friendly GUI. That's what most people will be using. :-/
And that is probably the motivation for this design philosophy of limiting options that Redhat and Freedesktop champion where more and more, simple, clean, turing complete solutions are replaced by a discrete list of simple options.
But it does not change at all why I dislike it, and why I am annoyed by Freedesktop's pushing and insistence that many keep to these more custoizable solutions supposedly simply because they are “stuck in their ways” and that their philosophy is the future and “modern”, whereas all I see is more hurdles that stop me from effectively being the master of my own machine.
You have a misunderstanding of what Freedesktop is. Freedesktop is just a group of desktop developers that submit standards to each other, they don't really dictate anything. For that reason it tends to self-select for developers who are actively working on desktop environments, and the standards that tend to get submitted there are things that already have some level of adoption among open source desktops. You too can submit specifications to Freedesktop but you'll probably not have much luck with getting something adopted among peers if you're not already interacting with other desktop developers on a regular basis.
"Customizability" is also not the domain of a group that like Freedesktop that publishes specifications. Their purpose is to define some baseline of standard behavior for interoperability. The customization comes from individual projects. No offense but your criticism is pretty misplaced.
In theory it is that; in practice it has become a different face of Redhat because it's mostly Redhat developers that submit these standards there and the body is known to have a certain philosophy that is indeed exemplified by systemd.
They aren't particularly keen on hacking in how they design their software and standards and talking with them is frustrating to say the least.
I can't understand what you're talking about, I'm looking through the list of specifications and I don't see anything specific to Red Hat or systemd. I also see a lot of other names besides Red Hat people: https://specifications.freedesktop.org/
If Red Hat engineers submit something there it's because they think it's useful to the other desktop projects. It's just a collaborative space to put specs. If you have a spec of your own you can just create a new RFC. You don't need to persuade them to change how they design theirs, unless you intend to start working at Red Hat and start maintaining some of their software for them.
“People who write the code get to make the rules” is perhaps the most fundamental part of bazaar development ideology. If you don’t like it, write more code.
Writing more code does not always mean producing better software. Often it's better to just move to an already existing better software. For service management, there are many alternatives already.
> Writing more code does not always mean producing better software.
I didn't say it did! I said if you really buy into the "Linux is about choice", "Red Hat stole our fun OS" etc. bullshit that always circumscribes the populist anti-systemd position in these debates - then you don't really have a moral or tactical position other than "shut up and code."
I don't think that's true - I think there are other obligations to users which are just as important - but these all suggest systemd even more than the "master of my own machine" nonsense.
> you don't really have a moral or tactical position other than "shut up and code."
I don't believe that. People commenting negatively on some Linux software/trend do not need any such privilege or permission or show of stake. I think anybody with Linux experience is entitled to his/her opinion on what should and should not be done on their Linux machine and in the Linux software space, whether they produce new code or not.
That's just re-stating the grandparent comment! If you think everyone is entitled to their opinion then the only tangible way to express that opinion would be to produce new code. What use is the opinion if nobody ever implements it?
I don't believe that. There are many ways to express an opinion about Linux software, e.g. by writing about it, or by rejecting some software, or by adopting some different software.
I don't use systemd on my systems, I prefer sysvinit and openrc, and I sometimes express my arguments for why. Why would I have to write my own init system just to express my opinion that for some usecases, other software is better than systemd?
You've decided the best way forward for you is to write sysvinit and openrc scripts. That's an opinion completely focused around writing some additional code. You don't have to write it all from scratch to be expressing that opinion, you can express it a lot of different ways but that's all it encompasses. Unless we're talking about something else that I missed? Suggesting that something is better than something else is really meaningless without that.
The difference is that everyone outside of Red Hat an Freedesktop isn't creating artificial dependencies on each other's projects to “gently push them” so people outside of it again have to write more code to decouple them again.
They did write the code to fork elongind in such a way that it no longer required systemd; they did write the code so that GNOME could run without logind like every other system; they did write the code to fork of udev from systemd again so that it could run without glibc. — None of these projects suffered loss of functionality.
They're complaining that they have to, because every time, these dependencies that have no technical justification happen, it's always one RedHat project depending on another to create product tying and encouraging adoption.
The rule of capitalism is simply that product tying works in practice, however bad it is for the user and consumer. That's why Apple likes to come with custom cables that only work on their hardware for no technical reason, and that's why logind was incorporated into systemd and came to depend on it for no technical reason.
> They did write the code to fork elongind in such a way that it no longer required systemd; they did write the code so that GNOME could run without logind like every other system; they did write the code to fork of udev from systemd again so that it could run without glibc.
It sounds to me like the bazaar system is working fine qua its own standards, then. Red Hat writes what they need to solve their problems, party X writes what they need to solve their problems, the code is shared, everyone gets what they want.
Except: It sounds like what you really want is for Red Hat to solve your problems, not only their problems. That's definitely not how this stuff shook out historically. And it opens a door you might not want to open, because now you need to ask yourself why users liked systemd, and what obligations you have to support beyond your own needs in your own software.
> It sounds to me like the bazaar system is working fine qua its own standards, then. Red Hat writes what they need to solve their problems, party X writes what they need to solve their problems, the code is shared, everyone gets what they want.
The difference is that Red Hat's problem is not making enough money, and everyone's problem is software not working because Red Hat breaks functionality to make more money.
Red Hat's “problems” are similar to the “problem” Apple experienced that that heir hardware was interoperable with third party hardware, so they made sure to use proprietary cables that only work with Apple hardware.
> Except: It sounds like what you really want is for Red Hat to solve your problems, not only their problems. That's definitely not how this stuff shook out historically. And it opens a door you might not want to open, because now you need to ask yourself why users liked systemd, and what obligations you have to support beyond your own needs in your own software.
No, what I want is for Red Hat to stop product tying, which you conveniently ignored and didn't address.
What they're doing isn't solving any technical problems any more than Apple is doing by using proprietary cables when standard cables suffice.
If you're asking for Red Hat to stop creating integrations between different parts of their distribution, that isn't going to happen. That's one of the benefits you get from being a distribution vendor. Every Linux distribution that grows past a certain size does that eventually, that's how you create a cohesive system. They make money from this because customers actually ask them to do it. You're essentially asking them to kill their core product.
Describing systemd as "product tying" is ridiculous. The product is the OS distribution. Even in the heady days of a dozen of diverging Unix vendors, nobody was pitching their user-pluggable init system. What's next, coreutils and busybox are illegal, because I don't want my `cp` and `mv` from the same vendor?
> the logind way is probably safer to abstract behind a user friendly GUI.
How so? The canonical solution would be to ship a default `lid_down` script that could take the currently-active policy as set by the GUI as an input. But the nice thing is that adding/tweaking policies is a fully-supported operation, you just edit the script and the GUI config mechanism to support them.
I just made an account to respond to this. Shipping scripts is racy and error-prone. It's the wrong approach and is not how any actively developed desktop wants to do things. They want an API they can use from a service, that's what logind provides and that's why it's been adopted. You can also run acpid alongside logind if that's what you prefer, but it will be racy unless you also use logind's inhibitor API.
There is a difference between the best C, and scripts that you can modify in-place.
But, like some other comments have said, I was young enough to dodge the drama. I basically grew up with systemd and I don't feel like I'm missing anything. The biggest problem I have is that Ubuntu Server defaults (or used to default) to having the system wait on network interfaces, which is annoying when I want to build a desktop off of it. And sometimes on shutdown it waits for some daemon I don't care about to quit. Which is probably just a bug in the daemon.
The transition period of any new tech is usually horrid.
But now that things aren't buggy, being unable to modify them in place is a good thing. It sets a clear boundary that this is not how you do things on a modern system, so you can always expect systemd to be the same, consistent, non customized thing it always is.
i'm not going to argue it's a bad thing, i actually really like macs and think they hide a lot of details making time and space for other pursuits, although since you cede user serviceability it's important to purchase a service plan and that you don't try anything that wasn't provided for in the design.
maybe simple composable parts are past their prime. i always loved the simplicity of inetd. it was both self describing and inclusive. you could write an internet service without any knowledge at all of sockets and learn about the system by simply looking at it. but i suppose that's given way to large monolithic daemons these days.
personally though, i always thought the simplicity and the composability of unix philosophy was what bred systems that outperformed their competition in terms of security and extensibility. so many commercial systems reflected apparent power grabs within the corporate structures that produced them... i suppose that when you see foss monoliths like this, it's the same game playing out on a different field.
> i always loved the simplicity of inetd. it was both self describing and inclusive. you could write an internet service without any knowledge at all of sockets and learn about the system by simply looking at it.
You could achieve the same thing with systemd socket activation :)
You can configure systemd to kill lingering applications, it was defaulted to false because some old UNIX tools liked to linger in the background forever.
Anecdata: as someone not engaged in systemd development and one that likes tinkering with hardware tokens and luks, I found that at least the fido2/cryptsetup part of systemd was fairly easy and logical to follow and debug.
I would argue that locked and pentalobe metaphor is not very accurate - systemd is no more locked than most other open source software.
systemd made it possible for me to write new service scripts - system and user kind - in a way I couldn't do before. It seems to have empowered me, because it's actually quite easy to use.
System boot is a hard problem with dependency resolution, service management, etc.
Scripts didn’t properly solve the problem — they were like the easy solution for small ns. But that approach simply could not scale at all.
Systemd may indeed be somewhat more complex, but I don’t think that it would employ many accidental complexity— it is complex due to the problem domain. But imo that is a worthwhile thing to spend complexity budget on.
Right. I have a simple rule: don’t use a GNU/Linux system that uses systemd. It doesn’t respect the Unix philosophy. It’s trying to turn it into Windows.
I’m waiting for Microsoft to buy Red Hat. Failing that, IBM.
Early versions of Emacs didn't have lisp extension languages, which is kind of strange to think about considering the role elisp plays in GNU Emacs as we know it today. GNU Emacs with elisp is something that developed in the context of the MIT lisp machine scene. Sure it runs and was developed on unix-like operating systems, but lisp extensible emacs were never really some sort of embodiment of the 'unix principles'.
The first four Emacs editors, which had something like an extension language: Emacs (1976), EINE (1977), SINE and Multics Emacs (1978). EINE and Multics Emacs were written & extended in Lisp. SINE was using a 'lisp-like' language. A bunch of further Emacs-like editors appeared then.
GNU Emacs later (1984) was based on Gosling Emacs (1981), which was written in C and extended with Mocklisp. Mocklisp was a rather uncomplete Lisp, which was replaced with Emacs Lisp by Richard Stallman.
Emacs development began during the 1970s at the MIT AI Lab, whose PDP-6 and PDP-10 computers used the Incompatible Timesharing System (ITS) operating system that featured a default line editor known as Tape Editor and Corrector (TECO)
I hate the feature and mission creep of systemd and try to use other unixes where I am able; but what you wrote is downright insane.
There were several init replacements in the mainstream before systemd. OSX and Solaris both had their own, and they were as UNIX (officially certified) as you can get.
Also, as long as we're doing non-sequitors; Microsoft is more likely to buy Canonical than they are to buy RedHat or IBM.
The piece of software systemd is most like is launchctl, which doesn't come from Windows.
On the other hand, the biggest "Windows-ism" to reach Linux lately is io_uring. But it's both too technical and too useful to have some populist uprising against it.
systemd user sessions, socket activation, sysusers, dynamic users, systemd-homed, temporary directory setup, transient units, anything talking to the slice or control group APIs, containerization, firstboot, systemd’s whole “preset” system-wide configuration and policy mechanism for admins to say “what services do I want launched when installed and what services do I want to leave stopped until I configure them”, “stateless system” capabilities, and I’m probably forgetting another dozen.
I don't care about the init system - I reboot a Linux system once in a blue moon, for kernel updates. It's the rest of the complication and changes to a system that I had already made the effort to master. I understand that the New Linux is better for Enterprise, so that's what is going to happen. That doesn't mean I have to like Poettering gradually re-writing the entire base of userland. I just want to run simple servers, not a major enterprise.
that's great because the unix philosophy is pretty much a PITA as soon as you get out of grep | cut | tail | tee. I definitely don't want "unix philosophy" for operating system management, that is the wrong tool for the job.
Then you should not use UNIX then. There is another mainstream OS besides UNIX. Some people like to be able to know what their system does and like to use tools on which they can rely on even when things go wrong.
I think it's fair to say that Linux isn't Unix and goes well beyond the design constraints of Unix that were imposed when it was originally started in 1969. That's the era you stick to when you insist on only doing things the Unix way, but there are a lot of other ways to know what your system does and have reliable tools. In current times the most common usage of Linux is Android, which is ostensibly not Unix-like at all regarding its userspace. If you're talking about GNU, that quite literally means "GNU is not Unix".
yes, I don't use UNIX, I use ArchLinux which is a much better operating system for my computer needs. The "proper" UNIX I've used (macOS) really sucked in comparison.
I've used Arch, it's really excellent. But, not now that it's systemd. I use Devuan for my servers, and Alpine for my desktop.
I use MacOS and iOS for internet viewing and A/V. I have no interest in how they are implemented - superficial observation indicates they are a mess. I do ssh into my servers from MacOS - it's comfortable with my laptop actually in my lap.
Linux the kernel doesn’t respect the Unix philosophy either by a huge margin. Also, Unix philosophy is not even the Saint script — it can be good for some problems but terrible for others. In certain cases a distributed between 34 small programs is just a worse approach that will be bug prone and hard to test.
If you want the Unix philosophy, why not use an actual Unix? The BSDs not only all have traditional init systems, but are also more in line with how old-school Unix worked in general (e.g. base system segregation).
The biggest problem I had was that it was difficult to keep upgraded. In those days, it was basically a re-install every six months, plus a merge of any local changes. I understand things are much better now, and I may give it another try.
I like OpenBSD - it's so clean. I use i3 and don't care about audio/video, so my needs are minimal. I use MacOS/iOS to view the internet, but not for development.
It’s a functional improvement over old init, but the thing I hate about it is how unnecessarily arcane it is. It seems “designed for job security” or more likely designed for consulting revenue.
There is absolutely no need for its command names and everything else about it to be so counterintuitive.
I feel the same way about Postgres. It's powerful but it's insanely arcane and full of land mines and footguns. It's great for forcing you to hire consultants, DBAs, or pay cloud providers a lot to run it for you.
Which SQL RDBMS isn't, though? Postgres is a breeze compared to someting like DB2. High-performance relational databases just seem to be an inherently arcane target.
MySQL and SQLite, but the former is not a real transactional database, it's quite a joke compared to Postgres. The latter is a file based database. We use MySQL at work and I am exasperated by it. It's possible that even Firebird is better.
If Postgres is unnecessary arcane, then surely MySQL is that squared, given all the legacy traps? You know, stuff like "utf8" not actually being UTF-8.
Proper transactions, like the ones that aren't closed by create, alter or drop table, proper rollbacks, statement timeouts so that a single stalled statement doesn't block the table until you kill it, subscribe/notify among other things. My issue is primarily with performance under load, where you'll eventually end up with lots of blocked queries and an unresponsive database and the only way to solve it is to throw hardware resources at it.
Do yourselvs a favour and pick Postgres for new projects. If you properly vacuum your tables you'll probably be fine.
How do I list what's running? I have to Google that every time. How do I list units? What are units? What the fuck are... and so on. The output is hard to read. The files are all over. The directory and file names are counterintuitive. It’s just… needlessly arcane.
Git has the same problem but at least git is excellent enough software in other ways that it makes up for the horrible UX. Systemd is mediocre.
'I am altering the deal, pray I do not alter it further.' --Darth Lennart
I actually appreciate, now, what systemd is trying to do, but I am not certain why it has to take over home directories and everything else to do it. And I am really uncertain why it has to have such bad taste. I can get why it uses C, even though IMHO that is a mistake. But .ini files? D-Bus? XML‽‽‽
I am old enough to remember that problems with systemd were less technical and more political.
People didn't like the way systemd developers pushed the community to adopt systemd, specially when they asked for 3rd party developers to make systemd a hard dependency.
Unfortunately,people don't remember this today, and think users resisted to systemd adoption only because they didn't like systemd.
Network configuration (on say, Debian) is still a pain and has a lot of different ways of accomplishing similar tasks. Anything beyond the plain vanilla singular network device that is either dhcp/static is often painful. Systemd has some sauce, there is legacy /etc/network* stuff, there are daemons which can be configured graphically… it’s still a mess. In this arena, I think systemd has failed for the last decade. It introduced yet another way of doing things without considering the whole scope of what it needed to replace.
I’m old enough to remember the culture wars and even older to remember a well thought out approach to overall configuration in Debian given the available tools of the time.
Yes, this. The problem with systemd isn't systemd. It's that it never really replaced the other "legacy" parts effectively. Networking is a fantastic example. I'd be OK if there was one-definitive-place to configure networking, but when I read an article or follow a stack-overflow post, I'm not sure if what I'm changing is the most correct change or if its deprecated in favor of something newer. Systemd, I thought, would help settle this problem, but instead has only contributed to it.
Isn’t the number one problem here that you’re following a Stack Overflow post? Your system should have good documentation on how to configure networking, or some obvious sanctioned way to do it, such as a GUI widget. If your system has neither, that’s the problem. If it has one or the other and you’re using Stack Overflow anyway, that’s the problem.
Note I said “good” documentation - many Linux systems have lousy documentation and they lack an obvious, sanctioned way to do it. Again, that’s the problem.
Networking on linux is wildly powerful. Most desktop/SOHO installations would just be Wifi + DHCP. Maybe some VPN in the mix. But on the other side of the scale, there are routers with complex netfiltering, vlans, crazy GRE Tunnels, Tun/Tap, Wireguard, macvlans split between namespaces, bridges, etc, you name it.
Defining a straightforward configuration structure for that complexity is going to be insane. NetworkManager + Debian's /etc/network/interfaces (IMHO) seems to be a good fit: auto config w/ a GUI in NM in the simple case, and if the interface is listed in /etc/network/interfaces, you can do it anyway you want with Debian's system, with pre-up, up, post-up, post-down, etc scripts to your heart's content to setup whatever complexity you need.
My main gripe hasn't been that I want something done an exact way. It's that I want to achieve some semblance of what I'm trying to do in a reasonable amount of time.
NetworkManager is god awful in this regard. Even for simple desktop uses, if I want something more advanced than simply connecting to the Wi-Fi (even this requires finagling), then I'm fucked.
For example, I've been trying to find what it is that keeps overwriting /etc/resolv.conf.
After reading through various man pages and online forums, I still don't know. Instead, I just stick to one access point, and manually write to /etc/resolv.conf everytime I boot.
Compare this to OpenBSD: nothing fucking touches my /etc/resolv.conf. That file is mine. I write to it and other daemons read from it.
> Network configuration (on say, Debian) is still a pain
What specifically is a pain? Have you used it recently?
Maybe my end-user desktop network requirements aren't complex enough but it's been pretty damn smooth in my experience for the last few years. I used to avoid network-manager with wicd, but now it seems pretty stable. I use the network-manager and network-manager-gnome packages to get the toolbar thingy in i3wm, it's very simple to use. I also combine it with wiregaurd using the wg commands.
Are you doing something more sophisticated like routing traffic? or something else in a server context? I literally can't recall having a single networking issue that was Debian's fault over the last few years, throughout using lots of different APs.
I recently installed two Linux VMs on my laptop. One, running the latest Ubuntu, got the network interfaces (yes, I have two defined) working as I wanted. The other one, running CentOS 7.9 (because that's what we're stuck with at $JOB) doesn't, with the same network setup. Both are using systemd (something I've tried to avoid as I'm very old school when it comes to Unix administration [1]) and I've yet to figure out what is going on. The set up is easy, first interface, non-NAT, gets address via DHCP. Second interface, NATted through the laptop. CentOS interfaces are completely borked upon boot up, and it takes about 10 minutes of constant enabling/disabling the interfaces to get them "up". Pre-systemd, I might have had a chance to just hard code the network settings (which would work for my setup). These days? I'm feeling more and more computer illiterate as time goes on.
[1] I'm a developer who can admin a Unix system if forced to, but how to administrate Unix has changed at least three times since I first learned in the mid 90s, and I'm tired of the constant changes (shakes my old-man fists at the sky).
For me the best battle test was to set up a re-establishing vpn-pptp (or was it lptp?). The issue was that the daemon, name long forgotten, used a default route to establish a VPN connection, then rewrote it with an in-channel ip received at the handshake. But when it crashed or stalled, the default route didn't exist in the segment anymore and you had to re-dhcp it, somehow automagically detecting the situation.
Part of it was political, but it was also just practical. Poettering's designs are some of the worst for usability I've ever seen. They technically work, but you need a million StackOverflow Q&As to figure out how to get anything done or solve the many common problems.
It's like somebody decided that the problem with Linux was it wasn't complicated enough. And then everbody just went with it.
I keep coming back to the https://news.ycombinator.com/item?id=19023885 issue. There’s a default setting somewhere that makes systemd knowingly kill normal user processes it did not start. Why does that exist? It was never init’s job, and init is not something end users should have to be aware of to run stuff with POSIX nohup.
SystemD is kind of like MacOS, if you have a typical use case then it makes everything very easy. If you are doing something weird it can make your life a living hell.
> If you are doing something weird it can make your life a living hell.
I find it much easier to reliably accomplish non-standard things with systemd unit definitions, than it used to be with the ill-defined complex set of shell scripts we had before.
This issue is a lot bigger outside of services... I think the systemd service architecture is a clear improvement over SysV. The bigger issues I run into with systemd flexibility are related to the many other aspects of it. systemd-resolved, for example, has a very "opinionated" (to be polite) set of expectations about the environment and the systemd project tends to view any complaints about it not working outside of that environment as being the fault of the complainant. This leads to, well, DNSSEC is basically just broken and split horizon DNS is extremely unreliable. The former is sad but mostly doesn't matter, the latter leads to a lot of corporate networks having to disable resolved (I think configuring it to remove the semi-hardcoded "backup" resolvers reliably fixes this problem but honestly I find it completely ridiculous that resolved has a hardcoded list of DNS servers it just uses instead sometimes. It's unclear to me whether or not that's a bug at this point and I got tired of trying to follow the issues and mailing list threads where the developers were, uh, not amazingly helpful).
systemd management of mounts can be similarly narrow about the types of configurations it supports, and systemd-firewalld just sort of openly only claims it can support simple use-cases.
systemd-journald is also a net loss of flexibility compared to rsyslog but, on the other hand, rsyslog could quickly turn into an inscrutable mess if you used any of the advanced features (rainerscript...), so this may not be an entirely bad thing.
And in general systemd is, well, opinionated. I hate to bang on resolved too much but I just happen to have spent quite a few hours last week figuring out resolved problems. Resolved does not handle "dotless" domains correctly in a lot of existing environments (it's very particular about exactly how the search domain is set up). Poettering has basically responded that it's because dotless domains are stupid and no one should use them, so it won't be changed. I don't necessarily disagree that dotless domains are not a good idea today but it does mean that resolved breaks a lot of older corporate and institutional environments that have been using them successfully for decades. This manifests as "I updated my distro and the intranet stopped working." That kind of breaking change is not very common with core Linux services and isn't going to make many friends in the IT crowd.
I think systemd is OK for a lot of things, I don't even hate journald. The API pretty good.
But, I am 100% agreement on the whole resolvd thing. It is a complete fiasco for anything but someone's idea of a standard network.
Even the most basic things, like waiting for DNS to be up before mounting a network filesystem has to be done by writing your own units if you don't want to just try and mount and hope for the best.
Yeah, sorry, I'm blaming more on Poettering than is really fair. But firewalld is closely coupled with the overall systemd architecture and the projects are interconnected.
On the contrary - everything in systemd is much easier than before.
Consider a situation, where your sh-backed SysV init was missing some function you required. Something between start-stop-daemon() or ifup(). You could throw a bunch of shell commands (note: most shell "programmers" are not capable of doing it properly; they do some '==' comparisons in /bin/sh shebang, don't handle errors, ignore fact that variables might contain "funny" characters etc. - but in general most of the scripts I've seen during my 24 years in Linux was simply badly written, meaning at least using bashizms). And then ...you watch out for every single package update that would overwrite your changes. Or you could polish them and try to push upstream.
Now the situation seems to be worse - you can't just edit some random file to add missing function, you need to write it in source C files and recompile. Not a way for quick and dirty solution.
But instead, you can simply write the same (s)hell commands in a script and call it from appropriate unit. No messing with thousand-lines scripts, no risk of overwrites, no need for upstreaming.
Not the person you’re responding to, but try getting it to mount home directories over NFS.
I’m sure there are many other examples.
Honestly, even after 10 years, I still have’t had a single positive interaction with it vs. traditional init. From what I’ve seen, old-school init handled (and continues to handle) the problems systemd “solved” more robustly and elegantly. I don’t get it.
I remember spending days reading the arguments on the Debian mailing list. There was no push from systemd devs. (Endless threads about upstart, how tweaking upstart would solve everything and it's already in Ubuntu, and a lots of questions about guarantees. Are openrc/upstart/systemd maintainers willing to guarantee this or that? And as far as I remember, there were no guarantees, obviously.)
Before that when the hwdb and udev merged into systemd, people got the pitchforks out, it was the decision by Kay, but obviously "systemd devs" were somehow behind it.
When Greg KH supported kdbus, somehow it was again those meddling systemd devs.
Somehow OSS veterans just lost their agency overnight and became puppets? Nah.
My problem is that things start to depend on it, like snapd... I was looking into running anbox on wsl today, but I realized I need to compile stuff, because they distribute anbox in snap form only, so I kind of gave up...
> the way systemd developers pushed the community to adopt systemd, specially when they asked for 3rd party developers to make systemd a hard dependency.
Do you have any links for these statements? My memory is not great but I think I would remember anything similar to this.
For what it's worth, the OP argues the opposite - that systemd is insufficiently generic. Much of its complexity is a matter of conflating mechanisms and policy.
This is par for the course for what's essentially prototype-quality software that's trying to address a number of non-trivial issues - but let's not pretend that real improvement is not possible here, or that systemd-as-it-currently-is does not stray from what's often called 'the UNIX philosophy' or 'the art of UNIX programming', of which separating mechanism from policy is a key component.
> separating mechanism from policy as a key component of unix philosophy
Can you please elaborate on this? It sounds like a very desirable approach, but I don't really see it "in unix". (I can think of a few program suites that use processes and piping between them as the "protocol", eg. cups, the executable files in a directory that are then launched in a loop pattern is more common, but that seems to lack the policy part. Whereas OpenSSH for example lacks this.)
I use archlinux as a daily driver, but it breaks far more randomly than gentoo ever did for me. Archlinux's bugzilla is full of random breakages that happen at random system updates. Here's a random python bug that been happening, cost me a couple of hours to debug and for now I don't update python until they get around fixing it[1]. The fix? Download an older python version(not too old since that would break everything else) from the arch package archive.
I don't want to hear anything about stable or production systems or debugging from arch maintainers. Every so often I forget the journalctl and coredumpctl command line flags and have to google how to do basic tasks.
> Download an older python version(not too old since that would break everything else) from the arch package archive.
Unless you explicitly disabled it, you should have old packages archived in `/var/cache/pacman/pkg`. So you can simply downgrade via `pacman -U /var/cache/pacman/pkg/python-x.y.z`; no need to go to an (possibly untrusted) archive.
The official Arch stance seems to be "You should have read the blog before updating", and I enforce that by updating maybe 3 times a year at most.
I think I'm just used to the installer. I like doing everything by CLI. Ubuntu is always trying to push something odd like Snap, and I think Xubuntu is defunct. The Ubuntu installers are very opinionated on what kind of desktop they want you to have.
One of these days I'll just buy all the ThinkPenguin stuff and go to FOSS-only Debian with xfce. Debian has a great installer, too. Then I'll turn my desktops into cattle-style systems where I blow up the rootfs once a year for full system updates.
Arch user here since 2013 on my mobile systems. I do -Suy nearly every day, which causes me nearly null troubles. My experiences with any supported enterprise distro are much worse.
Did you check the linked bug and its suggested workaround? It's nothing an update note would address. It's just random breakage. There are a lot of these issues scattered across their bugzilla.
I used systemd for a time, and after looking into all the documentation I found it hard to understand and it's scope overreaching. It doesn't stand by the unix philosophy of mechanism not policy and is prescriptive in it's inplementation. As another comment stated, the politics behind how it was pushed never sat right with me and I believe it is leading the Linux community to vendor lock in.
I have been using s6 on Artix and find it a great init system and service manager with a clear cut design goal of knowing where to stop. It does what it is supposed to do and no more - by design. It doesn't have the user friendly interface of systemctl but that is being currently worked on. If one is looking for an alternative to systemd, I think it is an excellent option to consider.
Another happy s6 user here, and pretty much for the same reasons. I like the clear supervision process tree it creates. I'm not sold on the execline syntax, but it's relatively clean and easy to work with and since I already use a configuration management system, generating custom startup files (from a single template) isn't that much of a challenge.
I'm currently exploring how to generate minimal service environments from Guix package manifests. Being able to spawn svscan as a child under the root supervision tree means I can create each environment with its own service directory, and never worry about cross-contamination from whatever else is on the host.
I'm glad to hear it. I find s6 extremely stable. I've learned to come around on execline and have been converting bash scripts to it when I have the thought. The idea of it makes sense (to not load a heavy shell when kicking off a service and chain loading programs for small footprint).
I _think_ I did something similar to what you mention, though I'm not familiar with guix. Analogous to systems, what I _did_ like about systemd was the user level service management. So I essentially kick off a user privileged supervision tree for my session with s6 with its own scandir and service db when launching X. It is not under the root supervision tree but lives side by side at this moment since 1) I wasn't aware of s6-usertree-maker at the time and 2) i wanted any user to be able to kick off user privileged services without root intervention (e.g. setting up usertree-maker)
I'm not sure of any distros that have default s6, but Artix[1] allows you to pick your init/service manager. It is what I used and I am very happy with it.
I miss Systemd inside Docker containers so much. IMO, for an app, "one-process per container" is a huge pain and very messy. I have a massively popular software to back up that claim: https://github.com/discourse/discourse_docker
I use phusion/baseimage-docker[1] right now, love every bit of it. Having the database, web server(nginx) and the java backend all in one place works beautifully.
The only pain point is having to write `runit` scripts for startup and shutdown for each service, hence the wish for Systemd inside containers.
You can use systemd inside Docker containers. There are images (like RedHat's ubi8-init) which start systemd by default, and there are others which can be easily adapted to start systemd (for instance, the following 3 lines for a Debian image which starts systemd: "FROM debian:10", "RUN apt -y update && apt -y install systemd-sysv", "CMD ["/sbin/init"]").
Have you tried nspawn? Its part of systemd and integrates beautifully with the rest of systemd (for things like journald, networking, etc.) and is basically meant for what you requested.
It's not as bad as at first feared. Views here are along the usual lines; for many when there's an issue either it's something that's fixable inhouse (init scripts) or needs a support contract.
Having said that, personally (old school) I lost 2 or 3 hours yesterday due to not realising systemd was changing things I changed for a reason.
>I lost 2 or 3 hours yesterday due to not realising systemd was changing things I changed for a reason.
This is one of my go-to reasons when people ask me why systemd is bad. Once it overrode power settings I had configured in xfce-power-manager. Mainly I wanted the lid close to do nothing on AC power and to sleep on battery. It started sleeping for everything after an update once. I think in the end I couldn't figure out how to get the exact desired behavior with systemd and I disabled sleep entirely.
The other reason I often mention is the ridiculous shutdown timer thing that makes you wait for minutes sometimes for some service to end.
The main advantage to systemd is that a lot of unit files were already written and packaged with stuff, including --user variants, and also that it has a user mode at all (OpenRC doesn't yet). Fairly minor advantage, imo. In general I prefer shepherd, runit, or openrc. My needs are pretty simple, though. Most of the time I just need to start/stop/restart a service, which is pretty simple in every init system I've used.
When waiting for units to shut down, have you tried pressing Ctrl+Alt+Delete 7 times? It forces a reboot, and you can turn the machine off before linux has taken over again.
Pressing Ctrl+Alt+Delete 8 times (I think, 7 times to increment the counter, then 1 more time to trigger a force reboot) sends SIGTERM or something to all processes. But many wedged processes (like kwin calling a Nvidia libglx function trying and failing to acquire an spinlock) ignore the signal and can only be killed with SIGKILL, which systemd sends a few minutes after you press Ctrl+Alt+Delete 8 times. I work around this by enabling Alt+SysRq and pressing Alt+PrintScr+R (raw input), E (SIGTERM), I (SIGKILL) whenever systemd gets wedged.
You might be forgetting how bad it was at first. In fact I'd argue it was worse than initially feared, and the developer's supreme arrogance did little to help.
This was an amazing review, and love systemd or hate it, this article gives a fantastic history and exposition of what systemd actually is.
My favorite quote (and when taken in context is much less tongue-in-cheek thank you might think):
systemd is an event-driven object manager with dependency-like side effects which ‘boxes’ primitive kernel resources and userspace subsystems into a generic object type called Unit. These Unit objects are scheduled through the state propagation mechanism of ‘jobs’ and dynamically dispatched via a singleton object called Manager, responsible for launching jobs in ‘transactions’ which do merging, cyclic ordering and consistency checks and serve as the main point at which unit dependencies are pulled in. Unit startup is executed as a non-indempotent parallel dataflow with weak ordering guarantees on the job level, mostly independent of the active state of dependent units.
Any night you have insomnia, just read that a few times and you'll be out like a light! :-)
I’m a fan of this description. It might not be the easiest to understand or the first one you should read, but I wish more things had this kind of description somewhere (but not necessarily on the introductory materials or front page).
It’s buzzword bingo with the aim to inflate the writer’s ego and intimidate the reader. This stuff happens all the time when insecure people get to write things.
Which words do you think are buzzwords in the description? What I think of by buzzwords are things like “Blockchain”, “Machine Learning” etc. which are generic topics used to build hype whilst not really explaining anything.
In this case however, the words are all fairly precise and accurate (at least from my basic understanding of systemd). They might not be words used in everyday conversation, but they don’t need to be unless the document is targeted as introductory material (which I don’t think is the case).
That’s not to say this is the only way to explain it, but it is a very concise way, which I at appreciate.
I would say most of it is unnecessary buzzwords that don't help explain it. The whole thing can be simplified to something like "systemd is a task runner that has a dependency graph and transactions". That should also be fairly straightforward to you if you've used it a few times.
I'd say the other mistake the article commits is by trying to paint this as being a very complex design for the problem space. It's not when you consider the design constraints. You want services to have dependencies, and you want the updates to the unit files to be committed atomically to avoid stale data. Hence, the dependency graph and the transactions.
I highly disagree. Sure, it contains a lot of buzzwords, but it clearly describes systemd and its philosophy in a single paragraph. It's not the first thing you should throw at someone unprepared (because then it would really be intimidation), but it prevents you from reading 20 pages of documentation and not getting the gist.
Sometimes, a concise, high-level overview is what you need.
Maybe because I've worked in embedded Linux for so long (including on inits of various forms), but that description clearly articulates exactly what systemd is and does and I appreciate the direct, crystal clear technical language rather than some long form essay written assuming a clueless reader with 0 technical background.
You’re pulling my leg, right? The description could be improved, for starters, by forcing the author to not use the word “object.” That overuse alone makes me wonder if the description is satire.
I expect everyone here commenting is familiar with systemd. I doubt very much that anyone who was unfamiliar with systemd and read that description would then have any idea what it does or how it works.
I mean, I'll agree with you in so far as that the wording could be improved further. But I do really, honestly think it's a good summary.
> I expect everyone here commenting is familiar with systemd. I doubt very much that anyone who was unfamiliar with systemd and read that description would then have any idea what it does or how it works.
It depends on what knowledge you expect. If someone is familiar with general init management and some programming concepts, this short summary will get them up to speed (and that's what I/we were arguing). If you grab a layperson of the street and read them this, it's obviously not going to tell them anything - but that's not the target audience.
What I value is a short summary which brings a reader, who knows the concepts and the space a software is operating in, up to speed quickly. And that's what this summary does.
And while this list is good for BS Bingo, as these terms are being used in other contexts too, the overall description of what systemd does is pretty precise for just three sentences.
“Idempotent” is one of the key words used as a cudgel in this way. It’s great for the bullshit artist, as to a layman the dictionary definition is completely inscrutable.
> as to a layman the dictionary definition is completely inscrutable.
This isn't written for laymen, who would obviously never be reading about what SystemD is. It's written for specialists who do engineering work on or with Linux, and who actually need to know the technical details of their tools.
And anyway, I suspect plenty of laymen, and highly trained professionals in other fields like law, medicine, and philosophy among others, would have no trouble understanding what "idempotent" is. It's not inscrutable at all, but rather is one of the easiest concepts in CS and software engineering to understand.
Idempotence is a desirable property of systems whereby repeating an action has no effect. Mathematically, an operator is idempotent if f(f(x)) = f(x).
Idempotence is handy because it implies that one can retry safely if one isn't sure whether something succeeded or failed. Not sure if your file was saved? Save again! If it was indeed saved, the file continues to be saved. If the file was not saved, it is now. Yay.
I would kill for such descriptions of every system I work with.
Idempotent is surely a valuable piece of software jargon. I know exactly what the user intends, and have good ideas of how hard it is and what likely trade offs they will be making behind the scenes
Actually, it is just a word to describe the property that if you do something repeatedly, it leads to the same result as doing it just once. So the latter executions don't change anything.
And because it is such a common property when talking about algorithms and APIs the word is pretty common among CS people.
One very popular example are REST APIs because some HTTP verbs are idempotent and others are not [1]. This is also the reason why your browser asks you, if you want to submit a form again when refreshing a page that has been reached by submitting a POST-form.
> “Idempotent” is one of the key words used as a cudgel in this way. It’s great for the bullshit artist, as to a layman the dictionary definition is completely inscrutable.
Are you saying that being "idempotent" is completely inscrutable? Is it not a fairly straight-forward concept?
If we're going to be pedantic, "idempotent" (or rather non-idempotent) as used in the description is rather vapid and meaningless. Idempotency is a quality that describes the result of an action; as such, only actions can be thought of as idempotent. Applying the word idempotent to any noun that does not describe an action (such as "dataflow" in this case) is meaningless.
What does it mean for a dataflow to be idempotent in the first place? What purpose does it then serve to describe a dataflow as non-idempotent? What they meant to say, I guess, is that the dataflow may contain multiple actions for the same unit, and that systemd makes no guarantees about the end state of the unit in such a case. But even then, it is the execution which should be described as non-idempotent, and not the dataflow.
That feels unnecessarily pedantic. If the sole job of entity X is to perform action Y, and it always performs this action idempotently, then why not describe X as idempotent? It's shorter, and it's clear enough. It might not be precisely accurate in the dictionary definition of the word - but natural languages also have DSLs, and those DSLs evolve to adapt to their use cases. Saying "thing that performs an idempotent action" over and over again is a waste of everybody's time.
Idempotent is a very clear and quite basic term: being able to run something multiple times with no different outcome than running it once. A very valuable property for certain systems (even for simple shell scripts).
And the description of systemd just uses bog standard terms related to what systemd does and the resources it manages.
But I get the feeling that both this and the parent comment it agrees with were more about the putdown and talking bad at someone, than about a technical or other objection.
I think the whole problem with systemd wasn't just about technology. The results were spectacular. Before systemd normal Linux boot times were well over a minute. After switching you could get below 10 seconds on the same hardware. And that is just one aspect.
I think the core of the problem was somewhere between the style of introducing changes and the way how people communicated. After all, it speaks for itself that pulseaudio and systemd shared a lot of common criticism. Both are technically sound systems today, but in a way, systemd was just the escalation of the pulseaudio debacle.
SystemV init allows for init scripts to run in parallel, but that was rarely used. Most time spent was anyhow waiting for network (e.g. the MTA trying to resolve the hostname) and hardware initialization. Bourne shell isn't particularly fast, but ten years ago hardware used for typical Linux desktop/server systems was fast enough that it hardly mattered (it's used here for sequentially starting system processes, not number crunching). Not that it matters -- how often do you boot a desktop/server in a day? Performance mattered for VMs deployed in the cloud on request, but whether a full OS is the best choice for such services is a different topic ...
...so, how does a systemd unit definition differ from a Makefile (other than using a different syntax, obviously)? What features would you need to add to UNIX make (and PID 1, for that matter) in order to be able to separate out this whole dependency management business from the other stuff PID 1 has to deal with already?
For one thing make is imperative and doesn't actually know what it's doing. Systemd is declarative. You'd need to replace all the options service files give you.
Make doesn't do dependencies between unrelated projects. A makefile can't say "I want to run if X runs, but I should run before them".
Make doesn't have supervision, you'd need some kind of declaratively configured poller.
Systemd is also parallel, there's probably some complexity there that make might or might not handle well.
I think the most important problems would only be obvious once you started using a replacement system though. Systemd almost totally replaces the need for any imperative code whatsoever for process autostart, configuration, and supervision, and that might be hard to do in a more modular setup.
You don't need extra stuff besides the service file. Everything is extremely consistent and done the same way for every service.
Everything statement that comment makes about make is incorrect, except the bit about polling.
Polling is better-addressed by having each service implement its own error handling. Restart loops are the wrong policy for most services. In the rare cases where they are actually the correct behavior, they’re trivial for the service author to implement.
Those that don’t understand make are doomed to reimplement it poorly.
How are restart loops the wrong policy? If it should be running, and it's not, how else would I fix it except by restarting it?
They're not exactly trivial. There's restart delay, number of times to restart, passing stdout through so the journal can get it, and there's failure reporting.
How is PID 1 supposed to know if my service failed when the restart daemon is still up and running, but my actual app keeps crashing? How can it tell the difference between my app and a supervisor, or a fully failed app or just one failed subprocess that my app can handle? Now you need some scheme for that.
If PID1 doesn't know what's failed, how would I get a list of all failed systems programmatically?
Systemd could provide a standard supervisor component, but then it couldn't ever do anything but restart failed apps, or you're back to people complaining about monoliths.
You'd have to have your service file explicitly call it with some command line args making your syntax uglier.
And more importantly, it might be harder for the systemd dev team because now they have a communication protocol between this restart daemon and everything else.
Or they could leave restart daemons up to the user... and everyone would probably choose 3 different ones, that might or might not keep up with compatibility. That would break the "One and only one obvious way" principle.
When you start splitting stuff up in tiny bits, you get more overhead, all the extra nice features seem to evaporate.
Dependencies in make are equivalent to systemd's Requires= and After= combined. Systemd is far more general in this regard, both in that Requires= and After= are separate concepts, but also in that there is Wants=, BindsTo=, etc. and their inverses (RequiredBy=, Before=, etc.). Sometimes when I'm writing Makefiles I wish Make were capable of expressing some of these concepts.
It's not even the immaturity of the joke that bothered me back then, but that the response to the criticism was to imply that it is "corporate bullshit" and/or "giving up on having fun".
The systemd "culture war" is just a side-effect of the hegemony server Linux enjoys over desktop Linux.
A very small percentage of sysadmins really care about what init system is running on their servers. If you look at who's forking distros and making systemd-free variants, and look at who the vocal anti-systemd users are, they're overwhelmingly desktop-focused users. But this is strange because desktop Linux barely needs an init system at all. The long-running daemons on a desktop Linux system only need be started on startup, if they crash, restarting them is unlikely to leave the system in a workable state anyway.
However, because server Linux dominates the Linux ecosystem discourse and distro news, people have an inflated view of how important these components are.
So my take-away has been: it's safe to ignore systemd criticism, since it's overwhelmingly coming from hobbyists who have no real use for an init system anyway.
Vehement systemd defence also overwhelmingly comes from hobbyists who have no real use for an init system anyway. In fact, this whole argument is crap, most hatred of sysvinit comes from people who have never used it, is that a reason to discard all criticism of it? Of course not.
I have had enormous issues with systemd, not when using it for desktop purposes, but precisely when using it for server applications.
As an example, systemd can't handle a service which depends on a specific network interface (or specific bind address) before being brought up and the systemd developers blame the software for not listening to rtnetlink events or not relying on linux specific socket options[0]. This is unacceptable, it makes it incredibly difficult to run systemd on hosts connected to multiple segregated networks without lots of dirty tricks involving firewalls.
I never became comfortable with neither the scope of systemd as a project, nor the ever expanding set of skills required to effectively use it. I was tempted to use the word mastering, but that would make the goal seem even more elusive, especially since with every new version one's knowledge would have to expand to cover its new features. 10 years later, it still looks to me like the project with a solution for everything in the IT industry, following trends, trying to cover all the bases. Always new things to learn, always the goalpost moving further and further.
You don’t have to “master” it, but just understanding the basics of what is a service file will make your everyday life much better if you are working with systemd-managed systemd.
Also, don’t get bogged down by systemd-* services, they are related but are usually not part of systemd at all.
I don't actively hate systemd, but the distros I use these days (Void and Alpine) don't use it, so neither do.
I'm generally a fan of simple systems that don't do much controlled by simple text files. Maybe systemd is way better at runit for more complex workloads than a desktop or a simple server. But maybe it isn't. The intersection of people who like systemd and have tried runit is probably quite a small one!
On HN, reposts only count as dupes if the story has had significant attention in the last year or so. This is in the FAQ: https://news.ycombinator.com/newsfaq.html.
(but also, it's true, people tend not to use HN search before posting)
Trying to post a dupe often results in nothing being posted and the original submission being opened instead. I have no idea how accurate the thing that checks for dupes is, though, or on what criteria its decision is based.
The hatred for Systemd seems to come from those who see the unix philosophy of small, single use programs being thrown away in favour of a monolith.
Back when I had linux as a desktop (around the day of slackware 10) there were plenty of problems with simple stuff like mounting usbs, connecting to wifi etc--possible but a pain. I've hardly looked at systemd or linux in such a way for a long time, however.
Linux users tend to come up with whatever "philosophy" suits their agenda of complaining about all change. The unix philosophy gets thrown out the window as soon as it comes to Wayland which took a huge, bloated, and buggy Xorg and broke it down to something more sane and manageable.
Plenty of us who appreciate Linux and Unix are very, very worried about Wayland, which breaks a lot of things we have relied on and used for literal decades, and also seems to have some dubious lacunæ and misfeatures.
I am actually optimistic that in another dozen years or so Wayland might be pretty nice, but I certainly wouldn't consider it an examplar today.
Doesn’t wayland force window managers to be integrated into some behemoth bloated monolith?
My preferred WM is evilwm. I read the entire source to it in a few hours, and occasionally edit the source when I want to customize it for my odd workflows. I don’t think that’s possible with wayland.
Also, wayland forced a bunch of questionable security policies around screen capture and keyboard handling, and didn’t provide escape hatches required for basic functionality. Is that fixed yet?
I stopped paying attention years ago, to be honest.
What Wayland changed is that Wayland is not a piece of software like Xorg was. It is simply a specification. Which means you don't have potentially have a 100 line C program as your wm which just relies on xorg to do the heavy lifting.
KDE and Gnome have their own wayland implementation but this isn't really a problem. You can still write you minimal WM because their is a project called WLRoots which does the same job as the old x11 package. It lets you trivially interact and be compatible with the wayland spec without having to implement it yourself.
Screen capture and keyboard privacy is essential for the future of linux IMO. I don't know the exact state of it but I did not have any issues screen sharing with wayland compatible apps. There were some issues with x11 only apps but I think PipeWire solved that.
Wayland is a protocol only. A wayland implementation may choose to incorporate the functionality of a window manager, or can create some way to outsource it to a separate executable.
The security policies are not questionable. XServer is beyond fixable. A single rogue extension can screen share everything you do to a remote server, you would not even recognize that.
> Doesn’t wayland force window managers to be integrated into some behemoth bloated monolith?
Well, the wayland protocols are very poorly documented compared to X. But most WM's have to support a bunch of obscure X extensions to get a reasonable featureset, that bring in much the same complexity as Wayland.
I don’t care about philosophy; I don’t like being forced and I don’t like opaque systems where they aren’t required.
SystemD is essentially forced (for better or worse, not gonna argue) because it is a hard dependency for certain software and tightly integrated with other tools in the systemd ecosystem
It is opaque because I cannot read, modify or understand what it’s doing without breaking out gdb, heaven forbid I want to alter its behavior in a less than approved way.
This leads to weird situations like the MySQL unit executing a bash script to run: because systemd has limited preflight checking compared to alternatives.
Before someone claims I want sysvinit back: No, I don’t want that. I would simply prefer systemd to be smaller so that we can replace it with something better or more tailored should the need arise.
We’re locking ourselves in to prototype level software (which we best into working semi decently) with obstinate leadership team who deny any faults.
> It is opaque because I cannot read, modify or understand what it’s doing without breaking out gdb, heaven forbid I want to alter its behavior in a less than approved way.
In this respect I think that systemd is really a victim of the Unix kernel/shell style, where 'kernel' is read expansively to mean 'everything in C' and 'shell' is read to mean '/bin/sh'. The problem is that where shell is easily examinable and understandable for most technical folks, compiled C simply isn't (yeah, yeah, we all know someone who can debug in assembler; that guy ain't me).
It would have been nice to have had one system, with a typesafe compiled language. Scrutable, fast and safe. We have an example with emacs: one can always just M-. to a function's definition (even in the C core …).
It's a real pity that systemd hadn't been written in Common Lisp or some similar language.
I'm not sure, it seems to me that the hatred for systemd seems to come from those that learned something else and don't want to learn systemd. My analysis would be that the "unix philosophy" is just a convenient argument against it.
The only bit about SystemD I'm not fully sold on is D-Bus. I love that everything is exposed via D-Bus, but the C API is incredibly difficult to reason about. I've tried writing some higher-level bindings for it and it's still pretty difficult to get right.
Perhaps at some point they'll switch to something that's established instead of maintaining their own IPC protocol.
D-Bus is the established IPC on desktop Linux. The libdbus library is pretty old and not great, you'll have better luck using one of the other implementations. I can recommend a few:
- GDBus, if you're developing for GNOME
- QtDBus, if you're developing for KDE
- sd_bus, if you're writing low-level C
- zbus, if you're using Rust
For other languages, you'll want to look for bindings to one of those libraries, instead of bindings to libdbus.
I've tried writing higher level bindings on top of libdbus. Once I realised that libdbus was not designed for writing reliable or sane software (there's an error? there's an error! what is it? who knows) I decided to try writing my own implementation of the dbus protocol (the intention was to write C code generators for automating away writing a lot of the boilerplate). Once I had read the specification once I gave up. dbus is the most hideous protocol and most hideous library, its ONLY advantage is its ubiquity.
I disagree with this, in my experience D-Bus is very similar to other IDLs like DCOM, protobuf or ASN.1. If you're familiar with those, D-Bus should actually be easy in comparison.
It’s tolerated because it was forced upon people by distros. I don’t know anybody who likes it, and I would happily rip it out if it wasn’t so much work to do so.
I like it. It's consistent, well documented, and has matured to the point where the last time I remember it getting in my way was probably 8+ years ago.
If something were to be proposed as a systemd replacement today, I'm pretty sure I'd treat it with a lot of skepticism. I likely would use arguments the sysv folks made like "why fix what isn't broken" etc.
Excluding the kooks angered enough to send hate mail to the systemd developers, I can understand the old criticisms and aggravation from sysv folks. I was front and center as a relatively new user of Fedora at the time systemd was adopted. While I was familiar with sysv, I wasn't attached to it. I also found sysv a lot more confounding and obscure than systemd in the early days, even though the transition did have rough spots. But I'm definitely attached to systemd now.
It would be interesting to see Systemd compared to other modern init systems, like runit and the FreeBSD init. Seems like 2021 Systemd is being mostly compared to 2000's Upstart and Sysvinit.
".... and found that someone was about to release TCP/IP for Linux" (Fred van Kempen, linked interview 1992)
Wow. I had forgotten how ropey early Linux was. I mean, I rip on Windows folk about early versions of Windows prior to 98SE or NT4 (and can recall having to install Winsock on a home Win95 install and thinking why the hell????) but finding out 30 years later that my early struggles with Linux were not entirely down to my own incompetence is ... refreshing. I mean how do you Google (ok in those days Altavista!) a "how to" if the network stack is borked? How do you even download the drivers or updates if the interface fails to come up?
It sounds silly but I'm so glad to know it wasn't something I was doing wrong on my end. 30 years later and dayamn!
Have a spare generic PC or make sure you didn't mess double boot. Heck, now I don't understand either how we fixed network issues when phones could only play midi or call a friend for support. Good handbooks were a rescue.
How does it work in a Kubernetes world? I found today that "systemctl stop k3s" actually did not stop all k3s dependent processes. A separate script was supplied to do that. So on a Kubernetes box there are processes running in namespaces that systemd does not manage?
Not sure what you’re asking about. If Systemd didn’t start it, Systemd doesn’t manage it. Nothing strange about that…? Systemd does not start all the Kubernetes stuff.
k3s could stop everything, but it doesn’t. It’s a design decision.
It did start it. For example the k3s process did start a container registry, but it did not stop it. I thought one of the points of systemd was that it would reap parentless processes
Depending on the container runtime in use (containerd, crio), a separate process is managing and running those container processes (so k3s is telling that process what to do, but doesn’t “own” the containers). Also, modern container runtimes interface directly with systemd to launch transient units (think systemd services but without on disk config), so restarting a cri-o for example won’t restart those processes. That allows systemd to manage the process and for k3s/kubelet/cri-o to be upgraded / crash and recover.
Under crio to stop all containers you can (well, should but YMMV depending on any changes we’ve made recently)
systemctl stop kubepods.slice
Note that doing that a subsequent start shouldn’t bring the containers back unless k3s and crio were both running (the transient units don’t survive restart).
Note: there’s tons of complexity here that has changed over the last five or so years as container runtimes evolve, so if anything I said is wrong now it was right at one point and might actually be a bug.
Kube for instance didn’t use to gracefully shut down containers on node reboot, and over last few years we’ve gradually fixed all of the complexities so that you do get the correct shutdown behavior between systemd (which takes the “power down” input) to the unit (which encodes how long graceful shutdown should take based on the pod termination grace period seconds) to cri-o (which has to set the correct behavior in systemd) to kubelet (which has to stay running to keep updating the API) to the other components on the node (like container networking, which needs to stay up until all pod network pods are done).
systemd makes a lot of that a lot less painful than it would without.
It does exactly what the configuration tells it to do.
At one point some distributions accidentally configured the sshd or cron unit to do that. People were a bit surprised when restarting sshd or cron killed every single background child process.
My point is, this feels like an area where there is more work to do. Not surprising, the container landscape is a relatively new and rapidly evolving field that hardly existed when systemd was invented. But the fact that a piece of software supplies a killall.sh script reminds me of the old situation before systemd, when every software provided their own lifecycle scripts.
Which processes systemd allows to live on after stopping an associated service depends on the kind of KillMode= (see systemd.kill(5)) that was specified for the service unit. The default ('control-group') is usually what you want, esp. if you are looking to not recreate the kind of chaos improper process management SysV-style init scripts frequently caused.
Chances are the k3s service unit specifies something else (like 'none') - whether or not that is actually wise would probably require some research.
Thanks for explaining that. My comment was not a criticism against systemd, more a surprise that k3s did something that was not expected by me when I interacted with it through systemd. For example, httpd does not supply a kill-all-httpd.sh script as far as I know. Perhaps there are systemctl verbs that can be added to support what k3s devs apparently thought neccessary to implement in a separate script.
Well, yes and no. Systemd starts the k3s master process (or whatever it is called). It also stops this process. At this point, Systemd’s work is done and it is working properly.
Now k3s of course starts tons of stuff, mostly in containers. Therein lies the problem: By default, Systemd indeed tries to get rid of the entire process tree. It does this using control groups (cgroups). Unfortunately, it is easy for a process to “escape” its control group, at least for root.
Stopping the master process could absolutely be made to also stop everything.
It would be great if k3s did properly integrate with Systemd. It doesn’t, but Systemd is not to blame for that. That’s my point.
If systemd is your init, then it manages everything. It may not manage them at a service level ala systemctl stop, but that's true of pretty much everything. Whether or not k3s brings down everything with it is on k3s. I certainly see why stopping k3s might not stop all dependent processes: it's useful for live upgrades.
Yes the live upgrade argument was mentioned when I searched for it. So it makes sense in some ways but it surprised me, and I think probably this is an area that can be improved.
Systemd is like Windows. A monolithic one size fit all big shit. Most people will tell you that they are perfectly fine and the system is great. But they are just used to all the issues they are encountering or don't realize that they are caused by that.
Lots of people are not surprised when you tell them that pulseaudio was created by the same author. Because it is kind of working in most cases. But there are always, aside, a lot of annoyances or things that work badly without a real justification: hard to fix and understand what is going on when it doesn't work; sometimes it decides alone to not use or not show the output/input that you want and you have no possibly to decide; sometimes it is using 10% of your CPU despite the fact that you are not playing a single sound at the moment,...
10 years and I still can't figure out how to view custom service logs, darn, so I redirect them into a logfile.txt in my .service files. Well, okay.
I don't care how it works and didn't take part in condemning it back then either, and I think that probably it made something better (? not that I felt it), but from a pure sudo-user perspective it is a strange ini-based tech-ridden(?) launcher to me. I've managed small networks before it and haven't really experienced any trouble that it supposed to remove(?).
Tldr: I still don't know what it is and what is it for. Servers worked before and after it, maybe that is already a good experience.
