Since systemd became mainstream? If only you've seen the time, money and sanity I saved when I switched to systemd instead of suffering with prehistoric initscripts.
The bugs generate noise and make the news, the successes rarely do.
(And what's wrong with time sync, exactly? It seems to work perfectly with zero configuration required, for a large number of people, myself included.)
The question above was, "when has Systemd done it right?", followed by an assertion that they've been doing it right "since systemd became mainstream." I offered several examples, such as the '0day' username thing that was not only an example of a bug, but an example of very clearly Doing It Wrong on a design level.
I'd suggest you search for timesyncd issues on Google, but you preemptively announced bugs don't matter to you and declared "works for me!", so I don't know why you'd ask. So perhaps just stop and consider for a moment why, exactly, it is that your init system is expanding to replicate existing, functional, standards-compliant userspace daemons with limited, buggy, noncompliant "replacements".
> The question above was, "when has Systemd done it right?", followed by an assertion that they've been doing it right "since systemd became mainstream."
Yes, they've been doing a great many things right. That doesn't make them bug-free by any means, nor does it mean that every single thing they've done is right; it does mean they built something incredibly useful and working for a large number of people. People don't seem to talk about those as often; outrage carries so much louder.
> I offered several examples, such as the '0day' username thing that was not only an example of a bug, but an example of very clearly Doing It Wrong on a design level.
Yes, and that was broken. And it has since been fixed, but that didn't get nearly as widely reported. systemd now checks for that issue and reports it rather than running the unit in question.
(You could argue about its parsing of such fields, and that discussion is ongoing, but that's separate from the issue of running the unit as root.)
> I'd suggest you search for timesyncd issues on Google, but you preemptively announced bugs don't matter to you and declared "works for me!"
No, I asked the question of what you considered problematic about timesyncd, especially since you seemed to be talking about what you considered fundamental design issues. I keep a close eye on the large community of Debian folks running systemd, and read the bugs reported, and I had not seen anything notable related to timesyncd, especially not anything that would suggest a design issue.
I never said bugs don't matter to me, nor was I attempting to generalize my own experiences to suggest that it must necessarily work for everyone. You seem to be actively seeking out and assuming hostility where none exists; I would not be surprised if you find it, but I'm not looking to supply any.
You appear to want to have a particular discussion that I'm only marginally interested in, and I don't really have much time today, so I'll have to curtail this. I apologize for that.
Sounds like we're agreeing, sort of. A nasty bug was caused by an inexplicably weird design decision, demonstrating that they haven't been "doing it right" since it became mainstream. That is what I was responding to.
Backing into my interest in the discussion: it is that inexplicably weird design decision, made far worse by the authors' repeated habit of reflexively trying to make their problem someone else's, that explains why I simply don't trust them or their code. This is a pattern that has been repeated over several several iterations across years - the first one I was aware of was when they were crashing the kernel in debug mode, leading to the famous "fuck systemd" patch-showdown. The repeated replay of that pattern shows they haven't learned anything. That combination of arrogance and incompetence is annoying in college grad-hotshots, but they can be kept in line until they grow up; bluntly, it has no place in my systems, and makes me wonder why RHEL wants to burn trust and goodwill like this.
Moving on: briefly, timesyncd is sntp, not ntp, is client-only, doesn't track jitter, only jumps forward, and makes other mistakes. I've seen reports of the sntp implementation being wrong, or perhaps simply having interop problems, but haven't bothered to look because I don't use it. Those things probably don't matter for a gaming client or such, but it simply isn't an ntpd replacement.
And it still leaves the question of why your init system, produced by an erstwhile-enterprise vendor, is replacing unrelated daemons with reimplementations that look more like college-assignment toys than production software.
> A nasty bug was caused by an inexplicably weird design decision, demonstrating that they haven't been "doing it right" since it became mainstream.
As I understand it, two separate design decisions interacted there. One was "some fields should be ignored if not supported or if they use new syntax that isn't supported, so that a unit written for new systemd won't break on old systemd"; if they'd done that differently, it'd have generated many problems as people wrote units for the latest bleeding-edge version. The other was "parse and validate usernames and see if they look sane"; the ideal solution there would be "check if they exist and do no other validation if they do", but NSS turns out to not be viable for that in the context of an init system. Usernames should never have been an "ignore if not supported" field, but it's at least understandable how the issue could occur.
If your standard for "doing it right" is "every single thing they do is always correct", very little software will meet that standard.
> Moving on: briefly, timesyncd is sntp, not ntp, is client-only, doesn't track jitter, only jumps forward, and makes other mistakes. I've seen reports of the sntp implementation being wrong, or perhaps simply having interop problems, but haven't bothered to look because I don't use it. Those things probably don't matter for a gaming client or such, but it simply isn't an ntpd replacement.
timesyncd doesn't claim to be a replacement for all of ntpd; it claims to be a simple implementation of the common case of "I want my time to be correct". A client-only SNTP implementation is what they set out to build.
It's understandable that they made that mistake, it's not understandable that Lennart kept insisting it's not a bug until it blew up in the news. Remember that he actually got cranky that somebody assigned CVEs to the issue...
I agree with _jal that this kind of arrogant, dismissive behavior and repeat behavior doesn't instill trust and does a huge disservice to the project's reputation.
I do agree that the messaging and tone was painfully bad, yes.
I can sympathize with people who spend all day dealing with unwarranted rants and flames letting some of that leak out into their responses to everything, but yes, that should have been handled better.
That's the only thing I've seen people praise systemd for, and I happen to agree.
Nothing else systemd does do I think it does better.
If systemd had remained an init system and nothing else, it would've been a clear improvement worth breaking what it does. What systemd is today should be called the Fedora userspace suite, not a Linux init system.
… and yet, oddly, most of the distribution maintainers who reviewed them picked systemd. That doesn't mean it's perfect but it does suggest this conversation would be far more productive if you reviewed the historical discussions to see the reasoning behind those decisions.
The reasoning was hammering it down to everybody's throats by Redhat. It is easier to adapt to proposed solution (even inferior) where there are some guarantees of support and traction, than to reinvent everything and be left as sole maintainers.
It doesn't relate in any way to the quality of the solution.
You are giving far too little credit and good faith to the distribution maintainers who spent the better part of a year carefully evaluating the available options. I watched those discussions take place, and contributed to many of them. They included detailed evaluations of the service management capabilities, verbosity, transparency, debuggability, maintainability, tooling, backward compatibility, any numerous other factors.
That you do not like the outcome does not mean the process was inherently flawed or incomplete.
I certainly do not blame distribution maintainers. I blame Redhat, with their not-so-subtle hints of "we might not have any resources to support anything that comes from us for non-systemd systems". Which at some point extended to GNOME and other vital sine qua nons.
> not-so-subtle hints of "we might not have any resources to support anything that comes from us for non-systemd systems".
That honestly doesn't seem unreasonable to me. If you build a tool to make things easier to maintain, you lose much of the benefit of that if you still have to support other alternatives where you have to do everything manually. (For instance, maintaining a 100-line init script in addition to a 10-line unit file.) Asking people who care about that to do the work to maintain it seems perfectly reasonable.
> Which at some point extended to GNOME and other vital sine qua nons.
There are far more people complaining about the lack of alternatives, and far fewer people willing to actually write and maintain alternatives. It doesn't help that many of the people complaining take the attitude of "you don't really need that anyway".
AFAIR the gnome people were desperately looking for someone who'd maintain console kit and the systemd people stepped up and provided a working alternative, so gnome switched. After the switch, systemd basically was a hard dependency for every system using a recent gnome version. If the time spent complaining about systemd were spent towards supporting alternatives, things might look different today. It doesn't require coding an alternative, but organizing support, funding a patreon etc. would go a long way.
It's also worth noting that you can still, today, run systemd-logind (and several other systemd components) with the compatibility layer of systemd-shim on top of another init system. But even that doesn't have enough people willing to maintain it.
RHEL5 came with one init system. RHEL6 came with another. RHEL7 comes along with yetanother init system replacement.
Each of which has required software vendors who build software to run on those platforms to do non-trivial porting work. I know it's annoying the software vendors no end.
I've heard from so many end users that they can't upgrade to RHEL7 (or derivatives) because the software they need to support doesn't work with systemd yet. Stuff that makes them frustrated with both RedHat and the software vendor. Annoying your customers hardly seems the sanest business practices.
Luckily with Debian being on board, and thus Ubuntu, at least there's some incentive for vendors to work at it.
They should be frustrated with the vendor if an upgrade is being held up by the trivial Upstart to systemd conversion for a dozen lines of code. Supporting both init systems combined is an order of magnitude less work than SysV init, and it allows you to rip out a good bit of darmonization code which tends to have details less portable across Unix variants.
It's far more likely that the real cause is one of the many major dependency updates (6 to 7 is like a half decade jump) and the systemd mention is either an excuse or axe-grinding.
There seems to be an effort to rewrite history to "well, everyone adopted it, so it must be good, right?" That is very much not the way it happened. Go read the discussions on the Debian lists, for instance - RHEL was indeed throwing its weight around.
Add to that the games they're playing with interfaces, and at some point, it starts smelling like a miniature, farcical version of Microsoft in the 90s.
I don't even know what planet you're living on. In particular, Russ Allbery of the Debian technical committee gave some of the most detailed evaluation of systemd vs alternatives that I've seen from anyone from a user and maintainer POV. (He noticeably said, beforehand, he didn't think it would be a huge deal at first, but it turned out it was a big difference.) Many people did not like their decision, but suggesting that the Debian committee in particular didn't take the issue seriously and were strong-armed by Red Hat -- makes me think you have no idea what you're talking about, honestly.
I'd like to read this part about RHEL throwing their weight around on the Debian lists. Given your implication is that Debian was clearly "forced" to adopting it, I imagine the relevant evidence shouldn't be hard to find.
(Alternatively, you could actually ask other Debian maintainers yourself, like Josh, in this thread how it went. But you already did that and it didn't seem his narrative aligned with yours, so...)
Everyone seems very focussed upon Debian in this thread, even though the thread started out discussing all distributions.
It might be worth everyone's while remembering that Debian is not the be-all and end-all here, even though it did have a massive hoo-hah. The processes in other distributions were markedly different.
I participated in almost all of Debian discussions. I don't see how you can claim what you claim, at all. Care to explain? E.g. RHEL was throwing its weight around?
The process at Debian started years later than any other distribution. It involved various votes, at least a year of discussion, etc.
Then in 2017 someone ignores history and summarizes this into "RHEL was throwing its weight around"... ?!?
One question: did the ability (or easiness) to stay compatible with Gnome 3 influence the decision? How about the great udev scare? Also, we all remember that 4:4 vote, and all the drama later.
Your first sentence is directly contradicted by the second and certainly doesn't fit the history of the distributions. Debian, et al. have a long history of not following Red Hat when they see value – e.g. think of the patches which Debian developers maintain to make configuration easier and safer compared to Red Hat — and reason why they couldn't have done the same here other than that the people who actually do the work didn't think it was worthwhile.
It's simply disingenuous to describe Red Hat contributing a lot of engineering time for free as “hammering it down to everybody's throats”, any more than Linux was hammered down our throats over Hurd. I don't think systemd is perfect but I think anyone's standing to complain about open source software is bounded by your willingness to commit to supporting alternatives.
(And, lest you think I'm some sort of die-hard Red Hat fanboy I should note that I started using Debian in the Bo/Hamm era and have never found a compelling value to using RH)
On the contrary, I mostly use Fedora. Systemd won, it is sad, but I accepted it. However, I want to outline that the reasons for its proliferation were less than objective merit.
I'm not sure this really makes sense. It sounds like RedHat didn't hammer it down throats, but instead created a solution that had some guarantees of support and traction which distro maintainers then decided was better than the current alternatives. No one was holding a gun to distro maintainer's heads tell them that they needed to move away from the existing solutions or else, or at least I haven't seen any evidence of that.
So if they weren't forced into it and RedHat did indeed provide some guarantees, it does actually suggest what the parent says.
Even if what you say is true and systemd is the inferior system and everybody only picked it because Redhat promised to support it, that's fine with me. Support is indeed a major feature. Anything that is not supported is lacking a feature. I'll take the good-enough-but-supported version of a thing over the perfect-but-left-out-in-the-cold version any time of the day for anything that I'd put close to production.
I haven't had a lot of stability problems with systemd, but I hate it. For me the hate comes from its extremely arcane interface and configuration layout. It's as bad as git in that you have to Google to learn how to do anything at all and the way to do simple things is obtuse and non-memorable, but unlike git I don't use it constantly so I never remember. Doing anything with systemd involves searching for how to do it since the command structure, layout, etc. is bizarre and counter-intuitive.
There seem to be some Unix/Linux developers who have a mysterious affinity for obtuse arcane time-wasting cognitive-load-increasing design. It's like they see the ability to master crufty badly designed systems as a badge of honor or something, or maybe it comes from a drive for job security or consulting hours.
Example: "systemctl list-units"
What's wrong with "systemctl ls"? What would have been wrong with a shorter command that's easier to type like "sys ls"? It's a core aspect of the system so a name like "sys" would have been appropriate, easy to type, and easy to remember.
Even worse the output of list-units has overly long lines and manages to be simultaneously hard for humans to read and hard for machines to parse. It uses white space as both a delimiter and within identifiers, making shell script parsing with "cut" etc. impossible.
The entire design is like this: obtuse, verbose, clunky, hard to type, hard to remember.
I absolutely loathe this stuff. Using it inspires fantasies of causing physical pain to its designers. A little bit of thought could have resulted in a clean, sparse, intuitive, and discoverable design with memorable commands and a straightforward configuration structure.
Don't get me started on abominations like Debian packaging or Windows drivers, though those are somewhat forgivable as their ugliness can be explained by their age and the need for backward compatibility. Systemd was a green field design from the 21st century so it has no excuse.
I have to agree on systemctl's naming scheme is terrible inefficient. This is something I use all the time when I'm on Archlinux and while I've come to memorize systemctl's interface for the most part it's still very poorly thought out, favouring verbosity over simplicity at every level.
I've made enough aliases to make it useful and without having to type such a long name. It should definitely be a shorter name for such a critical piece of software.
One example is using user services vs system services and the various non-intuitive locations and name schemes of the various .service files.
I'm not a systemd fan, quite the opposite however I have often found package manager, service management clis less than friendly so I regularly write a little wrapper script just to provide a short cuts to my most used functions.
The package management system regardless of distro is pkg the the service management is serv. This would also be helpful if you are using more than one system and needn't be complicated.
"What's wrong with "systemctl ls"? What would have been wrong with a shorter command that's easier to type like "sys ls"? It's a core aspect of the system so a name like "sys" would have been appropriate, easy to type, and easy to remember."
"Even worse the output of list-units has overly long lines and manages to be simultaneously hard for humans to read and hard for machines to parse. It uses white space as both a delimiter and within identifiers, making shell script parsing with "cut" etc. impossible."
It's the Microsoftization of Linux. This gives me bad flashbacks of Powershell where exactly what you describe is the case, where the simple is made difficult and the difficult made impossible.
The overengineered, bloated monstrosity that was CORBA also springs to mind. Systemd has been one of the worst things that have happened to Linux in recent years.
While I can agree about some of the commands, they're still far better than what they replaced for the most part, and easy enough to wrap with aliases or little helpers if you prefer.
That said, the problem with "systemctl ls" would be that systemctl has commands to list multiple things things: units, socket, timers, dependencies, unit files, virtual machines, jobs.
I agree it's annoying that the output isn't formatted in more parsing friendly ways, though. Especially because there e.g. is an option to format the journal output in systemctl status as json -- why couldn't they do that for the main output too (indeed, I wish all tools had an option for that)
"Clean, sparse, intuitive, and discoverable"? I have yet to meet a single non-broken production init script that merits a single one of those adjectives.
That's exactly my point. Systemd had a chance to replace the arcane cruft of sysvinit with something clean and well designed, but instead they developed something almost as arcane and unusable.
It's pointless arcana to boot. There is no reason whatsoever that a green field implementation of something so straightforward needs to be so obtuse.
If the complexity of interface exceeds the complexity of the information it needs to take and provide, it's bad design. If the command or UI structure uses arcane terms when straightforward terms exist, it's bad design.
I've used plenty of init-scripts that are named after the daemon in question, support start/stop/reload/status and work fine. What more do you generally need?
The "work fine" part tends to break down horribly the moment something unexpected happens. E.g. pid files being left lying around and breaking restarts.
Then why do all production init scripts for the simplest conceptual tasks always seem to be full of hacks (magic comments and files to declare dependencies, polling, cross-script communication, special cases, tons of comments referencing bugfixes to explain the crazy shit the script has to do, etc)?
Some of those are just things you find in any code (comments aren't bad), some of it refers to specific programs (there will always be 'special cases' if a program supports more modes of operation than "start" and "stop"), some of it is good (bugfixes are not bad), and sometimes init scripts are written by morons.
Sometimes the code is bad. But the model isn't bad. People sometimes use "init script" to refer to the tools that run and maintain network services, and those are completely different.
An init script boots your computer. A service runner is responsible for interfacing with an application and performing complex operations. They should not be confused, but often are, as Systemd conflates the two.
The model of how the init scripts worked is terrible. There's no cross distribution bug fixing of those scripts. It's something added by the distribution. If there's an upstream script, it's often buggy. Or it didn't integrate with the 'speed init scripts up hacks' that various distributions were doing. Huge time sink for no particular gain.
Those scripts were different per distribution. Sometimes shared, but then had to be copied basically manually from one distribution to the other. With systemd it's just a configuration file shipped by upstream. Any fixes automatically get reused by all distributions.
The init scripts differed way too much across distributions. This including pointless differences in where various files are located. All of that has become way more standard, thankfully.
I did have various issues with a few init scripts of my distribution. It didn't happen often, but nowadays you usually can do with a config file, way easier.
Yes sure, the problem is me for wanting to spend more time with my friends & family instead of pulling my hair out until 3am to make those damn initscripts work.
