It's nice to see a proposal that fits so neatly within existing infrastructure and norms. I just hope they either standardize or drop the extensible type field idea instead of letting it turn into a semi-documented mess other projects can't easily adopt.
A dependency being optional on a technical level doesn't mean that it's optional on a functional level for everyone. `libfido2` mentioned in the pull request is a great example: if your rootdisk is encrypted using a FIDO token, you need that library in the initrd; but otherwise you don't. By making the dependency optional and initrd generators aware of that optional dependency, people that need it will have things just work, while others don't need to install the library.
You know that you use a token to unlock your boot device. You don't necessarily know the specific set of libraries needed to enable the feature, though.
Because the optional dependencies actually do things. By default an initrd-builder would probably copy all deps including optional ones but it now enables you to choose not to do that as well, an avenue that was previously not available.
I don't understand the initial motivation for converting regular dynamic library dependencies to dlopen dependencies. How does that help with reducing the footprint?
It makes the presence of those libraries optional: you no longer need them to execute the relevant tool at all. It'll just mean you can't use features which depend on those libraries. For libraries that are only pulled in for less-commonly used features it makes a lot of sense (the specific case they are doing it for is generating the initrd, which needs a copy of any libraries used by anything running in the initrd, which is almost always going to include systemd, but the systemd in the initrd is very unlikely to use any of these optional features)
I think it's also relevant that the xz exploit made use of the fact that by running code before main, they could modify areas of memory that later get turned read-only. Any library that does get loaded with dlopen can of course still attack the process it's in, but it has less tools available to it for evading detection.
The specific way sshd was infected would not have happened with libxz as dlopen library.
Debian's sshd only uses libsystemd for the notify api. I.e. it doesn't need any feature that uses libxz. If it's dlopen()ed, it does not need to be loaded into the process context to use an unrelated feature.
FWIW, IMO upstream systemd should split their monolithic library and allow users to pick better that way, but this has other implications on DX.
> FWIW, IMO upstream systemd should split their monolithic library and allow users to pick better that way, but this has other implications on DX.
FWIW, upstream systemd has the opinion that no-one should load the library for startup notification, instead they should use the well documented api and just write a message to a socket.
Indeed, that's what the Debian maintainer of OpenSSH did soon after the quick security fix. He replaced the dependency on libsystemd with some hand-made code that notifies systemd by socket.
https://salsa.debian.org/ssh-team/openssh/-/commit/cc5f37cb8...
That's not what they said in their update to the documentation[0] last week, which says that "although using libsystemd is a good choice, this protocol can also be reimplemented without external dependencies".
It calls it a "good choice". Did they say somewhere else that no-one should do it despite it supposedly being a good choice?
sshd should not have used libsystemd in the first place for the trivial notification. And the ifunc stuff is its own security nightmare. Papering over this by dlopen-ing some libs in libsystemd does not address the deeper issues.
> What you gain is that the vulnerabilities will be harder to track down?
No, they're just as easily tracked.
What you gain is that you can refuse to install optional dependencies because they are now optional when they used to be required.
That's a fairly big deal.
Of course, in practice all those optional dependencies will be installed anyways because of other things in the distro needing them. You can fairly object that not much changed, at least for now.
A better approach would be to eliminate a lot of these dependencies somehow. Another approach would be to sandbox the dependencies that cannot be eliminated.
Though only with this particular approach to the backdoor. If systemd had always had this approach (or distros hadn't patched sshd to link it in), the attackers would have focused on a different path from delivering malicious code to widely-used distros which executes in a priviledged context to network RCE.
Wouldn't this systemd feature add a convenient centralized point of attack to inject libraries? Not as open at the user level, but similar to a LD_PRELOAD kind of vulnerability.
Not in a way that isn't otherwise accessible, IMHO. I mean, if you're really concerned about injected vulnerabilities into high-trust software (and you should be!) you should be suspicious of any dynamic linkage at all. But if you're going to do it, doing it late and under affirmative control is almost certainly the right choice.
It's because of how dynamic linking/loading works on Linux. An ELF dependency means that symbols in the library can override symbols in the binary or in other libraries. That doesn't work when the library is loaded with dlopen().
To be honest, liblzma using systemd seems like an unimportant detail more than anything else.
On my laptop there's 141 binaries in /bin/ that link against lzma; an abbreviated list with similar entries omitted below. And that's a fairly minimal Void Linux system – the list for an average Ubuntu system is probably a lot longer. Sneaking in via kmod, grub-install, udevd, or xbps-install seems just as viable as sneaking in via libsystemd. Perhaps a little bit harder, but not much.
It will remain problem as long as you have a single command that runs your malicious code as root. Being directly linked against sshd was convenient, but there's all sorts of devious shit you can pull once you've got root, and it doesn't take that much more effort.
Wouldn't dlopen make you more vulnerable to runtime library attacks? Instead of having dependencies defined at the build/start up time, they will be loaded at runtime on demand. So the dependency library could be changed at any time and you wouldn't have any idea what code you are loading at this time?
So upgrading XZ could even attack already running sshd too?
I would be more inclined to have static linked libraries instead of having lazy loading of them for preventing this kind of attacks..
> So upgrading XZ could even attack already running sshd too?
With the specifics of how the xz injection attack worked, actually it wouldn't. libsystemd would have been loaded to implement the service-started notification, and then there is no reason to use anything further and therefore nothing to trigger the loading of the xz library.
Which is kind of the point of this change: the goal is to load libraries on-demand instead of eagerly, so if there is nothing demanding the library load, the library is never loaded.
Lazy loading would have achieved the same thing with less codebase churn. However, the `dlopen()` thing means that the dependencies are then optional as far as the packaging system go, and therefore you can run with fewer dependencies installed. Also, lazy loading can be turned off with an environment variable -- we would need a way to indicate that a dependency is optional in order to have something like always-lazy loading for it.
There's also "filters" and "auxiliary filters", which can be used to make dependencies optional without having to go to `dlopen()`. However, IIRC filter functionality is very bare-bones or non-workin in the Linux link-editors and ld.so.
Loading at runtime is a feature. You're supposed to lock down the environment for execution so the right libraries get used. The trouble with statically linking is that you have to relink everything to update. You also can't easily audit which versions are in a binary. It seems appropriate for some small userspace programs but not for complex software like services with many dependencies.
This seems a bit nitpicky: you just move the threshold for where you might load a backdoored library from first exec to first use which dlopens that library (which might even just be at exec anyway, I haven't checked the source to see how systemd handles it). I think the focus on the specific means that the xz backdoor used to get into sshd is misdirected: it's code that is run in all kinds of priviledged contexts and has thousands of different ways to turn that into RCE. If sshd didn't load it directly the developers of the backdoor would have simply taken a different route.
Instead of dlopen()ing libfoo.so.7, you should build a private wrapper library, normally linked with -lfoo, and then dlopen() that private wrapper. Then you get dependency tracking for free.
On the one hand, this seems like a slight improvement.
On the other hand, while it’s a bit of a slog, it’s possible to actually sandbox dependencies. libxz, for example, has well defined inputs and outputs. It’s possible to load it in such a way that all it can do is consume inputs and produce outputs that depend on those inputs. (And allocate memory and waste CPU, but that’s DoS at worst.)
That's true. But both in-process and out-of-process sandboxing are doable, as is writing code in a manner that is provably (for various definitions of provability) free of various classes of side effects.
Right, as Lennart points out macOS has supported this for years. And adding a DT_WEAK_NEEDED to ELF wouldn't be that hard. The glibc maintainers even work for Red Hat. Feels like systemd is the wrong place to solve this.
"Jia Tan" was pushing hard to get systemd updated to use the new xz because he saw this change in progress and wanted to get it in a release before this went through.
No they weren't. Jia Tan has never interacted with the systemd developers in any way as far as I know.
It isn't systemd's decision what version of xz-utils to use, it's the distro's decision. And Jia Tan did push the distro maintainers to update xz-utils, but the systemd developers have absolutely nothing to do with that, so your statement is incorrect.
You're correct, and I was overly vague. The key observation is "Kevin Beaumont speculates that knowing this was on the way may have accelerated the attacker’s schedule" Here's the exact wording, from Russ Cox:
2024-02-29: On GitHub, @teknoraver sends pull request to stop linking liblzma into libsystemd. It appears that this would have defeated the attack. Kevin Beaumont speculates that knowing this was on the way may have accelerated the attacker’s schedule. @teknoraver commented on HN that the liblzma PR was one in a series of dependency slimming changes for libsystemd; there were two mentions of it in late January. https://research.swtch.com/xz-timeline
Do you have a source for that? The backdoor was added to a minor release without a SONAME change, and generally such updates are done by distributions and not something upstreams concern themselves with.
I'm not going to listen to a 1.5 hour podcast to find the exact quote, but someone must've misspoke or misunderstood. I did some searching in the meantime, but there's no evidence at all of "Jia Tan" interacting with the systemd developers. They pressured distributions to update xz-utils, but not systemd upstream.
That may be true, but that wasn't the claim. The claim was changes in upstream systemd would limit the timeline for them. Not that they had interaction with each other.
I guess this has to be said, for the another billion times, systemd is not and has never been 'only' an init replacement. And even earlier 'init' systems did more then init.
Just because you call it an "init replacement" doesn't only make it that, it describes itself as a "system and service manager" which should clue you in, that (de)compressing stuff is something that is commonly done during system management.
> It's a very simple approach. All it does is insert an ELF "note" into generated binaries that declare these deps. This information can then be consumed by package managers, initrd generators and other tools.
Lots of things already `dlopen`, lots of things already obscure their true dependencies by doing so. This RFC only provides a new section for those things to declare their dependencies without making the OS responsible for loading them. I think this actually improves the world for everyone, rather than just Microsoft.
Anyone is welcome to make contributions to Linux or the surrounding ecosystem.
>Lots of things already `dlopen`, lots of things already obscure their true dependencies by doing so. This RFC only provides a new section for those things to declare their dependencies without making the OS responsible for loading them.
The OS should be responsible for opening dependencies. Otherwise you get an unaccountable mess. The ELF note shit is just giving users a false sense of security.
>Anyone is welcome to make contributions to Linux or the surrounding ecosystem.
Ha! This certainly isn't true. Even if it was, the source of "contributions" deserves scrutiny.
At least on Linux and macOS, the "OS" never was responsible for this and I at least do not believe that it should become responsible for this; the kernel merely parses the header to see if it lists an "interpreter" and, if it does (as dynamic executables do), it loads that one binary (it might also load the original binary and pass its address as an argument to the interpreter). The interpreter is usually going to be something like /lib/ld.so, and is provided by the C standard library you linked against. You might have multiple copies that work very differently, and a new programming language can implement one that is nothing like the one from glibc or adds new features, etc. This program then parses the executable and does the moral equivalent of a dlopen on all of the dependencies.
That said, I do agree that doing this in "notes" is dumb, and if they need some new feature of ELF like "alternatives" they should at least be coordinating this with glibc if not deferring the spec entirely to them. Hell: given that they previously were actually linking these things, I don't understand why a lazy loaded weak symbol wouldn't work. Does ELF not support those somehow? I thought it did. I don't think systemd is at all the best project to be trying to think through this stuff as they are always much happier adding a new complex mechanism when existing ones would work fine. If ELF is currently missing lazy loading, they should work with glibc to add that feature correctly, not try to end-to-end it through the notes field :/.
To your last point, doing it without involving glibc does have the advantage that it'll also work with musl, instead of starting yet another flamewar about missing non-standard features in musl.
The whole ELF interpreter thing is a security hole as far as I'm concerned. I'm not an expert on it but as far as I know, the existence of ELF interpreters seems to make it impossible to get a definitive list of dependencies.
Assuming you're actually proposing that every possible library is listed in the ELF and loaded before the executable starts by the OS' loader, I would encourage you to try writing an AppArmor or seccomp profile that bans `dlopen` and see how many of your favorite applications still run. This is a completely unserious take that continues to misread the motivations of the RFC (it's not about security) and couldn't even be applied even to just the core of most actually used Linux userlands without an absurd amount of work (think of PAM modules and friends).
Also, have I really misspoken? I was under the impression that many corporations with interests in proprietary software (including Microsoft) are regular contributors to the Linux kernel and ecosystem, but if I'm wrong I'd love to know the details.
The comments here suggest otherwise. It certainly can't provide a security function.
>Assuming you're actually proposing that every possible library is listed in the ELF and loaded before the executable starts by the OS' loader,
I didn't propose that but certainly the ability to load and execute code from anywhere is a vulnerability waiting to happen. Not being able to accurately enumerate dependencies is also an issue. If the OS was solely responsible for locating and loading dependencies, you could analyze any executable in isolation and get a definitive list of dependencies.
>I would encourage you to try writing an AppArmor or seccomp profile that bans `dlopen` and see how many of your favorite applications still run.
Just because it is that way now doesn't mean it needs to be.
>Also, have I really misspoken? I was under the impression that many corporations with interests in proprietary software (including Microsoft) are regular contributors to the Linux kernel and ecosystem, but if I'm wrong I'd love to know the details.
They do make contributions. But large corporations aren't "anyone" and their contributions are not unquestionable. They do have their own interests which are in conflict with the broader open-source ecosystem.
yes, the particular decisions could be fine or terrible; but having microsoft employees make them is a terrible idea
i mean microsoft is still the same company that makes your linux installations unbootable regularly when you install or upgrade windows, and which doesn't allow you to install linux on arm hardware designed for linux
I was under the impression that many large corporate actors with interests in proprietary hardware and software make regular contributions to the Linux kernel and ecosystem.
Also, why does the person making the change matter? If they're not a vexatious time waster, then who cares who they are as long as the changes are coherent?
Finally, I'll note that the GitHub issue linked is against systemd/systemd. It's not a binding change to the entire ELF format, it's a request for maintainers of distros to comment on whether or not adding this optional field might be helpful to them. They can just say "no", and who published the RFC remains irrelevant.
Is there evidence that any Microsoft execs are pressuring Poettering? Wouldn't other people that work on systemd and linux object to poor architectural decisions? This is a key feature of open source development, it's just not so easily subverted.
I mean, Microsoft pays him a regular paycheck. I think you're moving the goalpost quite a bit to ask that the community accept anyone's contributions until solid proof of malicious intent is shown.
Are you prepared to accept my contribution? I sign my name "JiaT57notThatJiaTan"
I'm not moving any goalposts, you're saying someone with decades of open source kernel development and a solid reputation is no longer trustworthy because of who now signs that paycheque. That's just nonsense, and has nothing to do with the scenario you posed.
[ ] When disagreeing, please reply to the argument instead of calling names. For example, arguing with "nonsense" and "has nothing to do with the scenario" detract from HN
[ ] Comments should get more thoughtful and substantive, not less, as a topic gets more divisive. Avoid saying "I'm not ___" in a reflexive way
[ ] Please don't use Hacker News for political or ideological battle. That tramples curiosity such as for Lenart Poettering's "a solid reputation"
[ ] Please don't complain that a submission is inappropriate.
Most distributions care more about whether people deliver software that solves real problems, than worries about some contributor's employer's behaviour from the last century.
Not just the core of Linux user land, but also the pre-linux boot process with systemd-boot. Don't forget that MS has enormous influence over UEFI too.
MS is gaining a lot of technical influence in the open source world. As soon as the regulatory environment shifts, they're going to take advantage of that.
For someone who works at MS, I'm surprised he didn't try to add delay-load imports to ELF instead, which is the PE equivalent that's existed in Windows since Win32.
