So if I'm understanding correctly, a useful hardening step would be to ensure that each dynamically linked library had it's own GOT & that the tables are marked as read-only once dynamic linking completes (i.e. you can't reach across dynamic boundaries to patch someone else's ifunc entries).
That would improve the supply chain security of code getting linked in somewhere but never executed.
EDIT: Or even better, perhaps ifunc should be implemented in a declarative fashion so that you can't just cause arbitrary code execution from each library you link against. That might be harder to implement at this point considering back compat, but probably is something that could be layered in over a longer time period (i.e. if you build any library with "declaratively linked ifunc" feature bit set, then the dynamic linker will force all linked libraries to have that feature flag or fail to launch).
Currently building most libraries usually involves executing a series of highly complex byzantine scripts, requiring turing-complete environment. This gives attacker an endless attack surface, and when the build process is hijacked - well, the opportunities are there.
Moving to a declarative build process with only a limited state machine as an executor would help. Requiring all source blobs being reproducible might also be something to think about.
This is how we got autoconf in the first place - an under specified and under powered build configuration language (make) which led to people that did need arbitrary logic to generate the build files at configuration time.
Don't limit the build system language. It makes builds more complex, not less.
Having a "simple" declarative build definition won't help if the thing that interprets and executes that definition is shipped with the package, as said interpreter is likely orders of magnitude more complex and harder to personally review. As is what happened with the xz example - the "attack" was hidden in code that is autogenerated by autotools, not the build or configuration definitions.
People put trust in distros and packagers to having something of a review chain - there's 0% chance you personally be an expert in everything executing on your workstation right now (outside of maybe toy systems). I'm not an expert in m4 or bash, but I hope enough experts are in the chain to get to my distro's package library are that such things are less likely. But that is all bypassed here.
I think this particular approach is a one-off, as I know of no other build environment where it's expected to have the generated executable of the build system "helpfully" packaged in the tarball as a packaging step.
If it is in some I'm not aware of, I hope that decision is being re-examined now.
No, the attack was not hidden in code that is autogenerated by autotools. If that was the case, rerunning autoconf (which most distros do) would have disabled the attack.
Instead it was hidden in plain sight in the source that is processed by autoconf.
> it's expected to have the generated executable of the build system "helpfully" packaged in the tarball as a packaging step.
While this is true, most distros as I mentioned above rerun autoconf to ensure consistency in the tests and possibly to include autoconf bug fixes.
The jumping off point was modified autotools output. re-running autogen did effectively disable the attack. The payload was stashed in some test files, but the build system needed to jump through quite some hoops to actually get that into the compiled library. Apparently the target distributions didn't re-run autogen for this package.
This is what the early reporting said but the article has additional info.
The code to include the backdoor in the build was in an m4 script.
The initial reporting said that this code was not present in the github source, but the post-autogen code (including the attack) was included in the github releases.
The article says that this modified script was present in the source on tukaani.org, which was controlled by the attacker and used by the distros as their upstream.
If you downloaded from github and reran autogen you were OK. If you downloaded from tukaani and reran autogen, like the distros did, you lost.
the "build-to-host.m4" file seems to originally be from gnulib, and if that is installed on the system is not required by the build. So I see that as "part of the build system" myself.
I mean the github repository with exactly the same .ac and .am files works fine with local automake/autoconf generation without that file existing. And thus no backdoor (the test files are still there, but "harmless" without the initial kicking off point to actually de-obfuscate and include their contents)
Gnulib is not installable, it is meant to be copied (aka vendored) in the sources of the program that use it.
> if that is installed on the system is not required by the build
This specific file defines a macro that is used by autoconf, not by the build. If it is installed on the system it is not required by autoconf, but then gnulib is practically never installed.
Your original message blamed the backdoor on "the generated executable". This m4 file is not a generated file and not an executable. It is simply vendoring like you often see in other languages.
I think it was more "hiding" as vendored code rather than really being in that category. The git repo never contained that "vendoring", as the m4/gettext.m4 file doesn't exist autoreconf just copies one from a system store, (which on my machine never calls the tainted BUILD_TO_HOST macros in the first place, which also doesn't exist in the upstream xz git repo).
"Vendoring" by copying untracked files into the tarball seems discourteous to the definition. It seems to rely on the "possibly odd" behavior of autoreconf to allow files that happen to have the same name to override system-installed versions? I guess on the belief that local definitions can override them is useful? But that certainly bit them in the ass here. As to get a "completely" clean autoconf rebuild it looks like you have to delete matching files manually.
For example if you've ever taken a look at the bluetooth specs you would not trust a single person in the world to implement it correctly and you probably wouldn't even trust an arbitrarily large team to implement it correctly.
Unless they had a long demonstrated and credible track record of shipping perfectly functional products and an effectively unlimited budget, i. e. Apple and maybe 1 or 2 other groups, at most.
> For example if you've ever taken a look at the bluetooth specs you would not trust a single person in the world to implement it correctly and you probably wouldn't even trust an arbitrarily large team to implement it correctly.
I messed around a tiny bit with Bluetooth on Linux recently. Going to rankly speculate that Bluetooth is such a special case of hell such that it makes a distracting example here.
I mean, as a joke suppose we wanted to design a 3.5 mm patch cord that pitch shifts down a 1/4 step for randomly chosen stretches. It turns out to be easy-- just remove the wire from the casing and replace it with cheap bluetooth chips at either end. You'll get that behavior for free! :)
Compare to, say, USB, where your point above would apply just as well. I wouldn't be distracted by that example because even cheapo, decades-old USB drives to this day let me read/write without interleaving zeros in my data.
Shit, now I'm distracted again. Does Bluetooth audio even have the notion of a buffer size that I can set from the sender? And I guess technically the receiver isn't interleaving the signal with zeros-- it's adjusting the rate at which it sends blocks of the received data to the audio subsystem.
Was Bluetooth audio basically designed just for the human voice under the assumption we're constantly pitch shifting?
Oops, I almost forgot-- the sample-rate shift is preceded by a dropout, so I do get interleaved zeros in the audio data! I actually get a smoother ramp in my Prius changing from battery to ICE than I do in my wireless audio system!!!
Yes. Other than the vulnerability of developers/maintainers the other
big takeaway I get from this incident is that build systems have become
dangerously unwieldy. There's just too many moving parts and too many
places to hide bad stuff.
Build systems have always been like this. It is in fact more recent build systems that are limiting the amount of crazyness you can do compared to the older ones.
Yes and no, but mostly no. This would prevent simple use of ifuncs in this way, but it's important to understand that the author of this could inject arbitrary code into the library that ends up in the address space of a sensitive process. At that point, all bets are off: it could remap the GOT as writable if it so chose, or (this is mostly here for the EDR people will certainly bring it up after reading this) if trying to do that is flagged as "suspicious" or the OS gains the ability to block such a transition, the injected code can subvert control flow in hundreds of other ways. It has arbitrary read/write, code execution, everything: there is no security mitigation that can stop it. If it so wishes it can leak private keys and send them to the attacker directly. It can spawn a shell. Trying to design protections at this stage of compromise is a fool's errand.
>it's important to understand that the author of this could inject arbitrary code into the library that ends up in the address space of a sensitive process. At that point, all bets are off: it could remap the GOT as writable if it so chose, or (this is mostly here for the EDR people will certainly bring it up after reading this) if trying to do that is flagged as "suspicious" or the OS gains the ability to block such a transition, the injected code can subvert control flow in hundreds of other ways.
I think it's important to push back on a very specific point here. It's true in general that if the attacker has added a backdoor in your library, and you're going to call that library's code, you've pretty much lost. Game over. Go home.
But this was a very different attack, in that the attackers couldn't directly target sshd. They couldn't manage to target any library that sshd calls directly either.
The library with the backdoor is code that was never actually called at runtime. This is important because it means it *doesn't* have a hundred ways to reach code execution. It only had a few select ways, mainly constructors and indirect function resolvers.
The amount of weirdness in glibc's runtime loader is not unbounded. There aren't actually a hundred places where it allows random libs to run code before main. And we should take a good look at those couple place that are clearly juicy high-value gadgets to attacker.
When glibc's runtime loader first loads a binary, first reaches a relocation for a STT_GNU_IFUNC, everything is still in a pristine state and no arbitrary code can run without being explicitly called. Attackers don't have magical powers that allow them to run code before you hand them the control flow. At this point in the runtime loader, an ifunc resolver cannot do anything without being caught. It cannot "just open /proc/self/mem" or "just call mprotect". It cannot "just disassemble the caller" or "just overwrite the GOT".
I really want to hammer home how different that is from letting the attacker run code after main. There's nothing you can do if you directly call a backdoored library. But we shouldn't let attackers get away with spending 400ms parsing and disassembling ELFs in memory in an ifunc resolver of all things. Overwriting the GOT like nobody's watching.
The backdoor isn't magic. For all the beautiful sophistication that went into it, it made many mistakes along the way that could have led to a detection. From valgrind errors to unacceptable amount of noise (400ms!) before main.
Well, they picked a target that doesn't get called directly, and found a way to sneak code into it without a static constructor. If that didn't work (and I don't fundamentally think it wouldn't–people aren't checking these very closely; the ifunc stuff is just obfuscatory bonus) they would target something that was directly used.
I would be happy with that result. Targeting something that's directly used by sshd means a much smaller attack surface. It's much harder for the attackers.
The danger with supply-chain attacks is that it could come from practically anywhere. Attackers can choose to target an overworked maintainer in a third-party library, and it's much easier for them than going after OpenSSH itself.
About the OpenSSH maintainers, they're known for being the paranoid amongst the paranoid. No one's infallible, but if attackers are forced to go directly after them instead of bullying smaller libraries, I'll have a reason to feel safer about the reduced attack surface :)
Think R^X bit style. The kernel would participate in such a way that you can seal the GOT to be read only once the linker is done with it. Then it doesn't matter that a library came in later as there's nothing it can do to modify the GOT as there's no mechanism to make it writable. As you mention, the same protection would need to exist for the executable code itself to avoid overwriting code mapped from disk (I believe that's probably true but I'm not 100% certain).
Defense in depth is about adding more & more hurdles for attackers which raises the costs involved. Even state actors have a budget.
Making the GOT read only is problematic as the dynamic linking is lazy by default, and R^X'ing the GOT would make the deferred dynamic symbol resolution or subsequent dlopen calls fail.
It would be simpler to statically link sshd (or, more generically, all sensitive system binaries) and launch them in the R^X mode via, say, specifying the required capability either in the binary descriptor or at the system configuration level (SELinux) – to force-launch the process in the R^X mode for its code pages. The problem, of course, is that not every Linux distribution comes with SELinux enabled as a default.
As someone else pointed out, sshd is set to be eagerly linked which sidesteps the problem of subsequent dlopen calls. But even subsequent dlopen calls should work in my scheme if every library gets its own protected GOT - a dlopen would create a new GOT for that library, populate it & seal it. It wouldn't have access to modify any other GOT because those would be sealed & subsequent dlopens of other libraries would use new GOTs.
> the tables are marked as read-only once dynamic linking completes
Alas this won't work. Dynamic linking is lazy, there's no moment when it's "complete". The correct function pointers get loaded and inserted into the table (in place of stubs) when called the first time, which can be arbitrarily far into the future. In fact in most large library ecosystems (gtk apps, etc...) most of the linked functions are never called at all.
in this case it takes advantage of the fact that sshd is compiled with early binding not lazy. The ifunc resolver function is called early, right after the dynamic libraries are loaded. It's eager binding (that's what LD_BIND_NOW=1 / -W,-z,now are doing) is a security feature, GOT table will be readonly early. Didn't help with security in this case lol
with lazy binding the function resolver might not even be called at all, the link tree is like sshd.elf -> systemd.so -> lzma.so. If systemd uses a ifunc symbol in lzma but sshd is not using that symbol: if lazy binding the resolver will never run, if eager binding the resolver will run. Something the backdoor also took advantage of
If you build for the host architecture you can completely disable ifunc without losing anything. In Gentoo it is common to build with -march=native, and disabling ifunc is as simple as setting -multiarch in glibc's USE flags. I've seen no negative impact from it.
Not really, except for a few researchy object capability based systems. (This is kind of the promise of ocap systems: functions you call can only use what they're given and nothing more.)
If you don't trust a library, you can run it in a separate sandboxed process (or app domain, or wasm container, or whatever your language provides).
That would improve the supply chain security of code getting linked in somewhere but never executed.
EDIT: Or even better, perhaps ifunc should be implemented in a declarative fashion so that you can't just cause arbitrary code execution from each library you link against. That might be harder to implement at this point considering back compat, but probably is something that could be layered in over a longer time period (i.e. if you build any library with "declaratively linked ifunc" feature bit set, then the dynamic linker will force all linked libraries to have that feature flag or fail to launch).