Probably worth considering here - if this was a state-sponsored attack, it's entirely possible (and exactly what I would do if I was running such a program) that there is one person or department actually writing the code and list messages etc, and another person or team who sits between that department and the internet and whose entire job is to ensure that everything that comes out from the development team takes place at actual times and with timestamps that are consistent with whatever story they're using for the project. Possibly even once in a while intentionally inserting a "mistake" to hint to investigators that the person's real location is another chosen place, which is also different from the real place where the project is being run.
It's likely that there is a whole team behind this operation, and part of that team might include tasks such as "socially engineer XZ project to accept a maintainer of our choice."
This was suggested in a different post[1], where there seem to be newly created accounts that came out of nowhere to create a sense of urgency, which might have accelerated the maintainer role transition.
I don't think he was from Eastern Europe, but if you want to look at UTC+0200/+0300, in Europe this only includes Finland, Baltics, Ukraine, Romania, Moldavia, and Greece. But notably if you look a bit down it also includes a good chunk of the Middle East, including Israel.
I don't know about holidays but Israel uses the same daylight savings as the mentioned Eastern Europe countries, ie switching from UTC+0200 to +0300 at around the same time - only difference is that they start daylight savings on Friday instead of Sunday the same week.
I don't know all the holidays but they have commits on Yom Kippur dates in 2021 and 2022. For 2023 it's harder to say because their commit activity is sparser.
Israelis are usually off from the evening of Friday (sundown) till the evening of Saturday (sundown) for Shabbat. Where did you get the idea that they’re off on Fridays during daytime?
In this context, middle eastern countries with Islam as the majority religion have the day off on Fridays.
Not sure why you’re downvoted. When you think of state actors, Israel comes very high (stuxnet etc), as well as the usual US/Russia/China/NK groups.
Unit 8200 in the IDF especially have a very notable reputation.
That’s not to say other counties don’t have capabilities (and this doesn’t look like you need the resources of a group like say the NSA or GCHQ for this particular attack - indeed it could just be a single lone wolf) but it’s noteworthy.
If the only think I knew was UTC+2 (and I don't even think I know that, I'm really cautious with assumptions about what mistakes that kind of people plausibly make), Israel would be very high on the list. But FWIW, 25Dec is a Catholic holiday, and even though it's mentioned on Wikipedia page about Israel holidays, I'm not sure how official that is. I mean, it doesn't seems to be, like, a real Israeli holiday.
I think no commits on December 25 is not enough to go by. He's been active for about 2 years, so that's what, 1-2 Christmases? I assume he doesn't commit literally every day. So it could be a coincidence that he didn't on those particular 1-2 days. Also, in many Orthodox Christian majority countries including those in UTC+2 they don't celebrate on 12/25.
But doesn't Israel also not follow a typical work week? Eg. No commits on Friday afternoon?
Yes, that's what I mean as well: I don't really think this analysis is any near to be considered any conclusive. It's interesting. But this is it. From what I'm seeing, there really isn't very much data. The article makes it feel like there is this huge archive of commits, that draws a picture of a guy working almost every day on these projects with some very much visible gaps in time, but there really isn't. The data is quite scarce. I would need to actually do the analysis myself to decide what I'm willing to believe, and I don't think I want to do this right now. But for now, I don't think we really have anything.
…What I would really want is some input from Microsoft. I don't know what they can reveal, but I believe they must have quite a bit more data, than us. Even speaking about times, well, he must have been using Github more than to just commit stuff. And I'd imagine they have some logs. Also, it's very fair to assume than he was using VPN all the time, but it's also fair to assume that he wouldn't accidentally slip like that, revealing his real time zone in 9 commits. So, yeah, for sure there are people who have waay more data than me, and probably know how to use it properly better than me too. Not sure they will be willing to share, though.
No need to bother with a tracking image. Just scour the message headers of any message sent in the past--there's decent chance there's a few more dates lurking in those headers, and even an IP address or two.
Years ago they would include the IP of the end user. Gmail and hotmail stopped including that more than a decade ago. But they both used to. I think Gmail stopped this practice years ahead of Hotmail.
I have a relative with mental health problems who lives on the street, and he sometimes writes me from public libraries -- kind of luckily for family members who are concerned about him, the webmail service he uses still includes that info in headers. So I usually know which public library he writes from.
It's not just that there's only two Christmases. It's that Christmas fell on Sunday in 2022 and Monday in 2023, and the article already established the attacker mainly worked Tuesday-Friday. The note about the attacker never working on New Year has the same problem (New Year's Eve/Day always lines up with Christmas Eve/Day, so would not have hit the Thu-Fri window either).
December 25 and and Jan 1 are regular work days in Israel, with the legal option that someone can't be refused to take those days off, but have it deducted from their accrued vacation time (there are other days that one can't be refused to take off, but if no available vacation days, have to be offered leave without pay for those days).
NK are one if the most prolific state actor countries. They aren't UTC+2 though.
Greece and Finland aren’t prolific. I’m sure they have the resources, many lone people would have. I don’t think they have the motives though. The US, Russia, China, NK and Israel have the motive, opportunity and track record. I wouldn’t rule out Ukraine either (and Ukraine is GMT+2)
I still think that an individual is the most likely candidate though.
“Trust no one! The minute God crapped out the third caveman, a conspiracy was hatched against one of them! ” - Gen. Hunter Gathers, OSI (The Venture Bros.)
There are parties who have effectively endless resources and motivation to mock up a false-flag event to steer the responsibility for this a certain way. They're all very, very good at covering their tracks, and even the most experienced security researchers will take months or years to just get an educated guess of who could be responsible. Trying to figure out who did this is a waste of time.
The lesson is this:
1. Never, ever install bleeding-edge software in production, for any reason.
2. Pentesters in your organization should be regularly trying to blow holes in your stack and let both you and package maintainers know the result. If they're not, they're not doing their jobs.
3. FOSS maintainers should audit the new code in each release for security issues, particularly for things that are obviously security-sensitive.
4. Donate to your FOSS maintainers; they do insanely important work.
These rules will pay off no matter who or what wants to break into systems.
This fortunately didn't get to stable. This was as close of a shave as you can possibly get without a security Chernobyl in your systems.
EDIT:
Downvote all you want; I'm right.
There are people who very much want to convince you and everyone you know that a certain party is responsible for this and every other attack you read about. They'll go out of their way to do that. There are geopolitical goals to be achieved by doing so.
We're currently talking about banning foreign ownership of an incredibly popular web app in the US right now. If you are a government or commercial entity that would benefit from such a law, how convenient would it be if there happened to be a GitHub user with a Chinese-sounding name who committed an attack vector to the project with a timestamp that looked like it could have come out of the PRC?
And let's say it is someone in mainland China. We find out they are beyond a shadow of a doubt and who they are on a personal level. Some big-time DA's office like the Southern District of New York puts out an indictment for them and requests extradition. The people who are behind this are almost certainly state-sponsored and there's no way their asses are being handed over to US Marshals, ever. You could even argue that if they managed to stumble into a situation where they could be extradited, the government responsible for backing them would "tie up the loose end" before letting that happen, lest greater knowledge of what the organization has been doing fall into enemy hands.
Besides the Bond-style intrigue, there's no practical application to the knowledge. You already know where your users are coming from, most of the time, and might be geo-blocking based on that alone. If not, you know you should be alert to other threats from that region. If you're not, you aren't doing your job.
> Never, ever install bleeding-edge software in production, for any reason.
But for any X which is mainstream today someone was always the first of X. And even then being the second or third is still cutting edge. For a certain kind of company it makes sense to play this way, but if it were everyone then we'd have a tragedy.
That's why you run X in your sandbox/QA/testing/whatever-you-call-it environment, safe from the prying eyes of the public internet and the private data of users. Once it's all good there, and the community has come to a consensus about it being safe and ready-for-release, that's when you can put it wherever you want.
99.9% of releases for packages like this are boring bugfixes and stuff, not earth-shattering new features. You're not at a competitive disadvantage for not having taken this version of xz and having routed all of your traffic through it.
GMT+8, the name (Mandarin/mixed-dialect first name + Hokkien surname) and most importantly the circumstance that the user was seen connecting from a likely VPN exit or hosted server in Singapore [1] are all consistent with a Singaporean identity, whether real or cover. For that reason, the opinion of [1]'s source that the name sounds fake should also be taken with some amount of salt (Mainland Chinese are not always well-informed about Diaspora Chinese communities). Worth noting that many of the holidays mentioned, likewise, are not public holidays in SG [2] - Jan 24th was, but Jan 22nd (listed as working) was a Sunday, inconsistent with a "working on regular working days" pattern most anywhere.
Would be interesting to see a greater corpus of Jia Tan's writings, especially older ones. There are fairly distinctive idiosyncrasies that may be used to distinguish Singaporeans, Mainland Chinese and speakers of various Slavic languages (and Anglo natives).
In the few commits from Jia Tan that I've seen, the English parses (to me at least) idiomatically as a native speaker from either the US or UK, and I did not encounter any signs of Singaporean English. Obviously, this is a very small sample so one cannot read too much into it.
Almost 10% of Singaporeans share the family name Tan. [1]
> the English parses (to me at least) idiomatically as a native speaker from either the US or UK, and I did not encounter any signs of Singaporean English
Singapore is an Anglophone country. Many Singaporeans are perfectly capable of writing (idiomatic) BrE or NAmE, given the medium of instruction in Singaporean education from kindergarten to university is English. Example: me.
That is certainly true, I did not mean to imply this is not the case. I have noticed however that in the Singaporean English acrolect certain words and phrases are used differently than they would be in BrE or NAmE ("I'll send him to the airport.") In less formal registers the differences become larger, which is what I was looking for.
Setting aside the gravity of the situation, I'm 100% gripped by the "mystery novel" side of this. Seeing the internet piecing together the Who? the How? an the Why? is fascinating.
For years I've had `gc` in my terminal mapped to `TZ=UTC0 git commit` for exactly this reason. No need to change system times or git settings. (though perhaps a better way in global config?)
It's not even for nefarious reasons. I travel a lot, and don't like leaking my travel itinerary on public repos.
It not only sets the timezone to UTC, but also sets the time to 00:00. So my commits will only have date information. I think only saving the date permanently in the git commit is good enough. No need to leak the precise time I made commits.
Same. My phone automatically changes, and I'll manually change my watch, but when I travel, I leave my laptop in my home time zone. Part of it is I just can't be bothered, and part of it is I want an easy way to know what time it is at home without having to fiddle with a world clock.
Time zone is purely a displayed time consideration. I'm not bothered that the display time on my computer is different from the time zone I'm actually in.
Poor Jia. Two whole years of work down the drain. If you're reading this Jia, remember that you miss 100% of the shots you don't take. Chin up, brother.
Yes, unironically, that, being a huge leak, really convinces me that we all have multiple RCEs on each of our systems. I mean, we already kinda knew that, of course, but even now that I'm typing this it's a bit hard for me to actually believe it. But realistically I think it's pretty much a proven fact. This level of sophistication… what did I even expect? Of course well-payed teams of high-level professionals are working in this area for a very long time by now. This is basically like professional football players competing against office workers in their late thirties who like to kick a ball on weekends. I am comforting myself by thinking that something a bit more high-level, written in python or even golang, or, oh hell, maybe even rust would be more a bit more transparent for other contributors… but even this is probably a lie. And if it wouldn't be, surely tons of super-popular low-level packages written in C/C++ are as good as incomprehensible for somebody who isn't specifically security-auditing this.
I never understood and continue to be baffled by the naïveté of FOSS people giving commit access to core projects to unknown parties. Assume they're a state actor or a crazy person, until proven otherwise.
That effectively means you cannot share commit rights with anybody. How am I supposed to vet someone as not a state actor? Any plausible test could be thwarted by a motivated individual.
A state actor is potentially the only person financially supported enough and willing to do this.
"You're in Zurich this weekend? Oh wow, me too. I'm holidaying with my wife. Let's grab lunch" as they proceed to prepare their military expense form for two business class tickets for them and their coworker
The "hacker" stereotype is certainly stale, but subbing "young person" (and implying not "old person") is equally silly.
I'm 41 and have been awake before 7am maybe 20 times in the past 10 years. Half the reason I still put up with computers is because it's one of the few professions where I can work a 11-7pm schedule most days and still excel.
There are morning people and night people. You are what you are, no judgements, but it isn't something you grow in/out of, nor should be expected to.
> There are morning people and night people. You are what you are, no judgements, but it isn't something you grow in/out of, nor should be expected to.
I knew most of the stuff in that article. If I wake up at 5am on full alert, mind racing, I read for a couple of hours and then go back to sleep when it's possible. I know that I can't function well on less than 7 hours, so I don't try.
The article seems to be taking about people on the verge of entering "the facility", on their slow slide into dementia, who are making do despite not getting enough sleep. I'll get back to you when I hit 70.
No, stereotypes aside, hackers who code diligently tend to wrap their "actual work" schedule around their external obligations, and while the morning is generally filled with distractions like dealing with "morning people", after lunch (an on into the evening) there are generally far fewer interruptions. I can honestly get much more done with an uninterrupted four hours than I can in the entire eight office hours of random phone calls and emails coming in, and I am no longer a "young person", and I'll do some stuff at night just because I know it'll take half the time if I'm not interrupted and I've had a few hours to at least intermittently mull it over. If you're having peaceful mornings, I envy you.
If we're putting stereotypes aside, wouldn't we also put aside the word "hackers"?
In the xz case, the term may be accurate, but we don't know the identity of the culprit(s) and have no empirical basis for even speculating about the work schedules of such people.
> If you're having peaceful mornings, I envy you.
Well, I don't work in an office. I work alone. Like a stereotypical "hacker".
> Notably, on Tuesday, Jun 27, 2023, we see two commits: 23:38:32 +0800 and 17:27:09 +0300. If we do the math, there is about a 9-hour difference between the two commits.
Isn't 17:27:09 +0300 == 22:27:09 +0800, making the commits just about one hour apart?
> Isn't 17:27:09 +0300 == 22:27:09 +0800, making the commits just about one hour apart?
Yes, and even worse:
> Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!
No, this is a difference of almost 10 hours...
Maybe the author inadvertently switched the two analyses? Or maybe the author is just not very good with timezone math... :)
Hi all! One of the authors of the original piece here. Thanks for all your comments – we picked up a lot of cool ideas for future analyses. Just to address four common objections:
- We agree it's unclear whether this was one person, or several ones. But even if it was a team, the timezone idea would still tell us where the team is located (the timings seem too much like regular office hours for a globally distributed team).
- It's also absolutely possible that commit times were changed, or commits were made/uploaded using timed scripts. Still seems like it would be hard in practice to use this to cover up a massive difference in timezone, though, because it would require adding a massive latency -- and xz wasn't developed in a vacuum, but in reaction to other online events like the filing of issues or mailing list posts.
- yes, we did get the two examples mixed up (the two times we say are 10-ish hours apart are actually only minutes apart, and the times we say are minutes apart, are 10-ish hours apart). Update is on the way.
- and finally, yes, UTC+2/3 goes beyond just Eastern Europe and includes part of the Middle East. We also clarified this in an update to the article.
Do you understand the time format there? It lists things as -700 or -800 for me, is that because of my local time zone (Chicago) vs. the mailing list (Finland?).
I think mail-archive.com just shows time stamps in UTC-0700 or UTC-0800 regardless of where you are. That's how time stamps appear on all of the few mailing lists I checked out there, regardless of the sender of the message.
Here is the a distribution of commit times and days of week (hopefully I didn't mess up time zones and plot everything in UTC): https://imgur.com/a/1js5T8L
Something feels weird about the timezone arithmetic here, though maybe I am nuts. Take this for example:
> Notably, on Tuesday, Jun 27, 2023, we see two commits: 23:38:32 +0800 and 17:27:09 +0300. If we do the math, there is about a 9-hour difference between the two commits.
Putting it all into UTC we get:
23:38 - 8 = 15:38
17:27 - 3 = 14:27
That's not 9 hours by my numbers. Or another example:
> Even more damning, on 6 Oct 2022, we see the following: one commit at 21:53:09 +0300 followed by another at 17:00:38 +0800. This is only a difference in a matter of minutes!
Putting it all into UTC we get:
21:53 - 3 = 18:53
17:00 - 8 = 09:00
I think these timezones were swapped (accidentally) by the article's author. If we do the calculation again:
I wonder what the "Lessons Learned" look like from the other side?
Are they whiteboarding the best strategy for distributing the timestamps of commits? Mandatory performance evaluations on all RCEs? Distribute the poisoned commits amongst more authors?
If you don't have your BIOS clock set to UTC and your OS is expecting it to be, you can get strange behavior like this. If they were switching between Windows and Linux, that could explain some of the offsets.
Most (but not all ) VPN providers keep logs and payment info that are subpoenable. You could use something like Mulvad with Lightning Network payments, but I am not sure that even that is fully anonymous.
The Witopia VPN that he used for IRC [1] is US based: https://www.personalvpn.com/contact-us/ and they don't mention neither LN payments nor not keeping logs.
I had a hard time choosing which comment to select to reply, so I chose yours since it's higher up. Apologies if it's irrelevant.
I don't know why most people assume that hackers even bother with stolen credit cards in the first place. I mean, they sure do, but those are your average Joes in the business of refund reshipping and other types of scams.
Those who want the maximum anonymity don't even bother with buying anything. It's as simple as going to one of the popular websites who leak databases, setting up OpenBullet software or spending anywhere from 1 to 5 hours writing custom mail:pass validators to spam requests to either API or login form through (once again) leaked proxies, etc. using leaked credentials. Or simply going into one of those threads titled 'x100 Mullvad accounts" which have already validated accounts with anywhere from 1m pre-paid to multiple years. And there's even a bonus of not being shown as a user of this account if you do not use official App and simply load configuration manually through ovpn, etc.
And then there's proxy-chaining if you're doing something truly nefarious. It's super easy to chain multiple VPNs with few socks proxies.
People behind XZ backdoor to me look much more smarter than myself, so I would bet they took care of this angle and will be untraceable.
There's... actually very little that stops you from sending mail from a non-local postal code.
I've occasionally sent packages postmarked as being from one zipcode from another; as long as it's in the same region, much of the postal processing doesn't care so much.
How do you do that? Walk up to the postal counter, ask them to postmark it, then ask for it back, drive to another post office and slip it in their outgoing pile?
Semi-Presorted post can be acquired pretty easily. Places like Shippo and PirateShip offer it. As long as you put enough money into the postage paid, the post office doesn't really give a shit if it comes out of somewhere weird. There is no requirement that "return address" and "sent from" area are the same so long as they're within the same postal zone.
This is why AMZN packages have a return address in Vegas or similar sometimes.
Departure postal code could be an area with like 1 million people. And if it is a nation state I am sure they could send the mail from another country.
I mean, even if I live in Manhattan, I can easily take a subway train to an area with a different zip code and mail it there. And that's almost the easiest thing you could do -- if you want to hide where the mail is from, it is trivial.
Dense urban areas are often blanketed by a range of spectrum sensors for the purpose of retroactive correlation (e.g Palantir) with other metadata sources.
Expert level opsec - very rarely make deliberate "mistakes" by having an inconsistent timezone, or tunneling through someone's compromised home device instead of a VPN, to throw adversaries on wild goose chases.
Shame on me for not reading more before my now removed comment. Shout-out for doing this level of analysis and guessing, as with everything in this incident, fascinating and tantalizing to wonder about.
It is interesting, at least one of the things listed as suspicious is something I do with some frequency. I will sometimes work on what is logically three commits and not chunk them out until the very end.
A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?
> A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?
I don't think much speculation is needed. RPM and deb catches every Enterprise Linux distribution, to the best of my knowledge.
rpm catches Red Hat and all derivatives like CentOS, Alma Linux etc, as well as SuSE, Amazon Linux, and even Microsoft's Mariner Linux (now renamed Azure Linux).
deb catches Debian and Ubuntu.
That's a huge global install base just through those two targets.
I don't think it was specific effort to exclude others, they took efforts to ensure it happened at packaging time, to reduce chances of detection. So eg other xz devs didn't notice it when compiling it, and so on.
> Obviously it would help avoiding detection since the backdoor has the side effect of slowing down things.
My understanding was that this was the result of a bug? But I understand that part less than any of it, so I well could be wrong.
>And to me it's less that they don't care, more that their script probably only works or only has been tested on the ones using .rpm/.deb.
Now that, strikes me as more likely. Maybe they wanted to be relatively sure that they wouldn't be detected and needed to select a test matrix to test against (SELinux/landlock config across rpm/deb distros, versus across every distro.)
I guess what I'm getting at is... did they have an intended set of targets and knew they were running deb/rpm, on top of any other factors?
> A bit random, but has there been speculation on why this seemed to only target rpm/deb packaging?
The payload only works on systems that connect their sshd to systemd-notify -- which is not true for many Linux distributions beyond Debian-based and RH-based anyway (e.g. Arch Linux doesn't do this).
It would be interesting to see how the commit timestamps change around time change. If he has a job in a country that observes DST they would likely show an abrupt change.
I would remind people that it is not difficult at all to have someone's sleep schedule match another country, particularly when we're dealing with what seems to be a state-sponsored attack. Times are a red herring - don't be fooled.
Maybe a consideration: this isn't necessarily a nation state. It could literally just be one or more individuals trying to set up some sort of crypto heist. At this point we need some sort of new adage to the effect of "never attribute to nation states what can be attributed to crypto"
To me this is false. They usually like quick profits. Look no further than Ethereum and defi chains like Optimism,Polygon etc. It is so easy to write a malicious contract that even I with more than 12 years in the cryptocurrency space got had and lost 1k.
The attackers would have to know the distributions used by the company they are targeting, that connections are open to the internet, that the company won't change their setup over the course of several years.
Everything you are saying is so far out of the realm of possibility that it makes me wonder if you follow this stuff at all.
If you had 2+ years of spare time, and these type of skills, the attacker would be far better off just trying to get a job inside the organization they are targeting.
> If you had 2+ years of spare time, and these type of skills, the attacker would be far better off just trying to get a job inside the organization they are targeting.
If some nation state actor dis this, I imagine they couldve easily modified times of their commits, takin into consideration holidaya and such. Also, nation state actors have tons of resources. They could have people waiting to make commits or schedule it with something like cron.
I suggest not putting a UTC+8 timezone graph as header image, when the conclusion is he's likely from Eastern Europe (not saying you should put one showing EET timezone instead).
I understand that people should read the full article instead of drawing any conclusion from a mere image, but lots of people don't (hence why clickbait works). I also think it's a little tasteless, if not misleading.
The whole enterprise is a little tasteless with a chance of worse. Not some substacker or tweeter speculating in itself - people are obviously free to let their speculation flag fly. But you start amplifying this stuff on an open internet messageboard and next thing you know, it's reddit 'found' the Boston bomber. The local downsides outweigh the gawking value and I say this as a dyed-in-the-wool internet gawker.
> understand that people should read the full article instead of drawing any conclusion from a mere image, but lots of people don't
People who don’t read the article and find themselves lacking an informed or nuanced understanding of the contents of said article are the only ones to blame here. Full stop.
That HN doesn’t have a culture of shaming those who don’t read the article is partly the fault of the rules here which finds “read the article please” to be a distraction/nuisance but don’t mind that this often results in misinformed discussion propagating toward the top of a thread.
> you will often not be able to change the time without adding latency, i.e., withholding commits and thus delaying project development -- and even then, getting it right every time is hard.
Someone with access to the right machines could easily check whether this is the case. A lot of projects have set up webhooks on github to sync new commits with their own bug tracker, for example, or testing infrastructure. Logs on these systems would tell you exactly when the commits were pushed.
> Generally, anonymity in the free software sphere is a good thing: software is inherently based on accomplishment and merit, and there is no reason to know anything about a person’s identity.
This is an interesting take. Most software engineers I know love to show off their FOSS contributions. It's a place to do what _you_ want to do, free of budgets and MBA interference. I guess in certain parts of the world you probably want to keep that sort of skill set under wraps, though.
I think personality matters a lot here. One person may flaunt their open source contributions to their colleagues because they like the kudos. Another person may quietly commit and never publicize themselves because they don’t like being the center of attention. Both can even be true of the same person at different points in their careers!
Giant, fragile, pointless build infrastructure is always suspicious. Just take k8s or any other Go cli utility unable to build directly as it was originally intended back when there was go get and now go install.
Simplicity and clarity translate into security, maintainability, and reliability.
Would be nice if whoever made the analysis would attach the actual data in csv or something. I mean, it's a bit handier to overview than actually cloning the repo(s) ang using git log, if I'm not gonna dive deep.
Maybe something that can be done if it hasn't been done already: semantic analysis across the different platforms they communicated on, to determine if it was the same person. Also semantic analysis on the same platforms using different cohorts of time (days of the week, months of the year, quarters, whatever).
Also: analysis on the email chains they interacted with. It's a possibility that they may have responded to their own emails in their mailing lists with fix suggestions/requests for clarification etc.
Everyone here is acting as if it's impossible to script git commits or email replies as a scheduled task or even put someone on the night shift to click send - to give the appearance of being somewhere you're not.
There was WAY more than one person involved here. Every email, commit, etc. was a deliberate task that was carefully planned.
Saying "Eastern Europe" as opposed to "Moscow" is a little coy, isn't it? They're in the same time zone and one seems likelier than the other, to put it one way.
Edit: speaking of, these guys might also have some time pressure at the moment with regards to hacking stuff lol. Kind of fits the bill with what we know about the hack
I dont know what the XZ backdoor is about but, to assume someone putting backdoors into software would work usual working days and hours has to be very naïve. The whole premise of the article is based on a false assumption IMO.
If you’ve missed probably the largest cyber security story since stuxnet, and arguably bigger than that, I suggest you start looking at the last few days.
That’s a stretch. Stuxnet was the first acknowledged state cyber attack, utilized multiple zero days, and destroyed nuclear weapons manufacturing facilities. Bigger in scope sure, but bigger unconditionally? I don’t know about that.
From my point of view, from what we know today, it is bigger than Stuxnet because:
Stuxnet: aimed to delay one nuclear facility that was still being built
SSH pre-auth RCE: root access to most servers on the planet, impacting everyone from hobbyist self-hosters (that's me) to large businesses (of every type imaginable) to probably even some of the security agencies around the world. With SSH's track record, a lot of people choose to trust it as their internet-facing access protocol
I expect that whoever made this would have picked their targets carefully to avoid revealing the backdoor, so probably they'd aim (at least at first) at a handful of businesses of interest (advanced chip manufacturing, say) and governments, rather than causing tangible widespread issues in some way. Theoretically, though, (perhaps upon being discovered) if they'd just drop `rm -rf && poweroff` on all reachable systems (perhaps having it spread into networks in a worm-like fashion, setting a timer for 30 seconds so that it can propagate), most computer-based systems would just stop working (either killed themselves, or from failing routers or other systems they relied on), and lots of them would lose data because their backups either weren't working or were also impacted. Consider just how much involves a computer today, from infrastructure to hospitals. That's a whole lot more impact than a delayed nuclear program due to some unpublished Windows exploits
What modulates this potential impact is the question of how many important systems run something other than the OSes they were currently in the process of targeting (Debian, Ubuntu, and Fedora afaik), how many servers require a VPN (or similar) to access ssh and have no outward-facing ssh server anywhere on their network, and how many years it would have taken to uncover (more and more systems would have been running this over time)
> SSH pre-auth RCE: root access to most servers on the planet, impacting everyone from hobbyist self-hosters (that's me) to large businesses (of every type imaginable) to probably even some of the security agencies around the world.
No, absolutely not; that didn't actually happen. Most servers on the planet that are running x86_64 Linux are probably not running a rolling-release distro based on .deb or .rpm. Rolling-release (or a beta version of the next OS release) is required because I can't imagine one of them updating to a new version of a library like this until the next major release of their OS, and .deb/.rpm because those are the only build environments where the backdoor would get built into the library. (And on top of that, only systems where sshd is patched to link to libsystemd.)
Also: while I will admit that there are many businesses with abysmal security practices, most will not have ssh exposed to the public internet; that'll require access to the corporate VPN as well. You note this, but fail to recognize the impact.
Even hobbyists probably aren't hit by this in large numbers. I run a VPS for a variety of things, and it runs Debian stable. Assuming this backdoor attempt was never found, it wouldn't get installed on it until mid-/late-2025, when Debian trixie is likely to be released (plus some time for me to get around to upgrading it).
I did have the affected package on my laptop, which runs Debian testing. But I don't have sshd enabled on it all the time, and even if I did, my laptop is rarely on a network without NAT (and I usually opt to tether to my phone when in public rather than use public WiFi). All the other machines on my home network are also running Debian stable, and were unaffected.
I suspect that precious few systems even had the backdoor installed on them, and of those, even fewer were accessible directly on the public internet.
Now this could have been bigger than Stuxnet, if the backdoor had remained secret for -- and I think I'm being generous here -- another year or so.
See modulating factors at the bottom. In this case, it didn't because it was caught before making it into a stable release, but only due to sheer luck. The story is still so much bigger than Stuxnet to me because its aim was an actually tangible impact on millions of people's lives
> Now this could have been bigger than Stuxnet, if the backdoor had remained secret for -- and I think I'm being generous here -- another year or so.
Agreed on that! (To be clear, I still think it's already bigger, not because of the impact it concretely had but because I perceive it to having been very close; but I can understand this/your point of view very well)
It's not clear that the distribution mechanism and many of the other requirements would be so broadly met that "access to most servers on the internet" is a correct description of the scope.
Or that, once in the wild, that the author would have their pick of the litter when choosing targets. It's also true that the exploit has several kill switches in it, and so even vulnerable servers could be protected by simply setting an environment variable.
Which, honestly, makes this entire attack all the more strange. It had incredibly unclear value and longevity. It makes me wonder if the author didn't have different, parallel aims, or if this wasn't a type of practice run for a larger attack in the future.
Since they would have "their pick of the litter" why did they then include kill switches?
A scary scenario is a rerun of the Viasat hack https://en.wikipedia.org/wiki/Viasat_hack
I.e. wipe all servers worldwide at the morning of an attack, but before that you install the kill switches on your servers and those of your allies.
To make it harder to detect and reverse engineer. There's a plethora of disabling functionality in the exploit. This is bad, but the original post is being hyperbolic.
And that would likely be precious few, at least today. Only a teeny tiny percentage of servers out there were updated to a vulnerable version. And an even smaller percentage likely had their sshd port exposed to the public internet.
Sure, if the backdoor hadn't been found for a couple more years, the impact would have been much higher, as the backdoored version would have made it into actual current releases of the popular server distros, and companies gradually upgraded.
