Barracuda is providing replacement hardware at no cost. This critical piece of information is buried halfway down the article:
> In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost
Obviously the time and effort to replace a device isn't free, but at least they're doing the right thing by acknowledging the issue and doing what they can to fix it definitively.
But correct - its the right thing to do regardless of the true, ultimate motivations...
But what I'd like to know is the impact of this? Like - how much corporate opportunity loss may have been created through information breaches which were unknown... That will never be known, unless we can assuredly say 'none' which is doubtful...
-
>Barracuda said the vulnerability existed in the Barracuda software component responsible for screening attachments for malware
Heh -- uhm... Isn't that like _THE_CORE_ component of the devices job?
Aside from ensuring filters on attachement egress blah blah...
--
>No other Barracuda product, including our SaaS email solutions, were impacted by this vulnerability
I'd like to hear them directly say that this specific ESG device line was NOT used in their Email SaaS offerings?
And to know exactly what ESG kit they are using?
Seems like a reasonable request if you're a large (or any) customer of theirs...
You’d be surprised how many “virus scanners” are implemented with poorly coded C parsers running as privileged user with no sandboxing… just saying, kind of asking for trouble.
I remember awhile back the lead of the chrome dev team getting into a twitter fight with some engineer from a popular antivirus tool because, as I recall, AV tools are basically just more attack surface. I tend to agree.
Windows/client AV tools have a who-watches-the-watchers problem. If you're trying to detect threats that exploit OS, you need to be even lower level. But if you're lower, then you're an even more inviting target.
Unfortunately, add on that no customer makes a profit off their IT security org. Which disincentivizes excellence and incentivizes feature check-boxing at the lowest price point.
Which ultimately produces a highly privileged piece of software developed on budget salaries.
One reason MS security was a game changer (once the company got off its butt and admitted security was an existential threat to their OS sales) -- they could afford to burn great magnitudes of money to deliver.
(No offense intended to any of the amazingly brilliant non-MS Windows AV folks out there)
Malware Bytes helps identify, stop and remove malware Crypto Miners!
In my social circle, the kids Gaming PCs getting infected is very common.
Recently one of my boxes got hacked via a QBitTorrent exploit, and I didnt have Malware detection running, other than the built in W11 system.
I installed Malwarebytes and it detected and correctly removed the malware crypto miner.
FYI they exploited QBT via the web interface, which had default settings, but wasn't exposed via port forwarding to the web. It might have been via UPNP, which was enabled. No idea - but it's a common exploit used to DL a torrent then run a post DL script .bat file to DL and run a crypto miner.
I'd literally had QBT running on Windows for a couple of weeks, having switched from a dockerized setup on a Mac Mini. How Windows allows the running of a .bat file to DL an .exe that can run a crypto miner is just a bonkers lack of security!
Naturally I had to nuke the box from space anyway :-)
> In qBittorrent before 4.1.7, the function Application::runExternalProgram() located in app/application.cpp allows command injection via shell metacharacters in the torrent name parameter or current tracker parameter, as demonstrated by remote command execution via a crafted name within an RSS feed.
A virus scanner needs to implement parsers for so many different formats on the budget of a medium software company, with a pressure to deliver fast. the chances of not making any mistakes are small, and antivirus needs to run with high privileges.
What could go wrong with a process running with the highest priveliges parsing every piece of data passing through the system. Windows defender has had some nasty ones, uncovered by project zero
This actually seems great. They’re taking the hard path here admitting the devices are compromised, that the state can’t be trusted, and that the machine needs to be tossed. Then they’re offering to replace the vulnerable devices.
I’m used to companies taking the “deny, deny, deny” route and these articles are then written by their community saying the response is insufficient.
Who would want a replacement "email security gateway" after the previous version was found to do the equivalent of system(<untrusted email input>)?
When is our industry going to agree on a standard where we go "no, that is too dumb, too far"? This "oh there was an issue but it's fixed now" doesn't begin to address the severity of the fuckup this is.
It doesn’t take much for a machine to become irrevocably tainted, so the fact that they are doing a recall doesn’t, in and of itself, say much about the magnitude.
Do you have some other data to substantiate your ire or are you just piling on?
> The vulnerability stemmed from incomplete input validation of user supplied .tar files as it pertains to the names of the files contained within the archive. Consequently, a remote attacker could format file names in a particular manner that would result in remotely executing a system command through Perl's qx operator with the privileges of the Email Security Gateway product.
I would not be comfortable to continue using a product that contained such an egregious flaw.
So you will never buy a car because every manufacturer at some point already fucked something up with accelerate/break which means none of these can be trusted anymore?
One of the first things I virtualized was a Barracuda spam appliance, the fans were extremely loud and the AMD chip was slow, so the interface to your spam inbox was slow, etc.
Pulled the hard drive and loaded it up into VMmware ESX 2(?). It flew and there was one less server/appliance in my rack.
Barracuda was one of this early companies that made a very useful product out of free software. I idolized that and wanted to work there.
Then they were taken over by some outside company, started making all kinds of products that weren't in their core, support faltered...I'm kind of surprised they are still around.
After a device is compromised, patching it won't help. Their software team (quite reasonably) says that after anyone else has got root on the system, their software patch system probably can't reliably be sure to get rid of the malware.
So, they are offering to replace any device in this position. The replacement is probably a refurbished one from another customer, but has gone back to the factory to be flashed by some debug port - and using that you can wipe all the flash memory and be pretty sure there is nothing evil left.
Given that this looks to be just a particular build of Supermicro server, I wonder what mechanism the malware uses to achieve persistence such that a reformat or FS restore wouldn’t take care of it. Does anyone know if these devices have supermicro IPMIs on them? Those are notoriously insecure (like most lights out managers) and a great place to hide malware persistently.
This points out a major issue with IPMI and "management engine" components on motherboards. If a vulnerability is found at the lower levels, you may have to replace the hardware. Vendors may be more reluctant to put that stuff in if it leads to legal liability.
IPMI firmware can be patched, as can the BIOS which contains the microcode that is loaded. Once the OS boots, updating the running microcode included is trivial (in Linux and I believe also FreeBSD?) and can be done before the system's network interfaces come up.
If your IPMI is in any way exposed to anyone other than your administrators, then you have other problems. These interfaces should be segmented away from all other networks, irrespective of any vulnerabilities they could have.
That's great and all, but if there was a known barracuda OS vuln that allowed attackers to gain root access, then they could gather information about the hardware, which includes the ability to interact with the IPMI (which runs it's own separate OS) to update the firmware or whichever way they're able gain persistent access to the IPMI device's OS.
If you are halfway competent, you do not use inband IPMI at all. IPMI belongs on a separate ethernet jack or at least a separate VLAN, connected only to a separate secured admin network. Which sysadmins can connect to, but which doesn't have connectivity to other parts of your network or the internet.
But I agree that you should filter all the relevant ports in all networks, just in case somebody screws up.
The writeups I’m reading suggest that the malware was specifically designed to maintain persistence via firmware (i.e. rootkits), typically attributable to state actors.
That article captured the imagination of the public, but has been derided by any and every cyber security professional worth their salt almost from day dot.
Despite this, Bloomberg have refused to retract it or supply any credible sources, presumably because and continues to draw traffic.
And yet "do we have any backdoored supermicro hardware" is a common question from non technical management, even today. It's infuriating that it hasn't been retracted as I'm still talking about it. It's also at the forefront of insurance questions
Easily solved by said security professionals auditing the security of the devices, refuting the claims in the Bloomberg article, and throwing it all up on a blog with catchy “No, china didn’t backdoor servers headed to US gov datacenter racks” title.
It would get awkward when they find the US backoor built in to every server headed everywhere ... I guess technically outside the scope of the headline so you can just leave it out.
The point is, if the original article is bogus and full of lies it should be trivial for the security community to debunk any back door claims and set the record straight, yet that isn’t being done while still calling for the original article lacking any proof be taken down.
Someone on one of these sides needs to provide evidence to support their claims. Clearly the authors of the original article have no intention or it would have been done already.
Careful what you ask for Barracuda: we were already looking at replacing ours with non-Barracuda hardware and now that you are saying old gear cannot be updated all I can say is thank you for making my case to my managers.
If you have to go through the cost of a replacement anyways - as in, the time and effort to actually do it - the cost of the hardware matters a lot less.
I suspect there is a higher level question at play stemming from the "deny" comments. Do you think Barracuda is uniquely vulnerable to these types of threats and, if not, would their competitors make similar efforts to recover?
The mindset that changing a system out has risks and that engineers/ops time is very expensive?
I mean, I hear you, it sucks that being honest about the issue is leading to punishment, but rationally this is just how people are going to end up responding. If they really want to avoid this, don't give it to them for free, pay them for it.
No it’s not. Considering changing vendors because a vendor did the right thing and sent out free replacements for potentially compromised hardware is not something any sane client would do.
The GP didn't say that though. They said they were already considering replacing one vendor with another, prior to this incident.
As I understood it, the calculus was 'do nothing and keep the current barracuda stuff for a while longer or find another vendor and replace the barracuda stuff?' Now the former option is gone, and the calculus is 'replace current barracuda stuff with barracuda or another vendor?' So if you were already considering switching vendors, now is an opportune time to do so. You're going to be paying the eng/ops cost now anyhow, and as many in this thread have pointed out, one or two fairly off-the-shelf hw appliances are not exactly a big spend.
>You're going to be paying the eng/ops cost now anyhow,
No you’re not. The people in this thread saying that have never actually managed technology because a vendor change that also requires a hardware swap is significantly more expensive than the drop in hardware swap.
The only exception to this is if you were already in the middle of the transition and were planning to phase it out shortly anyway.
It is an email server. You can set up some kind of system that “catches” incoming email while you take your hardware offline and replace it. Your internal users won’t be able to send or receive emails but anyone outside the organization would be none the wiser that your system was down. If you notify ahead of time and can keep downtime to under an hour, will it matter?
A large portion of organic at this point run their mail with Google or Microsoft, and are already paying for capabilities in those systems on par with third party offerings.
There is often a counter "but I still get some spam with exchange online/google workspace", to which I would counter that you will still get spam with a barracuda.
You’ll get false positives and negatives with any email gateway - even ML or “AI” backed ones. The offerings from both Google and Microsoft, while adequate, are someways off the efficacy of other solutions.
Is it likely they will analyze the firmware on the ones that come back, then wipe them and resell them either thru a refurbisher or even put the new software on the known-good server and use elsewhere, such as in the datacenter where they have their SaaS offering?
With all due respect (and I really mean that — bcantrill is absolutely deserving of tremendous respect), why would using an oxide server for the hardware be any different (better or worse) than a SuperMicro server?
And further, is it even possible to get Oxide equipment yet? Is there even a timeline? Or is it still vaporware?
A lot of questions in there! Taking these in order:
1. We aren't making standalone servers: the Oxide compute sled comes in the Oxide rack. So are not (and do not intend to be) a drop in replacement for extant rack mounted servers.
2. We have taken a fundamentally different approach to firmware, with a true root of trust that can attest to the service processor -- which can turn attest to the system software. This prompts a lot of questions (e.g., who attests to the root of trust?), and there is a LOT to say about this; look for us to talk a lot more about this
3. In stark contrast (sadly) to nearly everyone else in the server space, the firmware we are developing is entirely open source. More details on that can be found in Cliff Biffle's 2021 OSFC talk and the Hubris and Humility repos.[0][1][2]
4. Definitely not vaporware! We are in the process of shipping to our first customers; you can follow our progress in our Oxide and Friends podcast.[3]
The vulnerability found was an RCE, and apparently at least 5% of their devices are infected.
> The pivot from patch to total replacement of affected devices is fairly
> stunning and implies the malware the threat actors deployed somehow achieves
> persistence at a low enough level that even wiping the device wouldn’t
> eradicate attacker access.
Very interesting. I'd like to think it's probably some technical limitations in the recovery options rather than some weird magickery to persist beyond drive reset. Because, if it's not, everyone is screwed.
This is a problem of bad engineering: not using an immutable OS firmware model with a spare.
There ought to be 3 physically-separate flash devices: bootloader (never changed), OS (managed by bootloader - updates deleted unless signed correctly), and data (wipeable by a physical reset button).
If you don't do this, then you're inviting implants a priori.
> NSA can’t even stop their own tools from leaking, never mind take on this.
The NSA has kept plenty of vulnerabilities under wraps, the tool leaks were a pretty rare occurrence of the KGB/GRU trying to burn NSA ops without directly involving themselves.
Nice, first time in my life I consider buying something from Barracuda. But looks they sell to 1500+ desktops && 5+ officess. Oh well... ;)
But:
"somehow achieves persistence at a low enough level that even wiping the device wouldn’t eradicate attacker access"
It started. Barrakuda at least is nice to replaces hw but soon vendors will be bored by replacing shit they themselve making and customers will be left with voulnerable systems. Simple x86/PC ecosystem is shit, intentionally crafted or not. And, very, very sadly on whole globe only EU is capable of enforcing something to be better. But chances are minuscule...
Seems like they can only patch the application remotely - and not the OS - at least not at the same time (patch OS - then after reboot - let the app look for updates. The time between could have been enough to compromise it again)
Alternatively, they can patch both but the malware achieves persistence in firmware. Which may not even be a sign of particular sophistication, depending on what protections Barracuda had in place.
Or it could be as simple as that the software update application on the device has been patched to re-add the malware after flashing the software update.
This is often the case. VM appliances are all little Linux VMs where the user just doesn't have access to the internals. Heck, that's basically Docker containers too.
Also, Barracuda cheaps out on these massively especially at the low end. I've seen, in the modern Core i years, Barracuda 1U appliances powered by Pentium III processors. I suppose they are powerful enough for the job Barracuda is asking of them, but it's worth a chuckle to see how many years old the chipset they're shipping is.
The hardware is absolutely the cheapest part of the stack for them.
>“One of the goals of malware is to be hard to remove, and this suggests the malware compromised the firmware itself to make it really hard to remove and really stealthy,” Weaver said. “That’s not a ransomware actor, that’s a state actor."
The convention on HN is to link to previous threads only when there are interesting comments there. If readers click on a link like this and don't see that, they'll be disappointed and sometimes come back and downvote the link. That's probably what happened here.
You're of course right that there were lots of submissions but on HN reposts are fine until a story gets significant attention, and the current thread is the first one to do that.
> In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost
Obviously the time and effort to replace a device isn't free, but at least they're doing the right thing by acknowledging the issue and doing what they can to fix it definitively.