i work for a diesel truck maintenance and repair shop and its been hell on earth this morning.
- our IT wizard says the fixes wont work on lathes/CNC systems. we may need to ship the controllers back to the manufacturer in Wisconsin.
- AC is still not running. sent the apprentice to get fans from the shop floor.
- building security alarms are still blaring, need to get a ladder to clip the horns and sirens on the outside of the building. still cant disarm anything.
- still no phones. IT guy has set up two "emergency" phones...one is a literal rotary phone. stresses we still cannot call 911 or other offices. fire sprinklers will work, but no fire department will respond.
- no email, no accounting, nothing. I am going to the bank after this to pick up cash so i can make payday for 14 shop technicians. was warned the bank likely would either not have enough, or would not be able to process the account (if they open at all today.)
Remote monitoring, analytics and diagnostics have a significant impact on uptime, utilisation and profitability. You're thinking in terms of a single machine, but the managers of machine shops are thinking in terms of a complex process across many machines and often across many sites. Some of that functionality could be delivered using an airgapped network, but a lot of important features essentially require an internet connection.
That’s not a lathe nor a CNC system. Again, which CNC manufacturers are installing windows + crowdstrike on their machines just so they can spy on their customers? You’re all just spreading conjecture. This attitude isn’t at all as widespread (nor profitable) in low(ish) volume B2B hardware sectors.
These industries have terrible track records wrt security and even software robustness, but they don’t routinely spy on their customers for weird marketing reasons. If there’s remote connectivity it’s for real reasons (eg remote maintenance, updates etc).
The suggestion that CNC machines run internet connected windows+crowdstrike just so the manufacturer can spy on their customers strikes me as pretty ridiculous and your garage door story doesn’t really relate. Much more likely that they do it for (possibly bad) non-malicious reasons.
Why, whY, WHY...are these things connected to the internet?!
It's so that the support engineer at the manufacturer can log in to troubleshoot. And then company IT support sprinkles a layer of antivirus on top. That's how we got here.
>> Why, whY, WHY...are these things connected to the internet?!
Because SCADA systems. It's worthwhile to have an overview of an entire plant up in the main office. You can easily see what's running, what's not and what's got problems that need fixed.
Now for a small shop running jobs individually, they should definitely NOT be connected to the internet or even the LAN. But hey, some people think a thermostat needs to be on the network so there's that...
Tinfoil hat: Government might want to track/limit/<remotely brick> CNC machine usage someday to say prevent weapons manufacture and encourages this behavior in a similar manner to the way it encourages social media platforms to censor speech. Some of the really advanced CNC machines have GPS in them and won't work in "bad" countries.
CNC literally stands for "Computer numerical control". They're like the OG 3D printers, they just work subtractively than additive, and at much much much better precision.
You absolutely need computers to control them and loading up models via USB sticks becomes annoying rather fast, so naturally the control computers are network connected.
"Network connected" or "conveniently programmable" !== "Internet connected"
It was a rhetorical question. I'm sure the GP knows what the machines are and why they might need some kind of convenient data supply.
Both manufacturers and on-site IT teams have simply gotten cavalier about internet connectivity, network isolation, automatic updates, etc -- convincing themselves that the catastrophic risks that come along with these processes will either not happen to them or will only happen when someone else can be blamed.
Because almost everything industrial runs windows because that's what the devs of those companies were most familiar with since MS-DOS days and evolved organically over time to modern versions of Windows due to great backwards compatibility and platform familiarity.
Those aren't embedded systems though, but mini PC computers. And embedded systems often run bare metal C code, not always Linux, especially for spindle/servo control where they get their commands from that PC.
Can confirm this is the norm in machine shops. I encounter systems running dos, 3.1, 95, 2k and mostly XP constantly. I rather prefer the old dos systems of the obsolete stuff. Less variables. It is easier and more reliable to freeze the tech in time than it is to manage updates.
My last CNC job was just a 98 pc that dropped into dos to load programs, this must have been right around when win10 came out.
Sneakernet and floppies made it secure enough, but the main network where all the orders were handled was... terminal based.
There are a lot of things running Windows because it's pretty straightforward to write a user-mode driver to interact with custom hardware compared to Linux, where every driver needs to be in the kernel and built with the kernel. Yes, there's DKMS, but it's still more of a faff than the relatively plug-and-play mechanism that Windows offers, especially since Vista.
I like the idea that technology is so unreliable in star trek because the computers are all centuries of software accretion with Windows way down the stack somewhere.
The late great Vernor Vinge explored this in A Fire Upon The Deep. One of the characters is/was in a former life a programmer-archeologist. The idea being that so many thousands of years in the future every relevant program has already been written, so his job was to comb the archives for the right mix of codes and integrate them, rather than right something new.
“So we've got this CNC controller written in Rust from 2036, and, ah, here is a GUI for something like that written in late 90's Visual Basic 6… Just combine those and…”
“So uhm, you do know what you are doing, right?”
“Sir! I am a programmer-acheologist! Oh this is fascinating… Hold on, I must unearth and preserve this beauty of a BAT-file before we can go any further.”
Because Microsoft is giving away licenses to unis, esp in developing countries. IT jobs are seen there as a way to earn a good living and you get hordes of people who know nothing but Windows. That's how you get into the situation where most of the toolchains for embedded systems run on Windows, software for embedded systems is written on and for Windows, and so on. And then, one botched update fucks up everything.
Machinery shipped to users usually do not allow for the users of the machinery to "boot into safemode". Thank John Deere and the anti-"Right to Repair" crowd for that.
These things are "cost optimized" and don't feature the kind of remote management iDRAC/openBMC/piKVM that would allow it to be remotely fixed. Embedded windows connected to the internet is super ***.
CNCs might not allow direct Windows access for end-users and require on-premise support from the manufacturer. Our cnc can be remotely serviced… if Windows boots.
If you’ve got physical access to the machine it’s your machine. All you need is a USB port.
I’d expect that the manufacturer puts out their own fix which basically copies crowdstrikes suggestion. I’d even suspect it by the end of the day today.
The fix is really simple, and luckily also very simple to automate. It’s going to be a lot of running around for IT staff (if deputized helpers!) but this should all be over by the weekend.
> If you’ve got physical access to the machine it’s your machine. All you need is a USB port.
You're a few years out of date here. Physical access is not the end like it used to be. We live in an era of hardware-backed anti-tamper and signed loaders/kernels.
If you have a way around it, I suggest you start reaching out these companies because you could make a lot of money.
- our IT wizard says the fixes wont work on lathes/CNC systems. we may need to ship the controllers back to the manufacturer in Wisconsin.
- AC is still not running. sent the apprentice to get fans from the shop floor.
- building security alarms are still blaring, need to get a ladder to clip the horns and sirens on the outside of the building. still cant disarm anything.
- still no phones. IT guy has set up two "emergency" phones...one is a literal rotary phone. stresses we still cannot call 911 or other offices. fire sprinklers will work, but no fire department will respond.
- no email, no accounting, nothing. I am going to the bank after this to pick up cash so i can make payday for 14 shop technicians. was warned the bank likely would either not have enough, or would not be able to process the account (if they open at all today.)