That's one of the ideas behind use of tape storage today interestingly. It's pretty hard (although not impossible) for hackers to get at tapes sitting on a shelf.
There's no perfect scheme, but the best mechanisms I've seen for this in practice involve HSMs so that nobody, bad guy or good guy, can change the keys used for encryption without physically pulling the module, and it can be disconnected and stored at a safe location once the backup is complete. Each redundant set of a backup has an HSM, and the HSMs are cold stored in different physical locations.
But yes, this is hard to get right as I hinted by the "but not impossible" quip.
Many years ago I was working for a defence contractor and an idiot hit the TOTAL ERASE button on an HSM. This all went to hell when no one could work out how to load the keys into it again for nearly two weeks...
Practically you don't want to do that. You want them to be useless to someone who steals a tape since it could be a long time to never before you realize it's gone. All the tapes on the shelf look alike.
Since the tapes are physical, can't you protect them using different strategies? Like how banks protect stuff contained in safe deposit boxes? I imagine the stuff stored in safe deposit boxes can be quite sensitive too.
That's Iron Mountain's main business model, but it's considered good form not to leave it to them. Shit happens, and tape drives make it super easy to encrypt anyway.
That and banks don't seem to really try very hard nowadays, at least for retail customers. Maybe rich people have access to a better class of safe deposit boxes.
No, you encrypt at the tape drives themselves typically. Part of how they work is the also compress at the drive, but if you encrypt first there's no common entropy and the compression techniques work against you.
And picking and choosing is a recipe for disaster when something inevitably slips through and is leaked. Encrypt it all from orbit and let god sort it out.
If you’ve been physically penned aren’t you just rolling dice from then on anyway?
A financial org I worked for back in the day planned on slagging hardware in the event of a coloc breach of any sort including fire response. How else could you be sure?
Well, as a backup, USB drives don't have that great of a lifespan compared to tapes. They're also not nearly as space dense or competitive for $/GB once you've amortized the drives.
And for an org, there's still attacks in both cases. Do your backups get requested through an automated system? Well you've just made a very slow tape library with people rather than robotics. Are the tapes/USB sticks encrypted and the encryption keys stored in an online system? Then the hackers have a mechanism to deny you access to the contents of the tapes, which is probably their goal anyway.
Yes a USB drive with a text file is pretty safe from such attacks, but it also doesn't scale to the backup concerns of even relatively small orgs.
Unless you're practicing continuous restoration to ensure that your backups are actually backing up what you need, the odds that you got everything critical are not good.
The idea that throwing a USB in a drawer does better is absolutely ridiculous.
I think this illustrates where folks can go wrong.
Many folks and business TODAY are backing up to NAS / RAID arrays etc. Many are easily corruptible by ransomware. We see this REPEATEDLY with many infections of large orgs.
The basic story here is that something not editable has great value even if not perfectly current. If you do a diff every 4 hours you have covered a LOT of ground already.
I've sat through continuous DR architecturing discussions - the massive cost / complexity / scale can just be out of control for many users.
In many use cases Active Directory for example behind by 4 hours is not game ending. With notice to users that recent edits may not be captured you may be able to get back up and running pretty quickly.
I'm not singing the praises of "a USB in a drawer" — the medium doesn't matter, the number of copies doesn't matter. The only criteria I am considering is whether the backups are tested to prove that they actually work as intended.
There are shades of gray, here: continuous restoration is a best practice, but backups which are tested even once, and perhaps incompletely, are better than backups which are never tested at all.
Many continuous restoration / HA systems are surprisingly vulnerable to hacks because the level of management / coordination is pretty high. They work well if datacenter 1 goes down and you want to get to datacenter 2. They work less well (in my view) if someone gets root. A fair number of public victims you read about actually DID have continuous backups / dr setups.
OK, but how about at least testing that that it's actually feasible to restore from that S3 bucket of yours?
Verifying that restoration is possible once is better than not testing restoration at all. As for verifying restoration on an ongoing basis, if the attacker has root I suppose we can argue about the cost/benefit ratio but continuous backup validation doesn't hurt.
Where did I say that I don't test restores? I think you are misunderstanding the S3 object lock feature. With S3 object lock, you write your backup, then can read to verify or do a test recovery, but you cannot modify or delete for the retention period.
In the package I use it's called Veeam Ready-Object with Immutability and there are now a fair number of solutions that meet that target.
This has been a pretty unsatisfying exchange. :( It's a good thing we don't work together, we seemingly cannot get on the same page.
From the start, I've been trying to add the qualifier that regardless of whether you throw your backup into a USB drive or an S3 bucket, if you don't test restoration, your backups are probably incomplete.
"Unless you're practicing continuous restoration to ensure that your backups are actually backing up what you need, the odds that you got everything critical are not good."
"OK, but how about at least testing that that it's actually feasible to restore from that S3 bucket of yours?"
These comments have been both off topic and unhelpful. I gave an example of S3 with object lock (or veam equivalents) that I liked - to be helpful to others exploring this question.
Continuous type services are what this story is about - how things that are disconnected / not continuous may fare better in a hack / ransomware situation. Lecturing me that I need "continuous restoration" and or making random claims that I don't test restoration (huh?) is weird.
If I was to give some feedback -> focus on one thing. If you don't like S3 with object lock, say that. If you want to add some comment about restorations from S3 with object lock being tricky for some reason, say it and explain it. If you have something about "continuous restoration" helping in malware hacks explain why. In many cases this can pass corrupted data over to your backup site if you are doing HA/DR work where continuous stuff is most common.
The "one thing" I have been focusing on is testing backups to prove that they actually work as intended.
The state of industry with regards to backups is wretched. If backups are done at all, they are often not tested — people just copy a bunch of files onto a RAID, or a USB drive, or an S3 bucket, or whatever, but never test restoration because that takes extra effort and may be inconvenient unless the system has been architected from day one to enable seamless restoration of live systems without obvious interruptions in service.
Specifically with regards to ransomware hacks...
The first thing that an attacker who gains root must do is corrupt production data. If there are no backups being created, or if the backups don't actually contain all critical data, then the hack is complete.
If there are backups being performed, then the attacker must also hack the backup system sufficiently that restoration from available backup data is infeasible (or at least more costly than just paying the ransom). This adds an addiional layer of complexity to the attack.
if restoration is being tested on an ongoing basis, then the attacker, in addition to corrupting the production data, must also spoof monitoring so that the the corruption of live and backup data doesn't get caught and interrupted before the corruption finishes. This adds yet another layer of complexity to the attack.
A sufficiently competent and thorough attacker may overcome all these difficulties, but criminals are not always perfectly competent — they don't need to be because there are so many target organizations that are not competent either.
It seems that your specific organization does thoroughly test backups to prove that restoration actually works as intended. Kudos! However, I wasn't evaluating your specific case — that would have been weird and inappropriate as you were just giving a (helpful!) example, not laying out a comprehensive system design and asking for review. My point was to add on the condition that in principle backed up data must be not only be 1) current and 2) not corrupted, but also 3) complete. And the way to prove completeness of backups is to test them, ideally on an ongoing basis.
