>Since the storage media in these IoT devices would not last long if true deletions were being performed constantly, “deleted” data is often simply invalidated and moved to an unused page in the block (in a process called “wear leveling”). These invalidated pages, which still contain the data, remain present until a block fills up with them and a true deletion is initiated.
>Accessing flash memory that has been factory reset in this way does require some amount of technical skill and specialized equipment, but nothing that is a real barrier to the average enthusiast working at home.
By that logic my iPhone 11, my Android phone, my Mac or my Windows PC doesn't wipe personal content either after factory reset
I believe iOS handles this by ensuring all flash-written data is encrypted with an on-device key.
A 'wipe' then definitively erases that key - making further overwrites of the freed data unnecessary.
If the crypto is strong, and the key never exfiltrated, then this may be even better than an explicit overwrite, which on some media (like magnetic drives) may still leave trace hints of the old data that advanced techniques could recover.
> If the crypto is strong, and the key never exfiltrated, then this may be even better than an explicit overwrite, which on some media (like magnetic drives) may still leave trace hints of the old data that advanced techniques could recover.
The idea that magnetic drives left readable traces behind after an overwrite is a myth (especially of modern drives; if I recall the article I'm thinking of correctly the myth may have had a _slim_ chance of being true with old drives [like 100MB days]).
The idea was that, since the drive head doesn’t pass over precisely the same area on the track every time it writes to the disk, the track has to be wide enough to account for this lack of precision. Thus, a single overwrite pass could leave a portion of the track’s edge unchanged and readable with an electron microscope or something.
I always figured that a write wouldn't change the magnetic field 100%. Say it's 90% effective at changing the magnetic field. There's now four options:
I agree, there is only one reasonable method to consider a drive "wiped": encrypt all data written to it with a key; wipe == destroy the key. Drives are basically little computers now, they have full-featured CPUs, OSes, memory, write caches and persistent buffers, they compress data before writing it, lie to you about where the data is (ok fine, it's a legitimate abstraction), and who knows what else. My point is that drives have become way too complex for wiping by "overwriting" "every" "block" to ever be an option again (and yes all three of those words deserve scare-quotes). The answer to this problem has become: always encrypt, lose the key to wipe. Period.
From your comment I'm not sure if you're aware but there are two established standards that do just that:
ATA Secure Erase is an optional security feature implemented by many flash-based disk devices specifically because write degradation and overprovisioning preclude them from being wiped via overwrite as you would magnetic media.
More SSD disk controllers than you may realize have an AES encryption engine and persistent storage for the key. The controller is encrypting/decrypting on the fly transparent to the user/system. A secure erase simply rolls that key (and flags all the cells as empty).
As noted in the sibling comment, there must be trust in the device itself that the old key is unrecoverable (or the data recoverable if the prior key had been compromised).
hdparm on any linux liveusb (e.g. Clonezilla) and an eSATA dock and you can nuke a few disks a minute with two commands:
hdparm --user-master u --security-set-pass p /dev/sdX
hdparm --user-master u --security-erase-enhanced p /dev/sdX
hdparm has other arguments that can tell you if this is supported and how long the drive says it'll take to do it.
The other standard is TCG's OPAL. An AES media encryption key (MEK) is generated to encrypt/decrypt all data on the fly. The MEK is encrypted with a user-defined key encrypting key (KEK) and that encrypted MEK is stored on the drive. At boot the user (or some external process) needs to supply the drive with the KEK to decrypt the MEK and begin transparently decrypting/encrypting data on the fly. Splitting the disk into multiple "locking ranges" with independent MEKs and KEKs is also described in the spec enabling secure wipe of just a portion of the disk. Lose the KEK or destroy the MEK and the data is unrecoverable.
I'd say this option still requires trust in the disk itself but since TCG dictates not just the interface but also the implementation and has a corresponding certification process (as well as reliance on an off-device key), I reckon TCG OPAL devices should be more trusted than those that simply claim to implement the ATA Secure Erase command, but I haven't had an opportunity to test any.
It's interesting that even the drives themselves use this strategy to "erase" data; it doesn't actually overwrite the cells, it just always encrypts all data that goes to cells and rotates the key when it's asked to erase. This strategy is clearly superior, it doesn't cost any wear on the cells to perform, and it's instant.
The multiple locking ranges is pretty neat, I'd never heard of that.
How can one trust it though? Presumably the key has to make its way into the drive to actually decrypt the data. How can one be sure the key was actually deleted and not archived somewhere in the controller’s storage?
Ideally, data is encrypted at the CPU (or coprocessor or TPU chip) and sent to the drive encrypted. The drive never gets the key, its only job is to store encrypted blocks correctly. In the TPU case, the TPU chip controls decryption and the key never leaves the specialized, tamper-resistant chip; wiping means sending a command to the TPU to rotate keys, thus instantly destroying any access to the data.
A coworker once insisted on fondling discarded drives with extremely strong magnets, then taking a power sander to the magnetized part. I wasn't sure if this was overkill or not. Is there any wisdom in this?
When I worked for the UC Observatories we physically destroyed most retired drives, just in case of sensitive information. Some personal machines' drives were excepted but there was a good deal of hammering.
You don't have access to "the whole disk". What the drive presents to the OS as an array of blocks is just an abstraction now. There are at least 3 commonly known ways that "filling" the disk with random bytes could fail to destroy the original data:
* If any part of a block is found to be corrupted for any reason, the block is transparently set aside and never used again, but its contents may still be present to read for someone that is willing to bypass the drive's firmware.
* To prevent uneven wear, ssd capacity is over provisioned which means it has more cells than what it tells the OS. In general, when you "overwrite" a block on your drive, that data likely does not actually overwrite the cell where the data used to be. The drive picks some other unused cell, writes the data there, and does some bookkeeping so it knows that this new cell is what the OS means when it asks for that block again. When the OS "fills the disk" it may not end up overwriting some of those cells due to the firmware's arbitrary wear leveling algorithm.
* Many drives have a local write buffer that uses different, faster persistent storage to temporarily save writes to the drive as it takes its time to write out cells. This is to prevent data loss in the case of a power outage. In case the drive hadn't fully written all the blocks to the main storage when it boots back up again, it finds the data in the write buffer and finishes the job. The write buffer has both of the previous issues except its even more arbitrary and depends on load profile which your random filling routine is not guaranteed to hit.
There is no such thing as securely overwriting blocks anymore, because "blocks" are a fiction presented to your OS by the drive's firmware. There is only one way to be sure the data on the drive is filled with random data: make it random to begin with, and ensure that it stays random by destroying the decryption key.
>* If any part of a block is found to be corrupted for any reason, the block is transparently set aside and never used again, but its contents may still be present to read for someone that is willing to bypass the drive's firmware.
Yeah, I honestly have not cared about how hard drives work in about 10 years since I built my last array. I have better things to do with my life than worry about this stuff. I just want to connect a drive, reformat/partition if necessary, and then go back to work. A single SSD now read/writes faster than a 16 drive array (got close in RAID0, but that's just dumb). Now, if SSDs could just get a decent volume size.
They've got what I consider decent size, it's just that the price can be a bit steep compared to an HDD, at least at quantities of 1. $700 will get you an 8TB SSD.
8TB at blazing speeds to boot in a single device. To get decent speeds from HDD would require at least 8 drives striped together. So not only do you need 8+ HDD drives, you also need the enclosure. That would easily double the $700 for the single SSD. Luckily, with USBC/TB4, these enclosures come with the controllers built-in instead of requiring PCIe board
There could be bits of plain text data left over in over-provisioned storage set up in case a block goes bad (e.g. a flash cell that can no longer erase).
Would that extra cell be used unless there is a need for it? If there is a need for it, then I assume it will be a need for it when you fill the drive too.
But yeah, it depends on the manufacturer of the drive, so you don't want to take the chance, Full encryption is unbeatable.
Flash devices maximize the longevity of the cells by wear leveling. The on-board controller maintains a map between the physical cell layout and the logical layout that's changing with every single write operation.
Extra cells are used both reactively and proactively.
Reactively: Error when erasing or write/verifying a cell? Copy contents to spare cell, flag old cell as bad, update the map, maybe complain if running low on spare cells but otherwise call it a day.
Proactively: Cells are only rated for so many erase operations which directly informs the longevity (total bytes written) of a drive. MLC cells can be written to 3,000 times and SLC 20,000 but the latter is much more expensive. For use cases where greater longevity is required (without spending SLC money), why not put a disk together that has twice as many cells as it needs and wear level across all of them? If you `dd if=/dev/zero of=/dev/sdX bs=64k status=progress` your 32GB drive and it'll complete after 32GB but you have no way of knowing how many more cells that drive has that you didn't touch.
(Also as one of the parents pointed out, disk controllers are getting smart. You don't know if it was transparently compressing data too causing you to touch even fewer cells. ATA Secure Erase is the way to go.)
I’ve always wondered this, instead of writing many passes of gibberish data that takes many hours, could I simply format the drive in seconds, encrypt the whole blank drive with a long generated password, then format and encrypt a second time in seconds and be done?
Or if I’m feeling spicy, do all this then hit it with a hammer until it’s shattered in a million pieces like my dreams of working in tech.
If the first encrption would encrypt the entire drive, then you’ve effectively done a 1-pass erase. It might work, but I would still prefer specialized software — perhaps the encryption method of choice would still leave some data unencrypted or untouched. The hammer method also sounds fine to me.
I’ve read opsec material that no matter how many passes you do it can still be read. Hard to know for sure unless you know the stack in and out like the back of your hand.
1) Modern research suggests that recovering data which has been overwritten once on magnetic media is probably not feasible due to the high density of modern magnetic storage devices and the use of dynamically aligning heads (e.g. that follow track geometry as they read) instead of absolutely positioned heads (stepper motors) that allow for more error in alignment.
2) Nonetheless, there is an appreciable risk of data remaining on the device due to non-volatile caches, remapped disk areas, and other factors that are not always well understood or disclosed by the manufacturer. Manufacturers have also been found to be unreliable in their implementation of ATA security features (e.g. embedded secure erase). As a result, with few exceptions it is U. S. government policy to permanently destroy all storage devices rather than trusting any kind of secure erasure. The typical NSA-approved method is either to degauss (surprisingly tricky to do right) and then crush, or reduce to 2mm particles with a device resembling a large, terrifying blender.
> As a result, with few exceptions it is U. S. government policy to permanently destroy
And as a result, (probably) Iron Mountain makes bucketloads because we don't just encrypt the drives (let's face it, that would be perfectly acceptable for 99% of US government computers - maybe Top Secret stuff deserves proper destruction).
Is that fundamentally different from the question of whether overwriting data can make it unreadable? If yes then you can eventually make the "last data" unrecoverable by overwriting it. And if no then presumably you can recover anything previously written to the disk, ergo infinite capacity.
Presumably the original comment meant that you can't guarantee the original data is unreadable with any finite number of overwrites, not that you can always read the original data after overwriting. I'm not saying that's true, but it's a logical interpretation that doesn't imply infinite storage.
Even if you could find the original crypto header, You still need to know the password to unlock it. On iphones, the secure enclave will hold the decrypted key while its running but you need to enter the password after a reboot.
Also if someone had the technical capability to do this. I would be infinitely more concerned about them having an exploit which allows them to hack my next phone and grab all the data while its running.
A lot of that opsec material is very dated back from when density was lower and you had a spinning disk. Do you have any that’s evidence-based that such techniques are needed for flash storage? If anything the opsec concerns are different. There’s very little that would force the controller to actually perform erase of specific blocks (eg the flash could blacklist blocks due to errors). To my knowledge that’s the primary concern, not that the data is still somehow recoverable once erased (which used to be true but I’m not sure holds for modern storage).
That’s probably true but the GP is likely drawing the distinction that iOS tries to do an “unrecoverable” delete as best it can whereas presumably the Echo is not doing this (encrypting the data and then deleting the key as part of the reset).
Concerning Windows, the help page about the factory reset states "You'll be asked to choose whether you want to erase data quickly or thoroughly. If you choose to erase data quickly, some data might be recoverable using special software. If you choose to erase data thoroughly, this will take longer but it makes recovering data less likely." [1]
Of course deleting data on an SSD is tricky due to wear-leveling, anything short of telling the drive to erase the entire disk with the secure erase command isn't guaranteed to work.
This is why FDE is mostly the default on all of the devices you mention. The secret keys are stored in the TPM or equivalent. Most SSDs for Mac/PC also have an OEM software option to trigger a secure wipe using special software from the vendor.
The problem here is that the Echo is closed so those options aren’t available to you. However, this is also a problem on other custom devices like printers.
>By that logic my iPhone 11, my Android phone, my Mac or my Windows PC doesn't wipe personal content either after factory reset
Your iPhone 11 zeroizes the cryptographic key used for the user data partition. It doesn't have to wipe it per se, but rather it renders it eternally inaccessible; then it creates a new key and sets up a new filesystem right over top of the old one (which now just has random data in the unallocated space).
Do iOS devices not leverage full disk encryption? In which case I’d assume simply wiping the encryption keys/fde headers would be sufficient? (Which it almost certainly does.)
> The “Erase all content and settings” option in Settings obliterates all the keys in Effaceable Storage, rendering all user data on the device cryptographically inaccessible.
Semantics debate around whether leaving data practically inaccessible is the same as removing it, but considering where we are, I'd say it isn't.
Granted, this is from 2018, but I doubt the defaults and concept have changed.
Securely deleting the keys is safer than simply practically inaccessible. That suggests their is some impractical way to access the data, care to suggest what that might be?
Overwriting flash with 0s. To be fair I think the iPhone should grant you that option (MacOs does). But to suggesting that leaving an encrypted block of data on the device is on the same level as leaving unencrypted blocks of data lying around is foolish.
Overwriting flash data isn’t going to give you access to the data. It might in theory be more secure, but I would favor encryption at rest rather than simply overwriting plain text.
It is possible in theory to recover some information from overwriting flash data with zeros, it’s also possible in theory for well designed modern encryption to be broken but neither of those are guaranteed at this point.
Essentially every encryption scheme in history has been broken. It's a bit silly to think that "well designed modern" makes it safe. The others were all well designed and modern at one time, too. The only scheme that's truly unbreakable is a one-time pad and that's... well... not really practically useable.
Personally, that's not a risk that I would care about (especially in the context of an Echo Dot or even an iPhone), but deleting the keys will not be better than inaccessible forever.
Even if you didn’t have the public docs explaining this, you could make an educated guess because resetting the iOS device takes the same amount of time regardless of how much data you put on it and that time is on the order of 10s of seconds. iPhones don’t have erase speeds of 12gigabit/s - I don’t think there’s any storage technology that could do that. Also remember that Apple takes seriously the concern about the data leaking because the OS decided* to blacklist a bad block. Once a block has gone bad you probably shouldn’t rely on attempts to erase it working out.
* if I recall correctly iOS devices don’t use a traditional flash controller and Apple does that mapping themselves (it’s also largely why Apple devices have better flash performance than competitors)
> you could make an educated guess because resetting the iOS device takes the same amount of time regardless of how much data you put on it
This is incorrect. Secure deletion should cover the entire disk no matter how much data was used before, in order to deny an attacker any information or to guide their recovery efforts. I would absolutely expect a secure deletion process to take the exact same amount of time regardless of how much data is used.
A better tell is the difference between the same model of phone with different storage sizes. A 512GB iPhone should take 2x as long to securely wipe compared to a 256GB one, assuming they’re actually overwriting the encrypted data with random noise.
> iPhones don’t have erase speeds of 12gigabit/s
It’s worth pointing out that a true secure delete is actually a write not a delete. But it’s true that your typical iPhone could not possibly overwrite its entire disk in 10s or so, although last time I did a reset it took much longer than that.
> This is incorrect. Secure deletion should cover the entire disk no matter how much data was used before, in order to deny an attacker any information or to guide their recovery efforts. I would absolutely expect a secure deletion process to take the exact same amount of time regardless of how much data is used.
That's a fair point. Still, mobile devices typically employ FDE to avoid the need for scribbling over the entire flash (also improves the lifetime of the flash in that way).
> It’s worth pointing out that a true secure delete is actually a write not a delete
Flash storage doesn't work that way unless you're referring to something else. To write a block you have to erase it first (makes the block all 1s or all 0s depending on the chip). An update of a block is technically then a read/erase/modify/write (although flash translation layers will elide the first 3 steps by writing to a free block & marking the old one as GC'able). Writing random patterns into flash to my knowledge generally isn't needed anyway as it was with magnetic storage. The only purpose of writing in a secure erase scenario is to try to overwhelm the flash translation layer to force every block to be touched.
> although last time I did a reset it took much longer than that.
I haven't profiled it, but any slowness erasing an iOS device is probably because the flash storage backing the secure element is optimized for reads rather than writes. Additionally, it may be erasing various metadata/caches in the OS storage that aren't covered by the encryption key.
If you could reliably read data after 6 passes then HDDs would be 6 times larger with the same hardware.
There is the further belief that state sponsored actors would be able to read them, but this relies that state sponsored actors 1) have hardware capable and 2) would only go to those lengths to get the data.
> But at least Macs have a built-in 7-pass erase option to comply with DOD standards
I recently learned that the "zero out data" options are not available in Disk Utility for SSDs. This probably fine if you were using full disk encryption, especially on modern T2-chip Macs that always encrypt the built-in storage even if you haven't enabled FileVault, but it was a little concerning when I was getting rid of an old failing Mac that had an aftermarket SSD installed without encryption enabled.
Yes it's really annoying because they also aren't available in the UI for fusion drives. You have to go into the command line tools in recovery mode, break the fusion drive volume down into its constituent pieces, and wipe the HDD separately, then rebuilt the fusion drive manually.
Interesting note: modern high-density NAND flash ("TLC", "QLC") is so sensitive to the data patterns being written (writing zero bits causes more wear than writing 1 bits - the latter being the default non-written state of the medium) that controllers need to use a scrambling algorithm to statistically balance the number of 0 and 1 bits that are written. Earlier schemes were based on LFSRs or similar, but then I guess someone noticed that an encryption algorithm would also do a similar job, with the advantage of it being much harder to create pathological cases (look up "CD-ROM weak sector protection" for an interesting rabbithole related to that) and "self-encrypting drive" also be a good marketing point.
On the other hand, it's a nightmare for data recovery, especially the "I didn't ever mean to encrypt this drive, what do you mean you can't recover it?" situation.
The clickbait-y part is that it is possible to retrieve a token from the previous user. Pretty normal as you say.
The non-clickbait is that the previous token is still valid as far as Amazon is concerned. They absolutely should have a process for invalidating tokens.
The flash on your iPhone is nicer than cheaper embedded devices / microcontrollers. I have no idea what Echos use, but I’ve seen modern ARM micros have a 50k flash life expectancy or less.
It’s not impossible to have a flash memory without a user data eeprom that you could “burn out” (in reality make less reliable).
I have no idea if it’s true in this case, but the straight comparison / jump to iPhone 11 might also be speculative.
There is a command you can send to the drive that will erase all data including from those extra bookkeeping areas. "ATA Secure Erase". The drive has to support it of course.
This explanation infuriates me so much.
They use technical terms and true technologies to lie about their security issues and measures. Flash memory should easily withstand north of 10k write/erase cycles, base on type.
Even if there was somehow some very low limit, users’ request to delete personal & sensitive data should be honored and any cost.
But then again, we’re talking about people paying actual money to have these devices in their homes.
High-density flash memory with more than 1 bit per cell, like in SSD's, withstands usually just a few hundred, some times only one hundred, write-erase cycles.
You can easily compute this value by dividing the guaranteed endurance by the SSD capacity. For example, an 1 TB Samsung 980 guarantees only 600 TBW / 1 TB = 600 write-erase cycles.
This computation works because the wear-leveling algorithm used by the SSD controller distributes the writes uniformly over the entire SSD, ensuring approximately the same number of write-erase cycles for every cell.
User/config partitions can be encrypted and factory reset can delete the user/config encryption key. Minimizes writes to physical blocks, user data becomes unrecoverable. This is standard practice for repurposing of multi-TB hard drives.
Ideally, the OS should be an immutable image and the user partition completely seperate. Than a factory reset simply has to nuke the user partition and place a fresh template over the top.
I can understand how something not designed could have difficulty factory resetting when data can be scattered all over the place.
I don't know about what is it like now (with all this A/B stuff), but /system/ on Android is supposed to be read only. As was ROFS on Symbian. That said, root could modify /system/ anyway.
It's still read-only on Android these days (there are multiple partitions for A/B stuff). Root can modify it, true, but I don't think anyone is storing any secrets on there.
With file-based encryption on the data partitions, Android hard resets should make all files inaccessible even if they're left on the drive.
A device that is a black box does a black box thing. The dynamics in the current system that have, and continue to, reward the privatization of knowledge, will end soon.
Come join us in growing the first Open Value Networks using DWeb tools like Valueflows/hREA, Holochain, SSB, DAT, ActivityPub and more.
Consumer electronics are becoming more durable. Where once I might have replaced a cell phone every year and my home router every two, I now have had the same phone for four years and my router for six. Second hand sellers and repair shops need the factory reset feature, but we all benefited when our firmware allowed the devices to become safely transferable and repairable.
From the cited paper, most of the acquired devices were not even reset:
> Not reset devices: A surprising number of devices (61% ) were not
reset by the previous owners. Due to the setup of our experiment,
we had no possibility of asking the previous owners any questions.
I'm doubtful this exists, but I'd like it if it were possible to perform a factory reset or account transfer completely online, and other features around improving the security in the resale market.
For those curious like me how the wifi password was actually leaked:
> WPA-supplicant is responsible for connecting to configured
access points after provisioning. We found that it creates its con-
figuration files on the user data partition in the folder “misc/wifi/”.
Here the file “wpa_supplicant.conf ” contains the Wi-Fi credentials,
such as the SSID and PSK.
so your problem is not the potential of the damm thing listening on you 24/7? it’s the wipe after factory reset?
i guess I could see some people mind one but not the other, but I guess people that strongly care about their privacy would not buy an echo to start with.
Then perhaps we should have an activation system developed by a company that you trust, which interfaces with the underlying Amazon Echo/Siri/Google assistant.
I have one I got ad a gift years ago that I'd like to get rid of. Is there a good way to get rid of this thing without tossing it or giving my info away?
I received one as a gift too. Used it for about a week, got tired of it and creeped. So I pried it open, used a flathead screwdriver + hammer to chisel/smash any chips that looked interesting, and recycled it with some other e-waste.
I felt bad destroying a “perfectly good” device, but my reasoning was that the world is better off with one less of these spyware devices, and I resolved to never buy a smart assistant device again. I wouldn’t have destroyed it had I been more trusting of Amazon, but now I’m glad I did.
>Accessing flash memory that has been factory reset in this way does require some amount of technical skill and specialized equipment, but nothing that is a real barrier to the average enthusiast working at home.
By that logic my iPhone 11, my Android phone, my Mac or my Windows PC doesn't wipe personal content either after factory reset
Feels like a bit clickbait-ish
But at least Macs have a built-in 7-pass erase option to comply with DOD standards https://techpatio.com/wp-content/uploads/2012/09/disk-utilit...