Great article discussing the benefits of optical discs and the author's own experience of using them!
I burned hundreds of discs a decade ago and read them all back perfectly, to my surprise. I strongly agree with the write-once behavior for photos and downloads. HDDs worry me in the long term because even if the platter holding the data is fine, the R/W head, mechanics, and electronics could fail, not to mention that a head crash could wipe out good data.
I'm extremely disappointed that the capacity of optical disc formats are essentially frozen, topping out at 100 or 128 GB for BD XL. They're definitely not keeping up with the continual increase in storage density and decreasing cost of hard disks and flash memory. Why are new formats not forthcoming? Where's my 300 GB Archival Disc? Where's my 1 TB holographic disc? Where's Microsoft's glass optical disc that's stable for millions of years?
I want to keep on burning optical discs and putting them in cold storage, but this gradually obsolete technology is not making a convincing case for me today.
Higher capacity optical storage is under development (see https://foliophotonics.com/ for example). My guess is that it's just hard to beat tape or justify upgrading/modifying existing optical disk readers.
The selling point now isn't that is a temporary storage solution that happens to last long like CDs and DVDs. I'm sure you have to make a pretty good case to get a customer to switch their high reliability data storage to a novel format.
Another issue I've had is bad blocks forming on my SSD drive. I thought I lost a bunch of photographs (nothing important thankfully, just amateur shots when I first got my hands on a camera), but searching around I discovered CHKDSK. 4 hours later and everything was back to normal.
My OS (Win10) is on the same SSD. When I made the decision to get a 1 TB SSD for the OS and media storage, deep down a voice whispered [dont do that]. That decision has come to haunt me 1 year (!) later. I used to use 128 or 256 GB SSDs for the OS and keep all my media on another HDD drive (which failed after 10 years).
How many people are buying these 128GB discs? Probably not a whole lot and that is the reason why nobody’s going to make the investment to create bigger discs.
If you want huge local offline storage, the solution is tapes.
But it is already known that magnetic tape is much less reliable in long term storage. The author is looking for long-term reliability, not just huge local offline storage.
It is not known what the durability is for an optical disc format that does not exist. There is no optical disc format that stores the required amount of data.
LTO tapes do store the required amount of data and if you want them to last extremely long you should make new copies every 10 years or so. Just like you should for optical media. Sure, some discs might last 50 years. But some others won’t be readable after 15.
oooh I can answer this one. About 15 years ago I saw a talk from a team at, oxford I think?, that had invented a way to store many terabytes on an ordinary blue ray by adding rotation to the scratches, which can then be read because scratches at different angles polarized the returning light differently.
Anyway, they had a working prototype and everything and the big companies like Panasonic all said "No sorry, everyone is shutting down their optical drive lines. We have determined that the future is internet storage and solid state. No one wants disk no matter how much storage they can hold." And that was that.
No, the memory cells lose charge over time. Even one year can cause data loss. Sorry, I don't have the link, but there was someone who tested how latency increased on drives that spent months on the shelf. The explanation was error correction working harder.
You don't need a link. I can corroborate with my own experiences. Flash memory slowly forgets information. One guy told me that he can't unplug is big SSD array because he's afraid to lose a few bits. He might as well call it RAM.
Archival Disc exists, they just don't support bare discs or readers. You have to buy a full Sony ODA Gen 2 system and...
reads the linked comment
notices my own reply
...oh. Nevermind.
So, I'm going to go out on a limb here and say that all of the new formats are not "forthcoming" because of basically two problems that happened at roughly the same time:
- Newer optical media formats became too expensive to implement in consumer devices
- Optical media is inherently user-hostile, almost as much as tape
Ok, first claim first. Back when Blu-Ray first came out, drives were hideously expensive; to the point where the PS3 was basically eating SONY alive because the drive itself cost as much to make as the whole console sold for. Apple outright said they'd never ship a Blu-Ray drive in a Mac because it was too expensive and the patent licensing was a nightmare. Even today Blu-Ray drives are still more than twice as expensive as DVD drives.
But, of course, this isn't the only hurdle, but it did dramatically slow Blu-Ray adoption. The spat between HD-DVD and Blu-Ray almost certainly didn't help either. All of that ultimately meant that people weren't buying Blu-Ray players and putting them in their living rooms; which meant that it didn't make sense to have one in your PC either.
The dirty secret of optical media is that it's remarkably user-hostile as a consumer storage medium. Discs want to be written from beginning to end, and preferably all-at-once with a single table of contents. This makes it very difficult to actually provide a drag-and-drop utility for getting files onto a disc. You can do this, but in most cases this is a hack involving multi-session filesystems that relies on readers knowing how to stitch together the sessions. This is why Windows had that entirely unhelpful "What do you want to do with this disc" dialog for optical media. If you try to use a Blu-Ray disc like a flash drive you will be extremely disappointed in it.
In the days of CDs and DVDs this didn't matter. The consumer benefit was that the discs you were making could go straight into a CD player or DVD player that you already had; the fact that you had to jump through some hoops to do it didn't matter. And if you bought CDs or DVDs you can play them on your computer, too. But far fewer people owned Blu-Ray players, and if they did the player probably also took USB drives. So there would be no reason to ever burn a disc to get a movie onto a player, and getting data onto them was enough of a pain in people's arses that they'd just use USB drives instead.
So with all that taken together, nobody is bothering to scale optical discs for consumers because there really isn't a market for them as a data storage medium anymore. The things that discs still do well - i.e. selling you prerecorded movies - also just so happens to still be fine with topping out at 100GB/disc, even for 4K video. Some games are starting to push beyond what a BD-XL can hold, but those games also are "live service" games that update way too frequently for "getting all the assets on the disc" to make any sense. All the supra-Blu-Ray optical data storage formats are all targeted towards the archival market because it's something that optical just accidentally does very well.
> What I didn’t realise at the time, was that all these CDs and DVDs would become a grand experiment of reliability and longevity. When I read data from these backup disks in 2021, I had a 100% success rate!
My own experience is pretty good, but not 100%. I burned a lot of CDs from 2001 to 2006 or so. Most were discarded at some point, but in late 2019 I ripped all that I had (39 CDs) with (GNU) ddrescue. The vast majority (77%) had some small amount of read errors. I was able to recover 99.88% on the worst disk (a brand I'm not familiar with: "D"). That disk was about 15 years old. That's pretty good, but not 100%.
This experience underlined to me the importance of having parity files in addition to the disks themselves. When I do a Blu-ray backup now (roughly once a year for the most important files), I also burn a DVD with par2 files.
Back in the 00s I didn't verify checksums, but as a test I checked zip, gz, and png files on the oldest disk I had (Jan. 2001, 100% recovered). Those files have internal checksums to detect corruption. All those files passed. I should probably perform the same check on the other disks.
Edit: I ran my script to check internal checksums on my disk image for my worst disk and 19 out of 635 files tested were corrupt, all png files. That is worse than the recovery percentage that ddrescue reports (97.01% valid vs. 99.88% recovered). That's roughly 0.2% lost per year.
Would concur. I went thru all my old optical discs in ~2020 and moved all backups to 100GB Bluray. Went from dozens of discs to about three. Then I destroyed most of the old ones. (Didn't bother with some... does anyone need a copy of Netware 3.x or 4.x anymore? Even though it was so cool at the time.)
All my CDs/DVDs burned from about '97 onwards were in perfect condition. I think one deteriorated, stuff I didn't care about given to me. I never bought super-cheap ones, maybe that helped.
I did save a few "mix tape" discs from the era for nostalgia purposes. Imagine 15 of the best songs from different artists on ONE disc! It was a big kick to make your own CDs at the time... would blow people's minds, when you showed them. :-D
My CDs burned in 1998 still read totally fine. TFA and your comment is, I think, what shall eventually make me move to BluRay. Any model/brand of writer/discs to recommend? (Linux here FWIW)
EDIT: btw ofc all my pressed audio CDs all work just fine. My car for whatever reason, although semi-recent, still has a CD reader and I was listening to some Ennio Morricone CD in the car 10 minutes ago!
For newer stuff, I have an ancient 5-7? years-old external LG BLuray/MDisc drive, and use Verbatim media. Can't speak to their longevity so definitively. They are working fine after three years. I also have a few TB external backup drive for additional-copy and rapid-access to the same data. I take an optical disc to a relative's house once a year.
Multiple copies, multiple formats, multiple locations, is the key to back up.
At the turn of the century half of my personal CD collection was mix CDs from friends. Someone broke into my 25 year old Toyota and took the CD wallet, spare change and the car battery.
That was 9 years ago but I still wish I had a backup of those mix CDs.
About a month ago I went trough all my CD/DVDs. Factory disks were all good. But burned disks, especially the later cheaper ones mostly were either in bad condition, or not even recognized as media.
I'm a bit skeptical of these conclusions without more detail on the methodology. Does "100% success rate!" mean the author was able to mount the disks and read some files off them? Or did they actually have some method to verify check-sums of every file to identify bitrot?
I guess that was just an instance of the "Survivor Bias". I burnt hundreds of CD-R/DVD-R/DVD-RW disks from a various of vendors(mostly Sony, TDK, Verbatim, Kodak, maybe some other brands which I could not remember clearly, each brand should be at least 50 as I had always bought 50 pack tubes.) around 15 years ago and left them untouched for around 8 years when I needed to retrieve something from those archives. So I decided to check the health of those disks which was quite disappointing. I simply tried if I could read the first 10 disks in each tube and it turned out, for a brand, all 10 were not even able to be recognized. Other brands performed from 3, to 8 in 10 disks could be read. That was just simply trying to read, so the integrity of the contents were not checked at all. The only good thing was that I managed to locate what I was looking for and it was in RAR format which validates the hash when extracting, it was good. So I immediately tried to move those I wanted to keep to HDDs and the loss rate was over 50%, and for those recovered, I didn't have much confidence regarding their integrity neither as no checksums saved. So basically I wasted lots of time and money by choosing optical drives as archive media. In the mean time, all files I left on HDDs were perfectly fine. This might be another survivor bias too.
It matches my experience. Even decades-old CD-ROMs only fail when they are scratched or physically damaged, which you can prevent by keeping them in appropriate cases and always put them back after use.
My own experience is different. I had several brands of CD media that lasted less than 10 years. Many of them you could see the damage (cracks through the shiny layer). Some were dark blue, some a light blue. Almost all of them were bad. I had them stored in a light proof zipper pouch and stored at room temperature in a closet. Maybe something special happened to them over the years but very few worked. This would have been some of the first CD-R and CD-RW media available. I opened them more than 20 years ago so the media would have been 30 to 40 years old today.
Some plastics can react with other plastics you have to be careful when storing long term in plastic bags for this reason. I'm not at all sure if this played a part in your CDs but it's possible.
Again - did you use some method to confirm there was zero bitrot on these discs? Or are you concluding they were still good simply because you were able to get data off them?
One option that hasn't been mentioned is tape archives. It's not for everyone, but tapes (LTO) do have a lot of technology and infrastructure behind them that is built for longevity.
In particular, you can do cost-efficient archives to multiple tapes and store them in safe locations. If (and that's a big if) the base cost of the drive is tolerable.
There was recently a wonderful post and discussion on LTO for "normal" nerds [1].
IMO the biggest problem with tapes is getting data off of them. The LTO standard only has compatibility for 2 generations back (i.e. an LTO-6 drive can read LTO-5 and LTO-4 tapes, but nothing before then). This means that if you have data you want to store for 30 years, you will also need to factor in the difficulty of finding a working tape drive and a computer that can operate it. With optical media, a new Blu-Ray drive can read a CD-R from the 90s so this is less of an issue.
But on the flip side, the LTO spec guarantees that this two-generation backwards compatibility will remain for the forseeable future. As the article mentions, it's possible that optical discs will become unpopular and cease to exist completely.
With tapes, you're in the same boat as a bucket full of institutions to whom archives are VERY important. That's a big plus!
One thing that keeps me from trying tapes is that they are still magnetic. If there's some sort of local electromagnetic event then I'd prefer to have a backup that can handle that. I keep some backup hard drives in metal boxes, which is probably enough (Faraday cage). However, optical disks can almost certainly survive a major electromagnetic event without special precautions.
> local electromagnetic event then I'd prefer to have a backup that can handle that.
First of all, the magnetic gauss needed to flip bits on LTO media is huge. It's not a practical concern unless you're choosing an incredibly inappropriate storage location.
Second, why wouldn't you store your tapes in a case or a vault that is magnetically shielded?
You probably store your discs in cases, instead of insisting they be naturally durable against rough handling and abrasion.
> First of all, the magnetic gauss needed to flip bits on LTO media is huge. It's not a practical concern unless you're choosing an incredibly inappropriate storage location.
(Some of the links from this answer are quite interesting.)
The issue is that I don't know (and I suspect it's difficult to know or even classified) whether an EMP attack (the main scenario I'm worried about, which can be quite powerful) would be sufficient to cause damage. I don't believe there is a consensus on this though I'd be glad to be proved wrong here.
> why wouldn't you store your tapes in a case or a vault that is magnetically shielded?
I would; as I said I put some magnetic hard drives in Faraday cages. However, I'm not confident that the Faraday cage would be sufficient. A lot of the discussion online about constructing Faraday cages to protect electronics is speculation. I haven't seen any clear data showing that during an EMP attack magnetic data is protected inside of a Faraday cage, or protected without one.
The EMP from a nuclear weapon is mostly radio frequency and would be picked up by antennas, power lines and such. (I know an ex-military ham who always unplugs his HF/shortwave radio from the antenna and power when he's not using it because that's what they taught him to do back in the day.)
I don't see that kind of EMP erasing magnetic media.
Interesting. Is a magnetic event more plausible for you than other types of decay? I'd expect physical damage from something like a flooding to be much more likely than some sort of magnetic interference.
But that probably depends on the location you live in!
I don't know how likely some sort of electromagnetic event is. I just figure that if I'm going to diversify my backup strategies then something dissimilar/uncorrelated would probably be a better choice. My data is on magnetic hard drives, SSDs, and optical disks, each of which has different risk profiles.
As wtetzner said, it's unlikely that a single event could cause all of these to fail. An EMP attack might take out my magnetic hard drives and SSDs but leave the optical disks unaffected. I've also had a hard drive fail due to impact damage that likely would not affect a SSD. Flooding is not something I've thought about before, but since my backups are located in different geographical areas, I should be covered there. Etc.
I have many degraded CDs and DVDs from 2000-2010. No drive would read them. Most from reputable brands. Most non-RW. All stored in dark places, indoors.
I have many degraded CDs from 1998-2010. In 2019 almost all of them would read, eventually, even as metal flakes came off the disks. I recovered most data from most. My CD-Rs were a mix of brands, nothing special or expensive. But my CD-drives are old too, from 2006(?) or so. I wonder if like with floppy disks and drives, modern CD drives are less capable than old ones because they're no longer a premium product?
> I wonder if like with floppy disks and drives, modern CD drives are less capable than old ones because they're no longer a premium product?
When I ripped CDs I had from the 00s in 2019, I used an external Blu-ray drive from probably 2018 or so. Seemed to work okay; see [0]. I didn't have access to anything older so I can't compare but I think both new and old drives are probably good.
Thanks. I anti-corrected it to red from blue when I found the video, and the statement felt really wrong when I wrote it, but I thought fuck-it - the video link is too good to miss! Lazy me.
I have a couple CD-Rs (from the early 2000s) that have completely lost their top layer. I suspect the cause is the label sticker that was used. But in any case, it's completely unreadable.
I have 5-1/4" floppies the early 80's that still read fine. I wonder if 3.5" floppies from 1995 would still work. I have a feeling the media was made more robustly in the early 80's because the sensors weren't that precise, so bits were written with way more field than they needed. Maybe in the 90's when so many bits were crammed on to the 3.5" that they weren't written as strong?
Perhaps the same is true of optical media. Are higher density optical disks from later in the era less robust than earlier tech.
Like the way Hobart mixers from the early 1900's are built like tanks and can outlast ones built today to save money.
During the first lockdown I dug out my old Commodore 128 and quite a few discs were still reading fine... But not all of them. I don't remember the ratio but I'd say about 2/3rd were still working.
Not very actively maintained these days but https://en.wikipedia.org/wiki/Dvdisaster stores error correction codes in the remaining space on an optical disk, without affecting its readability by unaware systems.
My biggest concern with Optical Disks is obsolescence and convenience. I bought some DVD media to do some backups but so far I have only backed up a portion of my photos. Right now my files are being stored on my MacBook's internal SSD, Backblaze, Google Drive, and a Time Machine backup on an SD card that stays in the SD slot. I do full backups to external hard drives every so often. I'm too lazy for any backup method that requires an involved process like optical media.
And I also find myself asking if it's even worth it. I don't think my files are that valuable, to be honest. I would be sad to lose them, but it might also be freeing in a minimalist sort of way.
If I am looking at a ~100 years timescale for preservation of documents, there are exactly two choices that I would consider. At the hundreds of years timescale, exactly one.
Those are, in order:
1. Microfilm/microfiche
2. Ink on 100% cotton paper, or ink on vellum
The properties of microfilm have been well studied. It requires no technology other than a magnifying glass to read, has a well over 500 year lifespan in studies, and has exemplars that have survived more than 150 years.
Ink on paper, on the other hand, we have exemplars that are almost 1000 year old, with ink on vellum more than 2000 years old. It requires no tools whatsoever to read, and we have been working and understanding how it is best preserved for hundreds of years.
I've got a printed book from 1575 here. Some pages at the beginning are in a bad shape but otherwise everything is still perfectly readable. I don't know which type of paper/ink it's using though. It's some book about plants: apparently one of the most printed book (after the bible) back in its days.
> How would you "print" your backup on a microfilm?
There are services that can do that for you. You send them document images, they send microfilm back. I've never actually done it, but I've considered it, an I had this random service in my bookmarks: https://overnight-scanning.eu/microfilming-service/
Same here. I just have a mirrored Storage Space for critical data with ReFS and checksums (though Microsoft did not make that one easy to understand so I am unsure if the data is safe that way).
UPD. just searched, and found, that I needed to enable periodic scrubbing in Task Scheduler: Microsoft -> Windows -> Data Integrity Scan section. I already had data integrity streams enabled (another thing you have to do manually after creating the volume).
In my movie collection, I replaced almost every DVD I could with Blu-Ray. Not just because the quality is better - it was, as the author mentioned, the massively improved scratch resistance. When you have kids, it becomes cheaper in the long run.
Also, we’re a quality over quantity family. We don’t want thousands of movies on several subscriptions. Everything we’ve ever liked fits on two four-foot-wide shelves.
"(The spelling disk and disc are used interchangeably except where trademarks preclude one usage, e.g. the Compact Disc logo. The choice of a particular form is frequently historical, as in IBM's usage of the disk form beginning in 1956 with the "IBM 350 disk storage unit").":
Laser disc, compact disc, digital video disc, and blu-ray disc are (from what I understand) officially spelled with the c. I'm in the US and that is the typical way to spell it that I see, but I also see the k is generally used for hard disk and floppy disk.
That said, there are a lot more instances of "disk" than "disc" used in the comments here so far.
> In 1979, the Dutch company Philips, along with Sony, developed and trademarked the compact disc using the "c" spelling. The "c" spelling is now used consistently for optical media such as the compact disc and similar technologies.
Finally, optical disks are more expensive than HDDs - at least in cost per GB. A 4TB NAS branded HDD costs ~AU$160, which is ~4c/GB. My last BluRay purchase was for 3 x 50 spindles of 25GB disks costing AU$330, which works out to be ~9c/GB.
For anybody with deep knowledge of the industry, is the price of discs an accurate reflection of costs or is it inflated by secondary factors such as attempts to make the media more expensive to deter piracy or Sony's desire for high profit margins on data storage? I well remember Sony's shenanigans from the 2000s when they made great hardware and then only accepted MemoryStick instead of SD cards while charging much more for their flash format. I was so excited for BluRay and then prices just never came down enough to be practical.
I thought BluRay was subject to the blank media levy in Australia, but apparently not! [0] Note that not every country got off without though - just a few lines down, the article notes the levy is present on BluRay discs in Finland, among others.
As an aside, I'm of the impression not enough people know about this levy. It feels to me like the kind of thing that fouls up the "nothing to hide" arguments regarding privacy; that is, a law few know about, but one that affects lots of people that wouldn't necessarily agree with being charged for something they may not be doing.
While I'm sure some people still pirate and rip to BluRay, I'd assume they're just as likely used for things such as home movies, format shifting (which is legal in Canada [1]), or just plain backups (which I use for important files like a keepass database or 2FA codes).
While this has drifted off-course slightly, it may at least have given you an answer regarding why BluRay discs didn't fall too far in price - at least if you're based in Finland.
RO media is great, especially when you're concerned about data integrity and may not completely trust a machine writing or verifying the media. Consider you're downloading sensitive data from the internet, or just writing sensitive data on an internet connected machine (maybe it's OS installation media, or important legal or financial docs, or you're a wall street or NSA whistleblower, or it's a Bitcoin key backup), and you're worried your machine may be compromised on account of it being connected to the internet; you may not want to stake the quality of the rest of your life on your computer not lying to you about file integrity.
What do you do?
Write it to RO media and take it to different computers to verify the checksum/signature. You don't need to worry about other machines, which are also potentially compromised, corrupting the data. You also don't have to worry about something like BadUSB, where the USB device's microprocessor firmware infects the host machine (or vice versa) before the data payload is even read. This means you can put the disk in your secure airgapped machine at home, even after it's been in the machine at the local library. You create a concensus of file integrity after reading and verifying the data across a range of machines.
Obviously there are other attack vectors, but with the cost of these sorts of attacks (badUSB, or simply getting rooted) going down, the accessibility going up, and the potential rewards becoming more valuable and widespread, it's an attack vector to consider for important data.
No. DVD M-disc are no longer manufactured, only blu-ray M-Disc is available. Which is a pity, because the DVD version was rated for longer longetivity.
When I was in grad school back in the '80s, 9 track magtapes were the standard for lots of things: backups and long term storage being some of them. (I still have one or two tapes in a box somewhere.)
About 10-15 years ago, it dawned on me that I hadn't seen a working tape drive in quite a while. Any tapes, even if the media were intact (highly doubtful, and the point of the parent article) would not be readable.
Much the same comment for boxes of punch cards. (My late colleague used to tear off pieces of the cards and use them for "filters" in his roll-your-own cigarettes.)
I guess my point is, even if the media are reliable (holes in cardboard? pretty foolproof from an ink-on-vellum viewpoint) the entire surrounding ecosystem needs to survive for archival purposes.
I wish I did more rigorous work on this but I managed a large Rewritable DVD backup for a satellite instrument with over 3000 DVDs stored in a filing cabinet for almost ten years. When I joined about three years in I added checksums to the backups and infrequently sampled and reburned older discs and found zero corruption after almost a decade stored in those conditions. Given there was a mix of R and RW burned discs I was actually pretty surprised there was no corruption because I had it ingrained in my mind these things shouldn’t last so long.
After dealing with several valuable SEGA Saturn games that succumbed to bit rot, and are therefore worth practically nothing - (nor are they playable, obviously) - I backed up whatever optical disks I had left and have never used optical media since. I certainly never will again.
I just can’t trust it anymore as an investment.
I’ve got vinyl from the 50’s/60’s that still play great but after losing those SEGA games I just see those little plastic disks as ticking time bombs.
9ct/GB for plain BD-R is a rip-off. They cost 0.08 USD a piece (if you reach MOQ on Alibaba), plus about the same as patent license fees. So 16ct/25GB, not 9ct/GB.
Taking storage overhead (cases of some sort; spindles aren't well-suited to store BD long-tem) into account, the Chinese 20ct/50GB (plus patent; iirc sub-linear scaling with capacity?) option of a DL BD-R may easily be cheaper over all.
My costing put it as positive compared to disk even without massive discounts, but physical volume is a problem, and cheaply constructed disks are volatile.
Aside from the author dismissing tape because "drives are expensive" and then concluding that the primary disadvantage to optical is "the discs are expensive" (wow, big surprise...) and then he doesn't bother to actually calculate the costs of either...
...and not describing how they tested for errors...
...and not having particularly rigorous testing methodology (the DVD was apparently left for three months before he got around to checking it)...
...no organization looks at a requirement like "archive these records for 50 years" and then commands its IT department to only store the data on media that will be expected to survive 50 years.
What you do is store the data in such a way that your chance of losing data per year is within acceptable margins, on the most suitable storage method at the time, and migrated periodically when that system's chance of data loss falls below acceptable margins, appears to be getting too impractical to operate/maintain, or there are improvements in reliability or practicality.
That can include factors like cost, availability of knowledgeable labor (ask the IRS how finding COBOL programmers is going), parts/service availability/cost, and so on.
Other comments:
> HDDs, especially NAS disks, have an “always on” assumption
Even in the early 2000's, commercial NASes were offering systems that could power down portions of the array that were unused. There was significant interest in minimizing opex, and that's mostly power. Nowadays unraid and other solutions allow for the same.
> And if you ever wanted to take disks offline and store them on a shelf, you don’t really know how long they’ll survive - unless you plug them in every now and then.
"you don't know if that HDD is gonna work if it's sitting on a shelf" is irrelevant because your backups shouldn't be sitting on a shelf for any significant period of time, regardless of the media/mechanism. See discussion above: you should be rotating your media off-site periodically, bringing it back for tests. The biggest problem with HDDs is that SATA connectors are not rated for frequent use; their connect/disconnect rating is often in the range of "hundred or so." (This can be partially addressed by using a cable that stays plugged into the drive, and replacing the cable when it is past a certain number of cycles.)
> if you ever wanted to take disks offline and store them on a shelf, you don’t really know how long they’ll survive - unless you plug them in every now and then.
> And yet, the expectation is, that you will be able to read the disks without problem - even with zero maintenance. Indeed, that was the outcome of my ~15 year experiment!
That's not the expectation, at all? CD-Rs used to have a lifetime measured in months before they start showing failures, especially if they had an adhesive label. I don't understand how the author possibly could be reading ten year old CD-Rs. That's basically impossible.
DVD-Rs faired a bit better, and bluray disks even better.
The issue with HDDs is the lubrication of the motor
> any NAS will automatically check for errors, and notify if problems are found.
No. No. No. NO. This is probably one of the biggest myths of RAID and NAS devices.
Just because you have a RAID array doesn't mean it is configured to scrub its arrays; this often has to be enabled.
Just because you have a RAID array scrubbing doesn't mean you're going to find out about it. Make sure your reporting works, and ideally report on success, not just failure, and have something else that reports on failure to receive
Last but not least:
Just because your RAID array controller (or software) finds a parity error or unreadable block doesn't mean it does what you expect, and most people expect "beep bop boop, parity/mirror inconsistency detected. Fixed, meatsack! Pat yourself on the back for being smart in using RAID."
Reality: "beep bop boop, parity or mirror inconsistency detected, so go and verify your files or restore from backup, meatsack."
RAID arrays cannot self-repair mirror or parity errors because there is no way to do so. The array controller has no idea whether it's the parity bit that is wrong, or the data. In a mirror, it has no idea which drive is correct - just that one of them is bad. RAID is not for data consistency.
This is why ZFS is usually the top pick for data consistency: it knows the checksum of files and metadata, in addition to the on-disk redundancy via mirrored copies or parity. If it finds an error, it can play out both scenarios to see which results in files or metadata that match their checksum.
> I don't understand how the author possibly could be reading ten year old CD-Rs.
The cyanine dye CD-Rs were the most unreliable but the later phthalocyanine and azo discs are more robust. Mine all readback fine so long as they have been stored well without sunlight exposure.
Small addendum: The default installs of OpenZFS automatically configure monthly scrubs (typically on the first Sunday of the month), which is a great thing.
Unfortunately, depending on your local mail setup and your user account setup, you won't get the mail that is sent on errors when it's delivered to the root account.
Worth looking into Millennium Disc or M-DISC[0]. Apparently it's media that can last 1000 years. Thinking of buying two 50GB disks and putting some backup files on them when I have a few coins to drop.
There are two types of BD-Rs, high to low & low to high. One is bad for long term and one is good (something something inorganic), but I forget which is which. From my research online bluray m-disc is just the better one but m-disc branded, apparently not the same stuff as the original m-disc DVD.
I personally was going through a large binder of old backups this week that's been in my parents basement away from sun for 15 years.
Many CDs and later DVDs. The DVDs were all seemingly fine but a sizable portion, maybe 20%, of the CDs are flaking along the outside. I was honestly kind of surprised.
Could do RAID 4/5/6 with Optical Disks too for additional paranoia.
Ideally there should be some machine to go through old backups and upon first detection of an error, then it could regenerate the array from the remaining discs of that array.
That’s the biggest problem with optical disks. With no automation, it’s totally impractical for one person to maintain a full backup of everything they care about. 100TB is like 2000 discs plus parity.
Personal photos and videos indeed, I would guess. The one piece of data you can never get back or buy back if your only copy breaks down. Also, a kind of data that takes a lot of space, compared with, say, text documents.
I burned hundreds of discs a decade ago and read them all back perfectly, to my surprise. I strongly agree with the write-once behavior for photos and downloads. HDDs worry me in the long term because even if the platter holding the data is fine, the R/W head, mechanics, and electronics could fail, not to mention that a head crash could wipe out good data.
I'm extremely disappointed that the capacity of optical disc formats are essentially frozen, topping out at 100 or 128 GB for BD XL. They're definitely not keeping up with the continual increase in storage density and decreasing cost of hard disks and flash memory. Why are new formats not forthcoming? Where's my 300 GB Archival Disc? Where's my 1 TB holographic disc? Where's Microsoft's glass optical disc that's stable for millions of years?
I want to keep on burning optical discs and putting them in cold storage, but this gradually obsolete technology is not making a convincing case for me today.
Previously: https://news.ycombinator.com/item?id=26038893 , https://news.ycombinator.com/item?id=29221418