This sort of cynicism really grinds my gears. Graphene was first isolated and charaterized in 2004, using Scotch tape to pull samples off bulk graphite. If you were surprised in 2011 that graphene was not already a household staple, that's 100% on you. It takes time to figure out applications, it takes time to mature them, and it takes time to figure out affordable, high quality, bulk production methods, and if the latter was easy it wouldn't be a material we first characterized in 2004.
Cynicism? I think this is how many of us feel when we see all these breathless reports about how much some revolutionary new battery technology will be 100x more efficient at half the whatever, or how some new revolutionary solar energy system will generate 10x the power at one-third the whatever, or how Blockchain will do this, or how AI will do that, or how the decentrialized internet will finally be a reality, or even, yes, graphene will do everything we could have every hoped for at less cost, more efficiency, whatever.
I'm as much a fan of the potential of graphene as anyone. I love the idea of graphene superconductors, etc.
However, to me, casting what you refer to as a "cynical" glance is actually being a realist. There's nothing wrong with cheering on research innovation while also chastising the ability to actually productize that research. So much can be promised in an academic paper that fails upon meeting reality.
In fact, I would venture to say that 90%+ of all academic research fails to materialize in any form into industrial applications.
So is it cynical to ask the question "will we ever see the light of day for this technology in a way that consumers will be actually able to use"? Sounds like a rational and realistic response.
It's probably unrealistic to say "just wait 20-30 more years and maybe you'll see a first application for this technology." Who has the interest or patience for that? Might as well read science fiction if you want to read about technology you'll never actually get your hands on.
There's absolutely nothing wrong with pragramtism and realism. But I'm all for research and academic development. Keep on researching, and maybe one day, some day, we'll see some benefit there. Until then, Graphene will remain the lore of technology potential, but not technology accessible.
Where exactly do you think the relentless process of technology comes from? Rechargeable batteries for example have become extraordinary cheaper in my lifetime. They also have higher capacity, last longer, weigh less, and are basically a pure upgrade with zero downsides.
Sure not everything works out, but breathless breakthroughs of 20+ years ago directly translated into what seems like minor improvements today. The only things that seems disappointing is we are comparing them to yesterday’s tech not what was current back when this stuff was first discovered in a lab.
What you're saying is obvious. Yes, of course, research, especially of the incremental kind results in innovations we see every day.
But there's a difference between the steady, ongoing improvements that follow from continued iteration and improvement and the whole "Groundbreaking, revolutionary technology that will change everything but is always decades away".
The fatigue doesn't come from the obvious incremental improvements, but those earth-shattering ones that always promise to be just around the corner but end up unattainable. From fusion reactors to flying cars, graphene-powered supercapacitors to 50%+ efficiency solar energy cells, and even, dare I say it, truly level 5 autonomous vehicles, these things are always just here, but fail to materialize.
No one is doubting the progress of technology. The problem is the science fiction-like promotion of technologies that fail to see any commercialization.
There is nothing incremental about many of these improvements.
Take say Helium filled HDD, that’s not simply an obvious outcome of taking 1960’s HDD technology and scaling things down. It’s very different technology that just happens to be in the same form factor so people assume it’s just all just tiny linear improvements.
But that's exactly the sort of incremental improvement I'm talking about. What exactly did adding Helium do to HDDs? we didn't invent a new kind of storage, we iterated and improved what we had already developed, and that iteration is itself incremental in nature. And the timing had to be right. If you were to go back to the 1980s and suggest adding Helium as a way to improve storage speed or capacity the industry would just have laughed at you because there were much more obvious ways to improve capacity and performance.
In fact, the necessity to use Helium in the first place is a result of all the steady, incremental improvements bringing us to a point where the use of helium could actually provide value.
A revolutionary, ground-breaking innovation would be to use Helium as the storage medium. An incremental innovation is to use helium to improve the storage approaches we're already using.
This is not at all the same sort of breathless hype as "graphene solves everything" but not a single commercial application that proves that it does. So, will ultra-high-density harddrives really be made with graphene in any scalable, commercializable way? Given all the other touted applications of graphene that have failed to materialize, unfortunately it seems that this is something that will need to be proven in a short amount of time (not 20 years) for those of us who have heard it all with regards to graphene to put any weight behind the claim in this article.
This article is literally about “graphene store ten times more data.” Your standpoint is it becomes a linear improvement if people get to 10x the storage by way of initially having 1.5x the storage capacity.
It’s not that helium or graphene alone instantly on their own get 10x improvements while everything else stays the same. Instead it’s about eventually hitting new limits but being at 10x capacity when that happens.
That's actually not what the article states. The close read on the article states that Graphene is used as a coating, which can function at higher temperatures so that when a different recording approach, Heat-Assisted Magnetic Recording (HAMR), is used, in laboratory conditions, the coating performs well. Graphene here is not being used as a storage medium. It is a coating medium, and even in this application cannot be applied until a different method for storing data (HAMR) is utilized. And even then, the claim of 10x improvement has to do with the potential of increasing storage density if HAMR is used, which can potentially be facilitated by graphene. But graphene in and of itself is not the reason for the 10x improvement.
The reasons why people are skeptical about graphene are many claims like this. Graphene has yet to show the ability to provide the benefits of the claims expressed in laboratory environments for many of the reasons other non-commercializable technology (such as flying cars) failed to materialize.
Maybe one day Graphene will live up to its claims, but the past almost 2 decades is showing that so far, it is good at generating sensational news, but not much else. Maybe we'll be wrong this time around. But it won't be graphene in this case that's the star of the show, it's HAMR. So, no the article, is not literally about "graphene stores ten times more data".
This is the sort of continuous problem with graphene. Graphene can enable something revolutionary if only this was also done, or if this approach works, or if this or if that.
I think you misunderstood. That’s why I said, “graphene alone instantly on their own.” You also don’t get a better HDD by taking a traditional HDD designed for normal air and running it in a helium environment.
But again, you’re sidestepping the point if in 20 years graphene is necessary for drives that are 10x current capacity and fit in existing enclosures by your definition it’s not revolutionary as long as the increased capacity showed up a little bit at a time and the transition point wasn’t obvious.
> I think this is how many of us feel when we see all these breathless reports about how much some revolutionary new battery technology will be 100x more efficient at half the whatever
If you can't cope with the time it takes to develop technology, then perhaps don't read technology news?
You are criticising someone for having an opinion. 15 years is sufficient time to get _something_ to a prototype that has investors clamouring, if not the first production runs.
By comparison, William Shockley filed his patent in June 1948, and Texas Instruments started making transistors for portable radios in 1954. Different technologies move at different rates. Perhaps graphene needs the modern-day equivalent of a Fairchild Semiconductor.
Also, the difficulty of "figuring out affordable, high quality, bulk production methods" of a technology has little to do with the serendipity of its discovery.
The first patent on a transistor was assigned in 1933 to a man named Julius Lilienfeld, who invented (conceptualized) the MESFET. He didn't get the same press coverage as Shockley, and also failed to commercialize. Shockley's 1948 patent was relatively late.
Perhaps we just haven't had a "Shockley moment" in graphene yet. It would be nice if these other people stopped getting so much press, though.
The fundamental difference between Lilienfeld and Shockley is that Lilienfeld didn't or wasn't able to even build a prototype. [0]
"Sadly, Lilienfeld does not appear to have ever built a working prototype of his device; indeed, there’s no evidence that he ever even tried. He also never published any of his work other than as patent applications, so the world would remain ignorant of his insights for another two decades, when Bell Labs started working on what would become the transistor. "
While Lilienfeld had probably given some inspiration or ideas to Shockley, it took actual implementation before it could be turned into product reality. So no, it wasn't 20 years between invention and application. Rather it took 20 years before someone actually put that theory into practical use. And once it was turned into practical use, industry commercialization followed rapidly. It didn't take 20 years after the Bell Labs' implementation for first applications to work.
So it's worth asking what is different here. If people are building prototypes and small scale implementations, what is truly getting in the way of commercialization?
Graphene was discovered. The transistor was invented. There was no question that it would be used to replace tubes/valves in devices such as radios. It was designed for those applications. For solving those problems.
Usually it’s more like 25-50 years. Sometimes it can be longer.
Figuring out how to bulk manufacture something is often just as hard as the initial discovery. There are a ton of technologies that are in the literature but can’t be purchased because they are too hard to manufacture.
Can you blame people for getting fatigued from this kind of news? (Also it's been 16 years ;) )
Every 6 months there is a new use of graphene that promises 10x something, new cancer treatment that is revolutionary, new AI that can solve the worlds hardest problems ...
It's understandable. Science journalism exaggerates results, even if the original researchers and their paper did not. And even the most promising scientific results can take decades to be applied in the industry and become available to consumers.
But at the same time, this is Hacker News, not Mature Consumer Product News. Here is the best place to discuss what tech in 2030 might look like, not what we can buy today.
this might be true for certain graphene stories (or paid advertisements where it's covered up who paid for those) but please don't say that "journalists" (meaning the whole group of journalists) have "zero morals". Because (real) journalists at least report more or less about the information that they receive and (yes) interpret it, that's better than zero morals (how much can be argued). Zero morals are what you find with certain politicians (or social media users or paid-for-studies-by-cartels) that pull their stories out of their behinds with no accountability whatsoever. Journalists still have a certain higher standard. And we shouldn't neglect this or say otherwise because this would lead to further erosion of trust in media. People should trust news sources more than social media posts because real news sources are still (!) more trustworthy than social media posts. Critical reading and thinking for thyself should be on the agenda everywhere, though.
Sure but the revolutionary cancer treatments and biotech stuff made it to market and now we have a new generation of cell and gene therapies that are incredibly effective send powerful used in patients. And mrna technology deployed to hundreds of millions of people to end a global pandemic. Biotech has been moving so fast since the human genome project that it makes no sense to call it a boy who cried wolf situation.
Also google assistant is pretty nuts if you step back and think a out it. Ai has been doing pretty alright as well. IDK what exactly your expectations are for revolutionary treatments and powerful AIs but what we have already is pretty insane.
When I was about twelve years old I clipped a newspaper article about Moller and the flying saucer he had at the time. I'm in my mid-50s now.
My consolation is that there are lots of other flying car projects now. Moller might be bad at taking stuff to production but also he was probably too early.
I believe the poster is jesting at the whole marking of it and let's be fair, it has been marketed as the next big thing and how it will solve so many things very soon; Many new science articles of how they did this and that and yet nothing.
Now, whilst I can appreciate that from lab to industrial production is a very long bumpy process and can often be longer than it took to do the many year lab work. That whole aspect is almost always overlooked in the marketing (news release) rewrites that filter down into Joe public news feeds. That I appreciate and get.
So for me the real issue is not cynicism slants, it's the way news media takes such science and run with it to the stage that they build such expectations.
Though we have been here before - remember flying cars and the public mindset in the 60's-80's era or indeed robots circa the World fair and how they would be in everybody's homes.
Oh and I'm extremely mindful that to do some science you need funding and with that, you often have to up-sell what your doing to be able to do that science in the first place.
Which is why I don't default view such cynicism as 100% down to the individual, when they are a product of so many links in the news/science chain and I'm just aware of that whole news production line from scientists after some money to do some science, to those that fund it wanting a ROI on their investment, down to institutions making carefully worded news release statements that will not say how or when, just push you into how this `could` or `will`(without specifying a year) do X,Y or Z better; And on paper they are right. But then the whole reality from lab to your home is a very long bumpy road and not everything is cost effective or that clear cut. Which explains your flying cars phase and kitchen robots phase of decades past.
REBCO was discovered in the 80s and we're only now rolling tapes into production. HTS are a game changer for what humans can do and yet it still took 40 years from discovery to first industrial use.
REBCO = rare earth barium copper oxide
HTS = high temperature superconductor
Apparently we can now make tapes of REBCO, which is a high temperature superconductor. The primary intended application is fusion research? Why wouldn't this be broadly useful?
Most industries consider LTS fields good enough for their application and don't consider the investment into HTS worthwhile. If an MRI works at 6 T then why would they want to spend 2-3x as much for a 20 T field? You might get a better product but you'll price yourself out of the market. Similar logic for NMR, gyrotrons, etc. Every application that I can think of that uses LTS would benefit from HTS.
Hopefully as supply chains are established and economies of scale kick in for HTS the price will go down.
I still imagine that the manufacturing process for graphene still involves scientists in white lab coats, hunched over sheets of graphite, painstakingly applying scotch tape to it and cautiously lifting it again. That's just the mental picture in my head.
> By coating graphene onto the exterior shell of the helmet, the result is a helmet with improved thermal comfort and safety. The graphene coating allows better distribution of impact force, making the helmet less susceptible to damage compared to helmets without graphene, even in high temperature conditions. It is the excellent thermal conducive properties of the graphene coating which dissipates heat quickly across the helmet and not only protects the inner materials from degradation caused by heat, but also provides a more comfortable user experience. [(De-)emphases added -- CRC]
So which is it: Can a mere coating somehow affect "distribution of impact force" -- which one would think is done by the actual structure of a helmet, not a surface coating? Or does it actually do this just indirectly, by "protect[ing] the inner materials from degradation caused by heat"? It feels very much like what actually provides the safety in this helmet is still good old glass or carbon fiber, and the graphene is more or less incidental.
Just like the headline of TFA here should probably have been something like "Graphene lubricant helps HMR achieve 10 x data density".
i always imagined if flying cars were a common thing we would need iron umbrellas to be safe from all the metal rain of falling parts from all the mid-air collisions every day...
What I think is significant here is that a massive industry has a clear use case for it, and it's an industry that is fighting for its life with respect to SSDs.
A 10x capacity jump might justify an initial investment as well, especially if the durability is a huge improvement. Economics of cloud datacenters may bridge the divide.
Also, I'd posit that that graphene doesn't need to be a perfect uniform surface for the HDD platter. If they can just get usable sections (ahem, sectors?) then it becomes workable and usable.
There's a decent chance you do and don't know it. A certain type of super capacitor is built with graphene, so you may have electronic devices with graphene in them.
The "graphene" probably does add some modest mechanical and perhaps thermal properties to the foam of the shoe. There are plenty of papers and I'd suggest that you search Google Scholar and Google Patents for 'graphene rubber composites'; you'll be overwhelmed with examples.
However when discussing "graphene" it's helpful to differentiate between different types of two dimensional carbon lattices based on size and mode of manufacture.[0] There are two broad categories when we're talking about manufacturing applications at present:
The graphene in those runners is probably an exfoliated graphene micro/nanoplatelets made form graphite microparticles (previously an in-demand commodity for Li-ion battery anodes). The process used, given the industrial scale required, is most likely a chemical oxidation/reduction process (kind of messy), electrochemical means, or intercalation. The end result is more like confetti or the bits of paper left over from the hole-punch. This isn't your high-quality, electronic grade graphene, however most of the mechanical, thermal, optical, and electrical properties of graphene remain, and this graphene confetti can be blended into composites for the sake of those mechanical properties or to provide thermal or electrical conductivity.
Any company can blend a bit of graphene into their rubber or plastic and have the product given that sciency sheen. But you really want that graphene to be firmly embedded, since there are potential safety issues _in vivo_ [1]; Health Canada issued a recall alert for graphene-containing face masks in April, due to the possible risk of inhalation. [2]
The electronic grade graphene is generally larger in those two dimensions and presenting fewer defects — a full sheet of paper compared to the confetti type. For early experimentalists, the best source was also through exfoliation, albeit using Scotch tape to peel layers off of a large (1 cm or more in lateral dimensions) single crystal of graphite. Nowadays, epitaxial growth (i.e. grow one crystal using a different crystal as the template) is preferred, as it can be mass-produced, even using roll-to-roll techniques. That graphene can also be more readily transferred from one substrate to another, which is necessary since you are going to want your sheet of graphene laying on an insulating surface, rather than the metal on which it was epitaxially grown.
There are several other flavours of "graphene" — nanoribbon, oxide, fluorinated, multilayer, ligated, crumpled, charged, patterned, etc... But the two most consequential categories to keep in mind are whether you are dealing with "confetti" or "sheets". The nanoplatelet confetti can get inside you if not thoroughly immobilised.
You can buy graphene enhanced shoes [1], not really sure its much better than carbon black but it sounds cooler. I generally agree with you, from nanotubes to graphene no one can seem to really make it into anything. Cool physics though.
I think some of the newer smartphone chargers use graphene. It also looks like some lithium batteries that use graphene are starting to hit the market.
You can find the same curve for most per unit costs when manufacturing scales. No one is doubting decreasing average costs, but similarly no one considers that a breakthrough. A breakthrough is something like an 10x energy/weight ratio, which really does require new chemistry rather than economies of scale or marginal improvements in weight savings that come from more efficient cooling or lighter housing systems.
So the solid state batteries that are being developed now would be an example of a true breakthrough. Cheaper batteries in response to a surge in market demand are not.
I wonder whether (spinning disk) hard drives will still have a market in 10 years' time? I'd always assumed they were a bit of a dying breed and when the per-GB cost of SSDs reaches parity they'd become a museum relic...
The current forecasts don't predict cost parity for the next few years. I wouldn't speculate as far out as ten.
What you'll see is that spinning rust will increasingly become used for cold(er) storage due to still offering cheaper bytes, and (relatively) ever increasing cost of IO (you get one spindle per il unit, but at increasing capacity, with barely a change in sequential throughout and no improvement in random access). This is happening in data centers now, either through explicit product tiring or clever offloading.
I would assume they will go away pretty fast because of power usage.
SSDs are currently targeting performance because of the cost, but it seems possible to make them even more (and much more) power efficient when used as high latency storage.
For enterprise servers (I'm aware they are not relevant for AWS/GOOG etc), more and more often high-end machines are offered as SSD-only - for such price bracket SSDs price is already not an issue.
SSDs have extremely finite lifespans since the writing process irreversibly degrades the media, while in HDDs wear theoretically happens only on power cycling and they otherwise have no intrinsic limits on how many write operations they can handle.
Hard Disk also (usually) give some hints before dying; noises, growing number of bad sectors, slowness, etc, while SSD (also usually) die abruptly without giving any warnings or leaving time for recovering files. That wouldn't be a problem in data centers where redundancy and backups do exist, but in desktop machines virtually nobody keeps backups, so it can make a world of difference.
> in desktop machines virtually nobody keeps backups
Unless you are me, who has had tough lessons at The School of Hard Knocks, with postgraduate degrees from The College of Getting the Shit Kicked out of You.
> Also, both MS and Apple nowadays make it pretty easy to keep rolling backups to an attached a USB HD.
Re Apple: this is true only for desktops or laptops. If you want to back up your iPhone you need a Mac to work with a local storage solution. And you cannot configure the Mac to use the HD to store the phone backups directly: the folder the Mac uses cannot be a symlink nor can it be configured. You have to sync all of your phone's data to the Mac, then sync the mac with time machine, which means your Mac needs to be large enough to store all of its data, plus the iPhone's.
I would love to know more about that. I've got a laptop for which there is plenty of storage on a NAS connected via wifi, but the laptop itself has the same size storage as the phone so can't actually backup the phone.
> while SSD (also usually) die abruptly without giving any warnings or leaving time for recovering files
I had SSDs dying gradually and HDDs dying abruptly too. I think this is just the difference between the controller hardware failing or the firmware encountering some unrecoverable condition vs. failure of the storage medium.
That said, there still is the difference that HDDs retain data much longer, flash cells leak charge over time. So for long-term storage HDDs are still a little better, but "long" here means many years, so you're still in the domain where you're risking age-related failure, redundancy will be required either way.
Mechanical failure is very common on PCBs, especially ones with thousands of solder pads and aging solder joints.
I keep a plastic lap pad under my laptop when carrying it around horizontally. I see people just letting the whole torque of the laptop bend the motherboard and it hurts me. Those laptops will one day not POST because the CPU, RAM, or GPU will have a pad lifted off the PCB.
AFAIK, SSDs usually give one warning before dying, and that's suddenly becoming irreversibly read only. They might not always survive a reboot (which certainly would be the first thing I'd do if my OS suddenly reported a drive as RO) but you usually do have a time window to get your data off safely.
An SSD randomly giving the ghost is one of my greatest fears when it comes to desktop computers, because there's no way to get any data off them when they die. With spinning rust you can usually send the drive to an expensive data recovery company who will likely transplant the platters and send you a copy of what's on disk, but if a flash chip is slightly damaged because of a short or whatever, you won't have such luck.
Yes, I know, I should make more backups, but every time I thought I'd gotten everything safely backed up, I found a file or folder that I forgot to include.
An ssd becoming read only, and having random read errors happens when it is overheating aswell.
I’ve had an nvme reaching temps 70+ celsius, because the geniuses who designed the mobo put the socket right where cpu hot air flows..
I aged a couple of years when the machine bsoded and then told me windows files are missing. I thought everything is gone, but fortunately when the ssd colled down everything went back to normal.
Had to put an extra fan in the case, and disable couple of cpu cores though to make the system stable.
Unless someone with more experience can chime in and add some more info, I’ll say that my understanding is that HDDs are better for archival since they last longer when not in use. Certainly the price of SDDs, while within reach for most nowadays, is still significantly higher than for HDDs. I still use HDDs for backup even though I haven’t used them for daily hard drive purposes in years.
As long as the platters are intact, a stuck motor will not prevent someone with the right tools from reading data off of the drive. There's an entire industry dedicated to recovering data from hard drives in various states of disrepair.
A bricked NAND chip, on the other hand, is an entirely different kind of problem.
This cries out for a missing product: A modern version of the old disk pack drives. You could pull out a cartridge containing the platters and store it in a hollowed out mountain for a century, but just drop it into a freshly built/rebuilt/exercised mechanism when you need to read it.
The magnetized material on the platter can slowly lose its direction (and thus the data) while laying around. So you should read all your data every 3 years or so to "refresh" the data.
Magnets only lose their magnetic powers very slowly. For example, samarium cobalt magnets might decrease their magnetic strength about 1 percent over a decade, which contradicts your claim.
Samarium cobalt are actually some of the best from this point of view, because they have a very high Curie temperature.
Most magnets made from cheaper materials with lower Curie temperatures, e.g. Nd-Fe-B or ferrites, lose their magnetization at higher rates at room temperature.
Losing a few percent of the magnetic strength for the whole magnet would be equivalent with massive data corruption on a HDD, because that is the percentage of magnetic domains whose magnetization has rotated, i.e. the percentage of corrupt bits. The magnetic materials used in HDDs have actually a much lower rate of spontaneous change of magnetization.
Nevertheless, when a single HDD contains around 10^14 bits, even a 10^-14 per year probability of a bit flipping at ambient temperature may cause one corrupt file per year of storage per HDD, which is close to the typical rates of corruption actually seen in reality.
>Nevertheless, when a single HDD contains around 10^14 bits, even a 10^-14 per year probability of a bit flipping at ambient temperature may cause one corrupt file per year of storage per HDD
Hard drives add ECC info to each sector to prevent random bitflips from corrupting data[1]. Also IME actual rates of data corruption is far lower than 1 bit per year. This is on a filesystem with checksums, so any corruption would be easily detected.
If the files are overwritten frequently, then the error rate would be much lower. Even if they are only read frequently that would lower dramatically the error rate, because the HDD controller will rewrite the blocks with corrected errors, which happen frequently.
In about 200 TB of high-quality HDDs from 2 vendors and in various capacities, between 0.5 TB and 8 TB per HDD, I have seen around 100 errors after 5 years of cold storage.
There is a temperature-dependent probability for the magnetization of a magnetic domain to rotate spontaneously, causing one erroneous data bit.
For materials with high coercitivity and at temperatures much lower than their Curie temperature, the probability is low but it is not null.
A few years ago, I went back to archiving my data on magnetic tapes (LTO). However, before that and after optical discs became to small to be usable as an archival medium, I have used HDDs for backup and archival, for more than a decade.
After returning to tapes, I have gradually transferred the old HDDs to tapes. I had a case with one HDD that would no longer spin after a few years of offline storage, but most functioned, at least enough to be able to transfer them on tapes.
All the archival HDDs were duplicated, for redundance, and all the files were stored with checksums to detect data corruption.
The checksums have been absolutely essential, because after several years of storage there were a lot of corrupted files, but for none of them the HDD reported any error. The HDDs were about half WD and half Seagate, all of them Raid Edition or equivalent.
I had a HDD archive with about 200 TB of data, and on average, after 5 years of storage, there was a corrupted file about every couple of TB.
However, because each HDD was duplicated and the errors did not affect the same file in both copies, I have recovered all files.
All the corrupted files were almost certainly caused by spontaneous magnetization rotation affecting a few bits, so what the previous poster said is not only theoretically correct, but proved in practice by at least one example.
Most people do not realize how frequent data corruption is on HDDs, because they do not use file checksums or extra file copies, to detect corrupted data.
Bit errors are frequent on HDDs, but most are corrected by the HDD firmware. However, from time to time the correction capabilities of the codes used in HDDs are exceeded and silent data corruption happens.
After a lot of experience, I never trust the firmware of any storage media, so I store checksums, in extended attributes, for all the files that I will keep for more than a day.
What is your recommendation for a cross platform program that can automatically create and store checksums for all the files in a HDD (or folder) and compare them with the files later and restore damaged files from another copy?
Ideally, whenever a file is modified, the checksums should be recomputed when the file is closed, after having been opened as read-write.
However, this behavior can obtained only by replacing the file system drivers with a custom version. That could be done on Linux but it cannot be done as a cross-platform solution.
So for cross-platform, you need a pair of scripts, one for verifying the checksums, recursively for all the files in a directory, and another script for updating all the checksums for the files in some directory, which you need to either
invoke manually after editing some files or downloading some new files, or you could set the script to run periodically, e.g. once per day, maybe prior to a backup script.
Nowadays a bash script can run on any operating system, so it is cross-platform enough, but writing a new pair of scripts for each operating system for whatever tool is used there for scripting is not much work.
The 2 scripts should consist each of a pair of scripts, a top level script that gets all the files from a directory (e.g. on UNIX using 'find "$1" -type f') and invokes the second script for updating or verifying the checksums for a single file.
The scripts that do the work for one file just need to invoke some standard checksumming programs, e.g. sha256sum and so on, which are now available on any platform, if you take care to install them. For verifying, missing checksums or checksums that differ from a newly computed checksum give an error, for updating, the checksums are recomputed if they are missing or if their timestamp differs from the file modification timestamp.
In the beginning, I was storing the checksums in one extra file per directory, but that had many disadvantages. Then I have switched to the use of extended attributes, which allow the storage of the checksums together with the file, transparently.
Extended attributes are much more convenient, but their cross-platform use is more complex.
For updating or verifying the extended attributes, you need 3 commands, list, read and write extended attributes. These not only are different on each operating system, but they are some times different on a single operating system for different file systems.
So you either need to write different scripts for each platform, or you must detect the platform and choose the appropriate extended attribute commands. The work is about the same for both choices, so it might be simpler to just have different scripts for each platform. If you use per platform scripts, they can be small, less than a page each, so it is less likely to make mistakes in them.
When you use extended attributes, you must take care that on open-source systems, like Linux or FreeBSD, not all file systems support extended attributes and those that support them might have been configured to ignore them.
So, especially on Linux, you must check that you have enabled support for extended attributes and you might have to recompile the common utilities, like fileutils, because by default you might have commands like cp and mv that lose the extended attributes. Also neither all GUI file managers nor all archiver versions (e.g. of tar or pax), handle correctly the extended attributes.
The main pitfall on Linux is that tmpfs has only partial support for extended attributes, so on many Linux systems, copying a file to /tmp will lose its extended attributes.
When copying files through the network, the most reliable way is to use rsync, preferably rsync over ssh. That will copy any file together with its extended attributes, regardless what operating systems are at the 2 ends.
Otherwise, Samba can be used between most operating systems. I have used it in mixed configurations with Linux, FreeBSD and Windows computers, without problems with preserving the extended attributes.
There are new NFS versions with extended attribute support, but all the computers must be configured correctly, with non-default options. I have not used such a version of NFS.
So the conclusion is that the files themselves are cross-platform if you take care to not use for transferring them methods that would silently lose the extended attributes, but the simplest way is to rewrite a pair of updating/verification scripts for each platform, because those can be very simple, while a cross-platform script would be large and cumbersome.
Out of curiosity, how would you compare this particular distribution of complexity to that of setting up and using ZFS?
ZFS gives me set-and-forget sha256 checksums that are kept up to date in as data is written in real time. And the checksums are block-indexed, so 100GB files do not need to be-hashed for every tiny change.
I also never have to worry about the "oh no which copy is correct" logistics nightmare intrinsic to A->B backup jobs in nightmare scenarios where they fail halfway through, as both copies of my data (I'm using 2-way mirroring at the moment because that's what my hardware setup permits) are independently hashed and ZFS will always verify both sets on read. (And reads are fast, too, going at 200MB/s on a reasonably old i3. No idea how any of this works... :) )
As for cross-platform compatibility, my understanding is that if the pools are carefully configured correctly then zfs send/recv actually works between Linux and FreeBSD. But this kind of capability definitely exceeds my own storage requirements, and I haven't explored it. (I understand it can form the basis of incremental backups, too, but that of course carries the risk of needing all N prior tapes to the current one to be intact to restore...)
The two mostly-theoretical-but-still-concerning niggles that I think a lot of setups sweep under the carpet are ECC memory distribution, and network FS resiliency.
All the ranting and raving (and it does read like that) out there about ECC means nada if the RAM in the office file server in the corner implements defense in depth (if you will), while files are mindlessly edited on devices that don't have ECC, through consumer/SOHO grade network gear (routers, switches, client NICs, ...) that mostly also don't use ECC. Bit of a hole in the logic there.
I did a shade-beyond-superficial look into how NFS works, and got the impression the underlying protocol mostly trusts the network to be correct and not flip bits or what have you, and that the only real workaround is to set up Kerberos (...yay...) so you can then enable encryption, which IIUC (need to double check) if set up correctly has AEAD properties (aka some form of hashing that cryptographically proves data wasn't modified/corrupted in flight).
SMB gets better points here in that it can IIRC enable encryption (with AEAD properties) much more trivially, but one major (for me) downside of SMB is that it doesn't integrate as tightly with the client FS page cache (and mmap()) as NFS does, so binaries have to be fully streamed to the client machine (along with all dependent shared objects) before they can launch - every time. On NFS, the first run is faster, and subsequent launches are instantaneous.
Spinning disks under GMR heads are fundamentally a 2D technology. You might eke out a few more layers with a GMR array (like radar for magnetism), but there will be a noise limit. Flash technology will become stackable. Not just multi-bit per cell. The major limitation to NAND flash going to multiple layers is the inability to do epitaxy on deposited layers. The other approach is to use new materials that offer some form of hysteresis in amorphous or polycrystalline form, like 3D X-point. This product line was abandoned for business reasons, but I’m fairly confident that it will make a comeback in a big way, either by [insert big Asian company] buying the IP, or a well-funded startup.
There’s also more exotic data storage like optical holography. That’s also 3D data storage, but with very little heat accumulation, which is the problem with anything based on 3D electronics. It is technically a spinning disk though.
HDD, in terms of Revenue and TB Shipped are still growing in Datacenter segment.
And people have been saying this for a long time. I remember a few vocal members in certain popular Internet forum keeps bagging on about it in 17/18. ( And we are half way 2021 already ) There are currently no roadmap from both HDD and SSDs they will reach per-GB cost parity. Not in the next 5, and nothing that shows this will happen in next 10 years.
As a matter of fact there are plenty in the pipeline and R&D that could drive cost /GB on HDD cheaper. I dont see anything substantial with NAND. Other than the usual multiple layers and die shrinking. Both of them have fundamental issues with cost.
There are some great services out there for people who want their designs made at that level. Speaking as someone who has worked in fabs - there are so many ways it can go wrong and so many ways you could kill/poison/burn/electrocute yourself if you tried to do it yourself. Your lithography would probably need a lot of tweaking too!
50 years is optimistic. The equipment needed for vacuum, plasma sputtering, and masking hasn't changed much in the past 50 years.
Sure, if you want to make a 1 um process you can do that with a purchased wafer, some chemicals, and a spin plate. If you want to make a nm-scale process you'll need to have a serious budget.
...and lose ten times more data when they fail? I found it a bit ironic that the stock photo of a HDD appears to be a Seagate 7200.12, which was one of the more infamous ones with dismal reliability.
(Most people reading the article probably wouldn't notice, but as someone who had to deal with a ton of bad ones, the tiny actuator magnet and only 3 screws securing the platters are a distinctive feature.)
- Spinning disks consume 5-10W when idling (as in ready to read/write idle).
- Spinning disks have 2 motors (spin and head driver) that inevitably fail.
That said the SLC(1) -> MLC(2) -> TLC(3) -> QLC(4) layers is a race to the bottom with increased complexity and failures over time!
These are the reasons big actors still use tape as storage medium!!!
There is no great solution, but 64GB 50nm SLC SSDs from 2011 have the best write per bit that will ever be seen (100.000), they only use 0.07W on idle and ~2W when writing/reading. So if your motherboard has alot of SATA ports it might be the best solution until some real innovation happens!
Edit: Maybe I'm the only person in the world that can have 5 million customers on 512GB? Atleast I can have them for 250 years if they all write to the database every day now no matter how expensive electricity becomes!
Wrong. One motor. The head moves with something called a voice coil. It's completely solid state. Open up a hard drive and watch it move back and forth while it's working. Of course this isn't good for the hard drive, but it looks basically instant to the naked eye. Very cool.
30x64GB = 1920GB. That is nowhere near the storage requirement people have, no case or PCIe extenders will give you hundreds of sata ports. And power consumption? 100 2W SSDs still consume 200W in use.
But this is not for (longer-term) storage. Consider scratch space for graphics, video editing, massive compilation. Something that needs a lot of fast writes.
A RAID0 array of HDDs is a traditional way to solve it; it's slower than an SLC SSD. A QLC disk won't be any faster than an HDD at high write loads, but would die significantly sooner.
Of course, for your home NAS you just take a bunch of 5400rpm HDDs and store your videos on the cheap. And for your gaming PC you take a cheap QLC SSD and enjoy the high read speeds while doing pretty little writes.
Enterprise SSDs use capacitor backed dram to buffer writes.
Some consumer SSDs do that, too. But with significant smaller amounts of dram.
This can intercept all the small writes that you e.g. do for metadata of the filesystem and via mapping can combine them into very few big writes. Or writes to the same position over and over again only need to touch the flash when the devices is powering down.
Are there any QLC enterprise SSDs? All the cheaper used enterprise SSDs I looked up where using TLC so far.
I'm pretty sure the only real factor in reliability is the SSD controller. The NAND will affect the lifespan of the SSD but when you have TBs of data a lot of it is written once.
1) They don't make them any more. SLC scarcity is about to blow up!
2) MLC is completely garbage! All SLC drives in the world will still be operational decades after all MLC has completely stopped working!
3) You need to have stable storage (quality), everything else (quantity) is less important because you can work around it with compression f.ex. OTOH when a bit stops working you are screwed!
Sidenote: You need to be able to power your machines on lead-acid batteries, so magnetic is out of the question, you also need to power the drives independently from the motherboard to get power stability and the ability to power them when the mains go out for longer times and not loose data! 0.07W x 8 on 100Ah = 3 months!
_HDD_sector.svg){kind=link}
I've been making this joke for ten years now, it's truly the gift that keeps on giving. I doubt I have a single graphene-enhanced product in my home.
Maybe one of these days...