it would be interesting to compare that to a file system with transparent encryption on an intel drive.
this is the best way i have found to think about what they are doing (it's a while since i read the article, so apologies if i am just repeating what is said, or get something wrong - iirc, anand hints at the below but doesn't state quite as much):
given an arbitrary, unreliable storage medium, you need to store both "raw" data and additional information for error recovery. it seems that until now there were technical / historical reasons that made it optimal / normal to store these separately, as i have described above (ie a disk stores a chunk of data and then has a relatively small checksum afterwards).
but there's no real reason why that need be optimal in all cases. for example, raid does something different. raid 1 stores two copies (ignore that each has error correction too, for the sake of argument). now raid has certain technical motivations (cheap disks fail, hopefully independently) that make that reasonable.
so what is new about ssds compared to spinning disks that is the enabling factor here? one guess is that since you can read from various chips at the same time you can do something like raid. for example, say you have 8 memory chips, then you could use a raid 5 style approach with one chip as parity, losing 12% of your space. or two chips losing 1/4 of the space.
my inference (and what i think anand implies at some point) is that when you sit down and do the maths, there's some number of parity chips (say) that allows you to start using cheaper chips (with higher error rates).
but that doesn't entail compression.
so either i am missing something, or the compression is optional - perhaps it is being used to hide the fact that they are having to use so much space for error correction? or perhaps it is just a marketing gimmick?
edit: or perhaps without compression it's actually too slow to sell? i think this may be it. and if so, putting compression on an intel drive will actually beat this.
or perhaps there's something about the approach that means the data have to be compressed anyway?
You're confusing two separate features. The improved error correction allows using cheaper flash. The compression/dedupe is what improves performance and endurance.
what's confusing me is that this is being sold as a "high end" drive. it's not. it's a drive engineered to be as cheap as possible, using lower quality components, that's being sold as "fast" only because it has compression. put compression (eg with zfs - i don't know what other file systems have transparent compression) on an intel drive and it should be better.
but the review hints that the two are linked. if they are, then perhaps it's not as simple as that.
sure, everyone knows that companies will spin things however they can (although i don't understand why some people - particularly americans - seem so fond of pointing this out; it's hardly the most positive aspect of capitalism).
but the review could have been a little more questioning. why not compare it to writing compressed data on an existing drive?
(and i'm sure i don't need to point out, to a connoisseur of free markets like yourself, that although they make a living convincing people to buy the latest product, and so work hand in glove with the manufacturers, they also have to compete for readers by reputation, which requires some level of integrity)
this is the best way i have found to think about what they are doing (it's a while since i read the article, so apologies if i am just repeating what is said, or get something wrong - iirc, anand hints at the below but doesn't state quite as much):
given an arbitrary, unreliable storage medium, you need to store both "raw" data and additional information for error recovery. it seems that until now there were technical / historical reasons that made it optimal / normal to store these separately, as i have described above (ie a disk stores a chunk of data and then has a relatively small checksum afterwards).
but there's no real reason why that need be optimal in all cases. for example, raid does something different. raid 1 stores two copies (ignore that each has error correction too, for the sake of argument). now raid has certain technical motivations (cheap disks fail, hopefully independently) that make that reasonable.
so what is new about ssds compared to spinning disks that is the enabling factor here? one guess is that since you can read from various chips at the same time you can do something like raid. for example, say you have 8 memory chips, then you could use a raid 5 style approach with one chip as parity, losing 12% of your space. or two chips losing 1/4 of the space.
my inference (and what i think anand implies at some point) is that when you sit down and do the maths, there's some number of parity chips (say) that allows you to start using cheaper chips (with higher error rates).
but that doesn't entail compression.
so either i am missing something, or the compression is optional - perhaps it is being used to hide the fact that they are having to use so much space for error correction? or perhaps it is just a marketing gimmick?
edit: or perhaps without compression it's actually too slow to sell? i think this may be it. and if so, putting compression on an intel drive will actually beat this.
or perhaps there's something about the approach that means the data have to be compressed anyway?