ZFS handles the three-tiered model well (with the L2ARC implementation). Also, the authors of this paper didn't consider the "write-limited" implementation that the ZFS authors use: they write at a fixed rate to SSD (e.g., 8MB/sec) and pull entries near the end of the RAM-based cache (i.e. the ones that are about to expire). Still, quite a great paper.
So, can anyone explain why the three-tiered memory architecture described here (RAM, flash, HD) isn't widely used in computing today? It seems to me like 32GB/64GB of flash memory in between disk and RAM would dramatically increase performance of virtual memory, disk access, etc. without increasing cost very much.
There was some work done by Sun recently on this, around last year I think, where they showed how much more performance could be gotten out of a system when flash is added to the mix. I can't quite find the link to the article now though. I'm sure there are others doing this too, but sometimes adoption rates are slow when there's not enough hype. But the issue is probably more to do with software.
I've been researching into this to try it myself for a database-centric web app, but I realized I'd have to write much of the software to incorporate flash. For example, you have to make sure that in-memory caches are delegated to flash and kept there according to an LRU algorithm. I'd also expect items stored in flash to be promoted to in-memory storage when accessed often enough, and demoted to disk when used less. And I'm sure there are other intricacies I haven't thought about that need to be considered to make a three-tier storage system work well.
Rather than writing application-specific code for all this, there really should be some general solution that handles this automatically.
Edit2: I found a higher-level explanation of flash in ZFS. What I didn't realize is that the software is all free to use commercially, though you'd have to be on open solaris: http://blogs.sun.com/jonathan/entry/not_a_flash_in_the
One problem is that in the most lucrative market, enterprise data storage, the woefully out of date benchmarks that are industry standard mainly promote raw throughput on absurdly large working sets, which makes caching of minimal benefit. Besides inertia, there are some established players who have a lot riding on that not changing, also.
Details in this kvetch from Sun engineer Bryan Cantrill (Sun's ZFS-based storage servers do pretty much exactly what you propose, but lose out to NetApp in the SPEC benchmarks): http://blogs.sun.com/bmc/entry/eulogy_for_a_benchmark
Hardware costs won't increase, you are right. But the costs to redesign the software to use an extra caching layer are going to be significant. Also, the current situation with flash memory being expensive won't last long, and moving to pure solid-state storage is still better than this 3-tier architecture. So, it's a large investment into a solution with only a couple years of lifespan.
Of course, this is probably wrong, and there is a startup somewhere doing just this, and they will be rich :)
There are lots of people in research looking at incorporating SSDs into the storage hierarchy in various ways. Not sure how much of this is going to be productized in the short term, but I wouldn't be surprised to see, say, a new release of a commercial DBMS that can use SSDs for some of the buffer cache.
Also, the current situation with flash memory being expensive won't last long, and moving to pure solid-state storage is still better than this 3-tier architecture.
For scan-intensive workloads (e.g. datawarehousing), I'd expect to see magnetic disks continue to be used for quite some time. Buying a petabyte of SSD would be expensive.
EMC do some cool stuff with flash/fibre channel/sata storage arrays with software that automatically moves data between the layers as needed, I'm sure I _need_ one of these for my media server:
http://www.emc.com/products/detail/hardware/symmetrix-v-max....
The author was careful to restrict his comments to SLC drives, which have reasonably long lifetimes.
The newer, cheaper MLC technology flash drives have far less useful lifetime, somewhere in the vicinity of 5,000-10,000 erase cycles. These drives just won't cut it for usages defined here, though they will work fine in your cell phone.
http://www.sun.com/emrkt/openstorage/0309engineers.html?cid=...