It seems pretty clear that PCIe-based solid-state drives using the NVMe protocol (and U.2 or M.2 connectors) will have much higher performance than AHCI/SATA over the next 5+ years (for both latency and throughput).
Since coreboot (with TianoCore or SeaBIOS) and the impressive nouveau and radeon projects, there's fewer places binary blobs can hide on a modern computing system.
One of the bigger exceptions is the drive controllers (which processes AHCI/NVMe messages and things like wear levelling). Besides restrict freedom and performance experimentation, it's also a security issue, given the scope for man-in-the-middle attacks [1]
Now at the start of NVMe's reign, it's probably an especially good time to start a project to make an open-source replacement to the proprietary firmware blobs of SSDs/flash memory! :)
Personally, I'd rather have open-source firmware for spinning disks. There's a lot of room for optimization of spinning disks. If the firmware could pass through more realistic cylindars, and if the firmware would quit silently re-mapping bad sectors, there's a lot of performance we could get out of spinning disk.
I mean, on the 'remapping bad sectors' bit, right now? to the OS, it looks like you are doing a sequential read, but if there was a remapped sector in there? you're doing a random read. We'd be much better off, at least on server systems with decent raid subsystems, just handling the bad sectors in software.
Right now, I pay almost double for my spinning disk because I want slightly better firmware that is designed to fail outright rather than retry, because all my disks are in raid. some consumer disks allow you to adjust the time-limited error recovery paramiters, but in my experience, it's super unreliable.
Is there any data on this? As someone who works closely with hard drive test teams, I've not seen any indication that there's much room for improvement on spinning disks (except incremental gains from increased platter density). The proprietary firmwares already do a lot to optimize performance and each vendor has their own secret sauce, esp for enterprise solutions.
The key to spinning-disk performance is to access it in a sequential manner. The effect of firmware that does things like transparent sector remapping is that you can never know what, exactly, is sequential and what is not.
The problems I have with stock firmware isn't that the stock firmware isn't good... it's that the stock firmware hides a lot of information that the rest of my system could use to make the access to the disk more sequential. It also makes the assumption that I want this drive to function by itself without complex zfs-style software on top of it that can handle errors.
The idea being that the drives are setup with a firmware that makes the drives okay in a configuration where it's a single drive running ms-dos. In a multi-drive, redundant configuration with something like zfs doing the raid, a lot of the "features" there (retrying for a long time on errors, remapping bad sectors) are problems that are handled far better at a higher level.
So... if your point is that I could just spend the money and get a storage appliance from emc or netapp, you are right, but those things cost on the order of 10x what local storage costs.
If your point is just that I'm being cheap, that's fair too, and that's probably why drive makers don't want open-source firmware; with closed source firmware, they can charge extra for the feature-set that is optimal for a RAID configuration, and less for the feature-set that is optimal for a single-disk configuration. Even though both of those feature sets might be pretty good for what they do, they are largely the same drive, physically, and even if they weren't, I'd be happy to have a somewhat less-reliable drive that reliably returned errors rather than hanging, if it saved me the rather large difference in price between 'consumer' and 'enterprise' drives. - right now, I'm paying the extra for the 'enterprise' drives even thought the overall reliability of consumer drives would be just fine for my use.
All I'm saying is that we work closely with drive vendors and test tons of drives at work. Most of what you're claiming here seems like speculation. For example, the remapping you talk about isn't constant. It happens somewhat frequently, but reads and writes aren't constantly reordered. Also, drive vendors spend tons of R&D on bin packing both on the physical disk and the data stream to try and eek out the most performance possible. Given how much the mechanical disk vendors are threatened by solid state technology, if there were a simple way to increase performance on drives, I'm sure they would provide it at this point. Even if they didn't do it for retail, their big customers would have access to it. I'm not aware of anything of the sort, so I doubt this is as big an issue as you claim.
Open firmware would be great for various other reasons like being able to have confidence in a secure erase solution or being able to have better telemetry to predict failures. That said, I wouldn't put some huge secret performance gains at the top of that list.
> For example, the remapping you talk about isn't constant. It happens somewhat frequently, but reads and writes aren't constantly reordered.
the remapping I'm talking about happens when a sector goes bad. if sector 5 fails, it's remapped to the bit of disk set aside as spares, no? no secret there. but when my computer reads sector 1 through sector 10, my computer thinks it's reading sequentially... but really there's a big random seek right in the middle of it. By my understanding, that happens every time I go to read sector 5.
>Given how much the mechanical disk vendors are threatened by solid state technology, if there were a simple way to increase performance on drives, I'm sure they would provide it at this point. Even if they didn't do it for retail, their big customers would have access to it. I'm not aware of anything of the sort, so I doubt this is as big an issue as you claim.
The point here is that it's not simple, and it's not a conspiracy or anything... and a lot of this wouldn't be a win for computers running windows with a single disk, and the big customers do have access to this sort of thing; like I said, emc and netapp and the other big vendors have their own custom firmware.
My guess is that the vendors don't want to do anything that requires the user to run special software, and all the changes I'm talking about would end very badly for the user if they weren't running a filesystem/raid system that was built to handle that sort of thing.
We have the ability to run special software and they don't give us these features. I'm sure they would if they thought would help.
On the remapping of sectors, that happens very infrequently. Look up smart reallocation count indicators for more info on this. Once a few percentage points of space has been reallocation, smart will assume the drive is failing.
How do you propose to do such an open-source firmware? If you do it by reverse engineering targeting existing hardware, you're continually vulnerable to obsolescence and (spurious but inconvenient) legal threats. If you do it by building new hardware, you discover just how small the market is for more expensive small-batch open hardware is.
Interesting article indeed. I wasn't aware of the politics and rivalry with Samsung behind it. I was kind of rooting for near-future RAM and disks becoming one. Looks like, instead, we will get Rambus v2 story.
My personal preference as a solution for this "problem" would be to make the drives dumb enough that the firmware wouldn't matter that much anymore. Something like ONFI flash with UBIFS. Flash chips are not such unique snowflakes that a generic software with some tweakable parameters could not manage them.
> Flash chips are not such unique snowflakes that a generic software with some tweakable parameters could not manage them.
Take a look at any of the recent reviews comparing drives with very similar flash hardware, yet with wildly different performance characteristics. Firmware matters.
I'd certainly like to see such devices with Open Source firmware, but don't discount the difficulty of the problem.
The flash data sheets are actually just the starting point for using NAND flash. It is not just signal timings and command sets. There are lots of quirks you have to work around to make it reliable and usable which changes with each generation of NAND.
Is there such a thing as a drive controller rootkit that can intercept/process data? Something that can understand common FS such as NTFS and they "lie" about what is really there? Perhaps inject additional executable data into an exe that is called? Could it break hardware encryption on the drives?
I seem to remember Snowden docs indicating that the NSA had HDD firmware hacks in their toolkit, and that such hacks are an example of what they mean by "advanced persistent threat".
Indeed, additionally I recall that the HP server's Iran used had some similar security issue on the RAID(?) controllers. But I hate to write stuff on HN without a source :-)
Wait, I don't understand something. 950 Pro is meant to be the flagship, right?
So how come, that Samsung SM951 is much much cheaper, yet just as fast, or even faster in every metric? The only thing that is slower, is the read speed(2150MB/s vs 2200MB/s). Better write speed, more IOPS, and cheaper.....so how is the 950 Pro a flagship?
The 512GB 950 Pro is $349, 512GB SM951 is as much as $490 if you can manage to find it. The 256GB SM951 is more available, but it's the same price as the 950 Pro at the very best.
And that's before you start to see sales on the 950 Pro, which you will definitely see.
SM951 is an OEM product and you're generally hard pressed to find it available as a consumer in any substantial quantity. The 950 Pro is a mass market consumer product.
I'm not sure what other people were expecting, but I'm pretty disappointed by these performance numbers. I does great in the "destroyer" benchmark, but that's the exception. Steady state random write perf is actually lower than the sata 850 pro!
In other words, if your workload isn't super heavy, this is quite fast - but then, why the heck are you buying this expensive, low capacity power hog if your workload is almost certainly fast enough on a slower drive?
At the very least, this drive should be beating or equaling samsung's own SATA drives reliably - but it's not. The power consumption is also worrying, especially due to the tiny form factor...
I can't think of a situation in which I would care about write bandwidth. Write latency is important for databases. Read bandwidth is of critical importance for everything. But write bandwidth? What's it for?
2+ GBps storage that doesn't require its own desk space and power supply and RAID card! Exciting times for us in film and video. Now all we need is more space and a price drop! ~4TB would be enough for 2k mastering. 16TB for 4k.
Can't wait! Time for dotHill-like performance to trickle down to affordable range. Last two things then are affordable 10-bit monitors and gfx cards with 10-bit output and alternative to infiniband.
Whats wrong with Infiniband? Cost is typically less than dealing with Ethernet NICs that can do RoCE.
Intel is also moving ahead with their Infiniband implementation as the "supercomputing" cross-chassis bus (QPI is a clone of Hypertransport, but Intel decided not to use it for supercomputing bus fabrics like AMD does with their externally switched Hypertransport implementation), so future Xeons and future Phis will have on-die True Scale controllers.
Nothing's wrong with infiniband, of course! Except for prices compared to average SoHo network infrastructure. Video mastering and editing networks usually aren't large. You need a fast SAN with fast network and up you go. Everything around it is so expensive though. In recent years there have been a massive price drop in fast storage, fast graphics cards and now we wait for cheap fast networks and 10-bit through and through.
We're talking mastering here. For editing it doesn't matter since it can and is done in proxies. Depending on the bit-depth and fps we're talking anywhere between ~127 MBps to ~253MBps (1+ - 2+ Gbps) per workstation. Usually there are several and a one heavy NAS or several smaller ones. That's for 2K (and HD - but then it's a bit smaller bandwidth). For 4K it's four times that.
The fiber QSFP modules are the same ones used for Ethernet, ergo, 40gbit Ethernet and 56gbit Infiniband are going to have the same issue with QSFP modules being expensive.
There are companies out there shipping NVMe SSD's in much larger capacities. Novachips is one that we've started testing with, and so far the performance is great.
Their 4TB models are around $1/gb mark; the higher capacity models are much lower $/gb. So far we've only tested the 2.5" models and haven't had a chance to play with the 8/16TB PCIe add-in cards.
This may be a stupid question, but will the 950 Pro PCIe go into a late-2013 and up Retina Macbook Pro? I ask because I looked into a larger capacity hard drive for my rMBP, and I found this: http://www.thessdreview.com/daily-news/latest-buzz/hard-less... and various other pieces saying that commercial PCIe drives weren't available.
Probably not. AFAIK the MBPs with PCIe SSDs use a proprietary connector. This SSD uses the standard one. It should not be physically possible to fit one inside the MBP.
Interesting. Are these meant to be consumer only? I didn't really see anything about data center applications? We run 24 850 pros in raid configuration as base SAN. I wonder what these would spec out in a configuration like that. Is there a PCIe card that can hold several?
Keep in mind that these 950 SSDs get very hot when continuously reading or writing since there is around 6W of heat dissipation on a tiny uncooled board. What happens then is that the thermal throttling will kick in and degrade the performance.
Have you already checked out the Intel 750 series? They are more expensive (the 1.2TB is $0.87 per GB vs $0.69 for the 512GB of the 950 PRO), but I think suitable for an application such as yours.
Are you running 24 drives for capacity reasons or speed? I can't think of many RAID controllers that could push that many SSD's at full speed, so it must be for capacity. Have you thought of using fewer, higher capacity drives instead?
I little of both depending on the end use. Most of the times we use 3 PCIe raid controllers. Some cases 1 24 port controller works great. Also, we keep at least 2 or 3 in the hot swap mode for reliability.
I only found out about NVMe a few months ago, mostly because I've been focusing on the dayjob, but I am really excited about the technology. for a long time, there wasn't anything but proprietary shit drivers for pcie-ssd and if you did get your hands on some pci-e flash, without the proprietary driver, at best, you'd get a MTD. MTDs are way less useful, if you want to run a filesystem other than jffs2, and mtdblk, from what I read, is kinda shit. But NVMe solves all those problems, as far as I can tell. (I haven't bought any yet, but will shortly.)
SuperMicro is selling chassis that have a mix of NVMe and regular sata ports in form factors similar to the traditional 2.5" sata hot swap bays, which is awesome... in my application, at least, it's not acceptable to have to unrack and take apart a server when a drive fails, so without that part (without chassis that have externally accessible hot-swap bays for NVMe) it's pretty much a no-go.
But all the pieces are here. I just need to get together the cash (see aformentioned dayjob) and then figure out what I've gotta buy quality wise. (the samsung pros are on the list, and of course, the nicer Intel stuff) - SSDs really are quite different from spinning disk in many ways; especially backups without zfs. just doing a snapshot and full dd-image of a fast ssd like this would probably be acceptable once a day; something I wouldn't dream of with spinning disk.
For some background, I've fallen way behind my competition, and I'm still entirely spinning disk while most of them are entirely ssd. I'm most of the way to matching them on ram/disk per dollar at this point (not on the front page, but most of my customers are upgraded and the rest will be done soonish) but even then, I'm on spinning disk and they are on ssd.
thanks. the successor to jffs2. as you can see, I haven't looked into this for a while (thus not knowing about NVMe until very recently) so I don't think it existed or was stable then.
For desktop it seems as though the last part that is holding up a major change in form factor design is the video card. While everything else gets smaller, these seem to get bigger and bigger (more fans, more RAM, etc)
Really exciting to see changes like these come. I can't wait to start adopting these into my PC builds. I might just try one for a new HTPC I am building (OS drive).
Not when normalized with read/write speeds. I can personally confirm this by scaling the PCIe bus speed of my own SM951 down to 2x lanes on 2.0 (which limits the r/w to about 550MB/s).
Probably a bit less than that depending on how over-provisioned the drive is. But it's not like you'll be able to write 700 times to the same cell deliberately and that value is only the worst case guarantee. If you followed Anandtech's destructive testing most drives last way longer than they're rated for under normal conditions.
Since coreboot (with TianoCore or SeaBIOS) and the impressive nouveau and radeon projects, there's fewer places binary blobs can hide on a modern computing system.
One of the bigger exceptions is the drive controllers (which processes AHCI/NVMe messages and things like wear levelling). Besides restrict freedom and performance experimentation, it's also a security issue, given the scope for man-in-the-middle attacks [1]
Now at the start of NVMe's reign, it's probably an especially good time to start a project to make an open-source replacement to the proprietary firmware blobs of SSDs/flash memory! :)
[1] http://www.bunniestudios.com/blog/?p=3554