As others have pointed out using consumer grade SSD drives for a database is a bad idea. Databases are going to do a lot of reads and writes so that's going to lead to problems down the road.
The second issue, is that since this was setup in a RAID I'm assuming that all of the drives were purchased at the same time when it was originally setup which means depending on the supplier you are probably going to receive drives from the same batch of production. Because of striping with RAID if you have some fault in a batch and multiple drives from the same batch then you can have failures on multiple drives that occur at approximately the same time.
The simplest solution is of course to buy enterprise grade drives but the obvious issue there is the price is much higher. Especially if you are going to be purchasing the largest drives as you always pay a premium for max capacity drives and the price jump on large drives is not insignificant.
When we first built out our SSD cloud for DigitalOcean at the end of 2012 we went with consumer drives because we were bootstrapped and SSDs were also significantly more expensive so we couldn't justify the price jump without knowing if we had product market fit.
But rather quickly we ran into a whole host of issues. Another common issue is that performance can degrade when consumer drives get to near full disk capacity so you can't really use all of the available space if you plan to do a significant number of reads and writes, which is pretty much what a database server is designed for.
That was the first upgrade we did after raising money, which is switching to enterprise grade drives, unfortunately you have to pay the premium but it really reduces problems significantly.
I do believe your experience, but started wondering what makes an enterprise drive more reliable than a consumer-targeted one. What's preventing the manufacturers from selling the same drive with two different names and prices? Reputation?
Enterprise class disks have significant behavioral differences when dealing with a bad sector, among other differences. Intel has a good paper on this subject. See 3.2.1:
I think mainly warranty, although other posters are saying enterprise drives have capacitors to help with writing during power loss. I'd have thought a good battery backed raid controller would handle this but "enterprise"="belt and braces" so I can see why a lot of businesses pay the extra.
Sorry, late to reply! The warranty differences are things like with enterprise they'll send you a new one as soon as the SMART data suggests it's going to fail. Consumer is usually you post it back and they send you a new one.
If you actually read the specification on consumer drives you'll see they're "guaranteed" for like 100 rewrites and you don't have any right to anything if they fail.
Disagree, Consumer grade hardware is a fantastic idea because it's so bad it forces you to write robust software.
Enterprise hardware is 3x the cost of consumer hardware, but on average it's at the most twice as reliable. If your software robustness and redundancy is high enough, the total TCO will be lower and no less reliable.
The real wtf with CamelCamelCamel is not that they were running consumer drives, it was they were running 3 replicas on consumer drives with what seemed to be a too heavy workload.
Competent ops being expensive wasn't part of the discussion, Good people are expensive period. In the long term, competent people provide better value. Competent ops would figure out to provide the same reliability guarantees at a lower price. Security likewise, nothing to do with it. That's just using circular logic to say using commodity hardware = bad security (wtf!?).
Sofware and by extension infrastructure gets expensive because of incompetent dev/ops, who are too incompetent to implement efficient systems and use a 'hardware is cheap, throw hardware at the problem' mentality in ops. This especially shows today with the fact that we have have mobile phones with 1000x the processing power of desktops 20 years ago that are sluggish when running a simple TODO app. LET THAT SINK IN. Those same 20 year old computers running the first pentiums were running Quake 1(!).
Back to ops, Google before they were "Google" were using custom built commodity hardware using a 'nodes die, deal with it' approach to software development. Obviously this pattern is a win with the popularity of k8s today / ephemeral nodes but it requires a more complicated development process.
This extends to incompetent ops as well. Who are outsourcing all their work to the cloud, paying a crazy premium for it, calling it a day, whereas the competent guys are able to do it properly.
Closer to the subject, Backblaze engineered a from scratch design to create a highly reliable/available/secure system based on non enterprise class storage that lets the offer storage at a fraction of existing solutions.
Unfortunately for databases under a heavy load it's out of the question.
You can have a server pegged at 100% CPU waiting on disk with spinning disks, and without changing anything else on the server RAM or CPU, and just replacing the drives with SSD you can quickly see that load drop down to 10%.
Random reads are an order of magnitude slower so for fast access there's a notable difference. Still a role for hard drives with sequential reads/writes, such as the regular backups they should have been taking! Slower in all areas but cheaper for the same space.
>On the evening of Saturday, January 26th, our database server had three hard drives fail. It was designed to handle two disk failures, but three failed disks made the situation catastrophic.
Like in the joke.
Our data storage was so well designed, that when first disk died, we didn't notice. When second disk died, we also didn't notice. When third disk died, we noticed.
They don't fail that reliably, unfortunately. It's not a light bulb with a certain number of hours on it. See Backblaze's drive statistics.
Edit: I just saw it's about ssds and not hdds, but while Backblaze might not have stats on those, I'm fairly sure my comment still applies. Not 100%, but I assume they account for failures due to predicable issues like write wear.
There are other comments in this thread that go into this, but your comment doesn't apply. Another poster used the light bulb analogy well to describe SSD failures.
It's very common to see SSDs that were purchased at the same time, that were likely manufactured in the same batch, fail within hours of each other. Im' pretty sure I even read this fact on Backblaze.
All SSDs were from the same batch and had the same amount of writes on them and all failed in a very small window. It's a known issue with consumer drives.
Luck? And hence it makes the news because such bad luck messes things up? We all know drives fail, I'm not surprised this happened somewhere on the planet. I would be surprised if it happened to me.
That's what you'd have monitoring and alerting for. The nagios/icinga/icinga2-world has check_mdraid, which works out of the box. Zabbix should have something similar
To me, using consumer-grade SSDs seems less the problem compared to not having a database replica (for quick recovery) and PITR -capable backups (for near-zero data loss). These are not difficult to set up and are definitely worth it if your data matters.
In most cases your disaster recovery scenarios should include the complete destruction of a single server.
Using consumer grade SSDs is a huge no no. For a company this size a simple server rack with something like production ready Seagate drives would do better in the long run. In my view, the SSDs they bought make CCC look like a school side-project.
I think some people rely on the historical fact that consumer 7200rpm SATA drives used to be nearly identical to their enterprise 7200rpm counterparts. Consumer SSDs are very different from enterprise SSDs; the former typically lacks power loss capacitors, which should be a nonstarter for anyone using them to run databases.
The reason to avoid consumer SSDs in "enterprise" situations generally isn't their IOPs... the IOPs of higher-end consumer drives is more than sufficient for a lot of enterprise-y use cases, especially read-heavy workloads (which I would guess is CamelCamelCamel's workload)
With enterprise drives you're paying for things like beefy capacitors to provide safe shutdown in case of power loss, write durability, etc.
When I was doing managed Colo we used Intel "Datacenter" drives (ex. Intel S3500 SSD) which were priced at around $1/GB (so ~3-4x as expensive as the consumer counterpart). They were performant and, luckily, we never had an issue with them.
>so ~3-4x as expensive as the consumer counterpart
This is likely the answer. If CCC has to ask for donations to afford the consumer version, I imagine they are in no position to purchase an equivalent amount of storage in enterprise SSDs.
I guess they're busy now trying to fix things, but since they didn't mention how to avoid this in the future, I feel like streaming replication [to much cheaper spinning disk] or some other form of point in time recovery is the way to go.
Consumer SSDs can absolutely be the right choice in some situations. Great for scrappy development environments or things that are refreshed regularly like running ETL migration runs. I'd advise underprovisioning them by 10% and keeping good (regular) backups though. Also being aware of the failure of two drives of the same age in RAID1 being common (same number of writes=similar failure time).
A shame that CamelCamelCamel seem to be running on a bit of a shoestring budget as it's a very useful tool. To be fair to the recovery costs though the majority of them are from a professional data recovery company and that ain't cheap!
Does anyone know whether they got unlucky with 3 simultaneous failures, didn’t notice earlier failures, or had some other issue that killed all the drives?
Well, from what I see in the pictures they are using consumer grade ssds to power up an enterprise like grade database. This is not bad luck, this is just bad by design.
As dddddaviddddd said, it's a price tracker for Amazon. To go into a little more detail: you can enter a product name or Amazon URL and it'll show you first and third party pricing history. You can also set a price threshold on a particular product and it'll either tweet or email you when the product price drops below that point. You can even do it without signing up or logging in.
I've set notifications on a couple items that I regularly order and when they drop below a certain price I order enough in advance.
> To go into a little more detail: you can enter a product name or Amazon URL and it'll show you first and third party pricing history
They also have browser add-ons to make that more convenient. They give you a CamelCamelCamel button that you can click while on an Amazon page to pop up the price history right there.
56TiB of gp2 SSD storage on Amazon would cost $6,500/mo (so $78,000/yr) - so if its a 1:1 setup they could afford to fail a couple more times. Now there's the question of would they need all that space in a cloud setup, but thats another story.
What concerns me, from the image, is that they seem to be using cheaper, consumer grade drives (Samsung 860 PROs I assume). This is asking for trouble - as they went through - 3 drives failed simultaneously. I've only ever dealt with Cloud infra, but even I know that drives that are bought together, tend to fail together. It's likely their new batch of drives will also fail simultaneously - backblaze has done a ton of research on drive longevity.
> drives that are bought together, tend to fail together
It's exactly the same as the lights in your house, all of these were made at the same batch with the same quality of material and used the same way, they tend to fall apart within a few days from each other.
What wonders me most is the fact that 44k $ is a great economical loss for them: i would expect CCC to earn at least a milion/year so that 44k $ would be nothing more than a little annoyance
$1M/year seems high, but even if that's right, $44k isn't small. That's most (or maybe all) of someone's annual salary. There only seem to be 7 people associated with the company [0], so that's a big loss.
They said they had backups, but it had been long enough since the last backup they lost enough data for a ~$30k recovery to be worthwhile?
This suggests whatever backups they had might be months old. Given that they are using a roll your own storage solution with consumer grade hardware, I doubt they are rolling in revenue.
The second issue, is that since this was setup in a RAID I'm assuming that all of the drives were purchased at the same time when it was originally setup which means depending on the supplier you are probably going to receive drives from the same batch of production. Because of striping with RAID if you have some fault in a batch and multiple drives from the same batch then you can have failures on multiple drives that occur at approximately the same time.
The simplest solution is of course to buy enterprise grade drives but the obvious issue there is the price is much higher. Especially if you are going to be purchasing the largest drives as you always pay a premium for max capacity drives and the price jump on large drives is not insignificant.
When we first built out our SSD cloud for DigitalOcean at the end of 2012 we went with consumer drives because we were bootstrapped and SSDs were also significantly more expensive so we couldn't justify the price jump without knowing if we had product market fit.
But rather quickly we ran into a whole host of issues. Another common issue is that performance can degrade when consumer drives get to near full disk capacity so you can't really use all of the available space if you plan to do a significant number of reads and writes, which is pretty much what a database server is designed for.
That was the first upgrade we did after raising money, which is switching to enterprise grade drives, unfortunately you have to pay the premium but it really reduces problems significantly.