In 2012 I was working at the Internet Archive. I remember the 2011 Thailand floods restricted the supply of hard disks and pushed up prices.
The best deal we could find was buying these Seagate hard disks as external drives at retail from Costco. We picked up a truck load. Then had somebody with a screw driver remove them all from the enclosures and install them in new servers.
The error rate was high, the drives didn't last long in production.
I did precisely this too, as I needed storage for a few thousand hours of video footage I’d shot for a documentary, needed it now, and nobody had anything other than external HDDs in any kind of useful size.
I was really careful - I had two NASes, each with raid 5 volumes holding a full copy of the footage, as it was irreplaceable.
I got up one morning to find a disc had failed in one of them. Swapped it. During the restripe, another failed. Oh well, I have the other NAS, I thought, and after replacing the busted disks and reformatting the failed one, started copying the data over again - and then a disk failed in the backup nas. And then another. And then another in the previously failed one. And that was that. Five disks, all dead, within a day of each other, all the same model. The WD drives in them were all absolutely fine.
I haven’t bought a seagate drive since, and never will again.
That's the reason I never put the same brands/models in RAID. It is logistically a bit more complicated, but I've often noticed failures like that really close in time.
Ye copying sick HDs always make me nervous. It is like taking your really old and scrappy car for a last ride to the salvage yard going over the whole country.
Hah, I remember a pallet or the now-empty IA enclosures showing up at Noisebridge. They were super useful - a USB3-SATA adapter. I still have mine for occasional data spelunking.
I don't know if Backblaze's usage predates it becoming the common way to describe this practice, but it's definitely caught on. For example see any deal post on an external drive on reddit /r/datahoarder
I worked at an ISP in 1996, where we would buy external Boca 28.8k modems in bulk from retailers and take the boards out to put them in our racks because they were cheaper - yet identical - to the "ISP-grade" versions that Boca sold as just boards.
Back then we referred to it as shucking the modems, so it's definitely not a new term in IT/computing (never mind the actual origin of the word).
The plausibility of a conspiracy theory diminishes as the number of people necessary to keep the secret increases.
How many people work in Thailand's hard drive manufacturing plants? Hundreds? Thousands? How many people in the logistics channels? How many people are vending food to these employees? How many people in the US Government? Oh, and pulling this off would also require co-ordination with Thai Government. That's hundreds of people right there.
At the time it was kind of an open secret in the industry. The floods were a convenient explanation, but a very large order from a client with a confidentiality clause ( very common in the industry) was widely known to be the main reason for short supply. It is possible that this client was someone else other than the NAS er... NSA but it is not very likely and at the time it was conventional wisdom among insiders that the "big spin" was the cause. There were even jokes about how all those drives spinning had to be oriented si that the precession wouldn’t slow down the earths rotation etc.
Just because a customer orders a few gazillion of product x and wants to keep it quiet doesn’t make some kind of nutty conspiracy theory. It happens all the time. The Thailand floods drew down production even more, but without the giant order, there wouldn’t have been the same impact.
Fun fact, any time two or more people want to keep something on the DL , it’s a conspiracy. Nearly everything that an intelligence agency does, is de facto, a conspiracy.
One of my favorite memes (in the sense of memetics) is the meme "conspiracy theorist". This phrase was weaponized by the CIA as a memetic self-defense mechanism. (Their own documents admit this!) But if you tell someone this, the meme itself is triggered in their mind, and their instinct is to not believe you, because you sound like a conspiracy theorist.
>This phrase was weaponized by the CIA as a memetic self-defense mechanism. (Their own documents admit this!)
That seems... totally expected? The problem comes when you're trying to use it as proof of something. Some conspiracy theories do turn out to be true, but that doesn't mean every conspiracy is true.
> but a very large order from a client with a confidentiality clause ( very common in the industry) was widely known to be the main reason for short supply.
So, just for the fun of it: Forbes estimates a capacity (at the time) of up to 12 EB [1]. Assuming they used 3TB drives, they'd need 4 million drives for this [2]. This article [3] shows the market for consumer drives at about 150 million in 2013 (figure 7). So the NSA order would've been in the ballpark of 2% of consumer drives sales - insanely large at the face of it, yes, but probably not enough to cause such a shortage.
I know of people who were delivering software to certain customers that were required/intended to work on EB scale datasets over a decade ago, and this was well before the Utah data center was a thing.
I have a feeling that just comparing the numbers without looking at any of the logistics is just a convenient way to dismiss a theory you don't like.
There's all kinds of things to think about like how 2% may be less than 100% but in an elastic supply of a globally distributed good, 2% of (total yearly) supply called upon all at once can be a _huge_ departure from the norm. Maybe the shortage isn't just a hard disk shortage but a shortage of shipping space to take those hard drives anywhere else but to the NSA. Maybe it's that the factory(s) don't make 100% of the drives in an instant and then just distribute them for the rest of the year, but slowly make a constant stream of drives, all of which were swallowed up for a period of time by the massive order, causing the shortage.
There's a million other factors besides just "this number smaller that this number".
Cant edit on phone but we can also consider the size of the drives. We can only guesstimate what sizes the NSA wanted but to your source we can say they needed 12 EB. Out of 150 million drives, there might not have been 4 million 3TB drives out there, this was 2014, maybe there was but they didn't want all the exact same drive, if you calculate that some of the drives are 2TB or 1TB the estimated numbers only get larger. I think it's totally reasonable that a single order of 12,000,000TB to a single location in Utah could grind the widespread consumer market to a halt for a bit, especially if they're almost all made in one place, which it seems most of this discussion is taking for granted.
It looks like you've picked only the most favorable statistics and ignored everything else in formulating your opinion/response.
> I have a feeling that just comparing the numbers without looking at any of the logistics is just a convenient way to dismiss a theory you don't like.
I really have no feelings for this theory either way, I simply don't think it's realistic.
> There's all kinds of things to think about like how 2% may be less than 100% but in an elastic supply of a globally distributed good, 2% of (total yearly) supply called upon all at once can be a _huge_ departure from the norm.
I generally agree with that. However, for one, we're talking about consumer goods within a growing marked - a demand jump of 2% should be within expected deviation and not generate the problems we've seen there.
> Maybe the shortage isn't just a hard disk shortage but a shortage of shipping space to take those hard drives anywhere else but to the NSA.
3,5" hard drives have a volume of ~ 390.000 mm^3. Let's say that's 3.000.000 mm^3 shipped. That would allow ~10.854 drives in a standard container, for a total of 370 (assuming 4 million drives). A large container ship can pack 850,000 containers. So yes, it's a large volume, but for global shipping it's a drop in the bucket.
> Maybe it's that the factory(s) don't make 100% of the drives in an instant and then just distribute them for the rest of the year, but slowly make a constant stream of drives, all of which were swallowed up for a period of time by the massive order, causing the shortage.
Sure they do. But, again: If we're assuming 2% of total drives, it would be the output for one week. Not ideal, but not a major shortage. And that's the worst-case scenario: Orders in the billions of dollars don't usually work in the time frame of one week; it's quite likely that the manufacturers knew of this well in advance and were able to upscale/order accordingly.
> Out of 150 million drives, there might not have been 4 million 3TB drives out there, this was 2014, maybe there was but they didn't want all the exact same drive, if you calculate that some of the drives are 2TB or 1TB the estimated numbers only get larger.
Sure. But 4TB drives and probably 6TB were also available at the time and it's reasonable that for extremely large storage, they'd shoot for larger drives - if only for space and heat efficiency. There surely were a few smaller drives as cache etc. here and there, but I'd presume the lions share to be large drives.
> I think it's totally reasonable that a single order of 12,000,000TB to a single location in Utah could grind the widespread consumer market to a halt for a bit,
I won't disagree with that it surely did impact the market a bit, but I highly doubt it was a major cause for the shortage. Especially when there's a perfectly fine explanation of a tsunami hitting where most drives are manufactured.
> especially if they're almost all made in one place, which it seems most of this discussion is taking for granted.
Most likely they'd order from multiple manufacturers, alone for capacity and failure probability reasons. In fact, if they actually ordered in one place, you could easily get evidence for this theory by showing the problems originating from one specific manufacturer.
> It looks like you've picked only the most favorable statistics and ignored everything else in formulating your opinion/response.
I highly disagree with that. I picked the largest estimate of the data center size and the capacity commonly available to consumers at the time; additionally, I only used a (low!) estimate for the size of the consumer market and completely disregarded the enterprise one. If anything, I probably overestimated the impact.
A said above, the massive buying surely did not help the shortage and I dislike the NSA as much as the next guy. But unless my estimation is off by an order of magnitude the numbers simply don't add up - even assuming the ordered with a delivery date of yesterday! - and given that there's a very reasonable alternative explanation for the shortage, I don't think this is a sound theory.
All big manufacturers bought whatever supplies they could get their hands on as soon as the impact of the floods was clear.
Companies are uniquely positioned to minimise risk in their ability to execute.
I remember this period well and aware that a number of large players basically bought up whatever supply was available to cover their needs in the short to medium term. The result was that retail pricing more or less doubled (if you could even get drives).
> The plausibility of a conspiracy theory diminishes as the number of people necessary to keep the secret increases.
Not if you can discredit those who dare to speak out by labeling them "conspiracy theorists," thereby discouraging others from coming out as well.
I remember an episode from an old American sitcom where people make fun of the dad (an airline pilot) when he claims to have seen a UFO. It is the same thing. Any deviation from the standard narrative isn't tolerated by most people.
... but aren't the observations itself part of the standard narrative, and the nonstandard part is how people explain them? (One standard explanation being that they are just some next-level high-tech stuff being tested.)
> but aren't the observations itself part of the standard narrative
Only in a technical sense.
When people talk about UFOs, they are not referring to known things like weather balloons, planes or fighter jets, but, instead, to something out of the ordinary. However, every "standard" explanation offered is one of these things.
I assume you mean UFOs as in "alien flying saucers".
I think the rationally correct belief is that we simply have no evidence of aliens at this time, and since we virtually know absolutely nothing about what pilots saw (all we know is that there are these observations), we just fall back to our priors (base assumptions). Sure, it might be aliens. And hence the ancient aliens meme. It might all be secret military stuff. (Or the mix of the two, hence the storm Area 51 meme.)
And in this case I don't see what's wrong with the standard narrative.
If there are truly aliens flying regularly in our atmosphere we should strap a few 4K recorders on regular airline planes, and we just have to wait.
Not to mention the increasing number of space launches and spacecraft up there. And increasing number of downward looking observatories.
You can come up with any number of scifi-ish explanations: aliens, humans from the future, lost/forgotten civilizations. But it could also be someone testing an uncommon craft.
Frankly, we don't know, and one of the reasons is, I believe, people are hesitant to take a serious look at it (or even talk about it) because they don't want to be classified as kooks.
You overthink this. The flooding might not have led to a crisis alone if the NSA or any other big buyer didn't place a huge order at that time. You also don't need a conspiracy to keep the procurements of a government agency confidential.
Because of Big Santa (ie. parents). And that promptly fails when the cost of maintaining the Santa system gets too big (ie. as the child's mental faculties develop, plus as the child interacts with the outside world not controlled by the parents).
So unless we all are living in a very well orchestrated NSA Truman show, it's unlikely that your argument applies.
That doesn't matter. Millions of adults orchestrate a secret conspiracy without any direct coordination even. The faculties of the target will only change the coordination and sophistication required.
If everything in the world were an illusion, I doubt that any group would be able to orchestrate that for all of us. But large groups can coordinate secret happenings, and they do, all the time.
The difference is whether the entire organisation would have to know or not. "Actually, our HDD factory is operating" is both a thing that lots of employees would necessarily know (unlike CIA assignments in Canada), and also those employees would not be as trusted as high level intelligence agency operators. Particularly for low level employees, like contracted cleaners or whatever "Don't tell anyone the factory is operating" seems like it would be a great way to get the information out there that it actually is.
You tell the employees that they are working for a HDD factory. A small number of people at the top of the company are in on it and keep sales limited to one or a small number of clients, all of whom are the clandestine buyer.
That would be similar to the way in which the fake Swiss cryptographic machine manufacturer was run. Their only leak risks were from employees who realized that their cryptosystem was easily broken. An HDD manufacturer would not have that problem because all of its work would be perfectly legitimate.
Perhaps you should read Hogfather or Making Money by Terry Pratchett. In which the importance of the lies that society tells itself is touched upon.
"All right," said Susan. "I'm not stupid. You're saying humans need... fantasies to make life bearable."
REALLY? AS IF IT WAS SOME KIND OF PINK PILL? NO. HUMANS NEED FANTASY TO BE HUMAN. TO BE THE PLACE WHERE THE FALLING ANGEL MEETS THE RISING APE.
"Tooth fairies? Hogfathers? Little—"
YES. AS PRACTICE. YOU HAVE TO START OUT LEARNING TO BELIEVE THE LITTLE LIES.
"So we can believe the big ones?"
YES. JUSTICE. MERCY. DUTY. THAT SORT OF THING.
"They're not the same at all!"
YOU THINK SO? THEN TAKE THE UNIVERSE AND GRIND IT DOWN TO THE FINEST POWDER AND SIEVE IT THROUGH THE FINEST SIEVE AND THEN SHOW ME ONE ATOM OF JUSTICE, ONE MOLECULE OF MERCY. AND YET—Death waved a hand. AND YET YOU ACT AS IF THERE IS SOME IDEAL ORDER IN THE WORLD, AS IF THERE IS SOME...SOME RIGHTNESS IN THE UNIVERSE BY WHICH IT MAY BE JUDGED.
"Yes, but people have got to believe that, or what's the point—"
I'd still recommend the book though. There are so many things that are little more than flaky veneers taken for granted in adult human life that the fact anything works at all generally comes down to an intentioned group of people 'conspiring' to make it so, and no one ever bothers to ask about it, as it 'just werkz' and creates value. It's only a problem when too many externalities get ignored, in which case, everyone starts to notice.
Once you understand this, the beauty of the 'conspiracy theorist' meme coming out of an intelligence agency to get people to stop taking those who ask inconvenient questions starts to take on a stroke of genius. You get people who rock the boat discredited out the gate, and reinforce the status quo.
...Until that document gets declassified and blows up in your face anyway.
But where would a person speak to disclose the secret? Internet is very effectively censored by the US monopolies now.
If something is not indexed by Google, it's effectively non-existent. FAANG censorship is fast and effective. E.g. there were many important US bureaucrats involved in the Theranos scandal (even current US president). When I try to find anything related to that matter now, Google returns pages with many links removed by censors. It's almost impossible to find people responsible for withdrawing money from US budget via Theranos.
Also, assuming that 3 letter agencies were involved in the operation, any whistleblower would understand that destiny of Epstein will await her/him in near future.
Also, there's no "independent" journalism, all media belong to few influential entities, there's only approved articles appear in newspapers, magazines, on TV and YouTube.
Literally anywhere, the internet isn't consored anywhere near as much as you think.
If someone posted a plausible/verifiable account of what happened in a HN thread, then it would've made it halfway round the world before the NSA could take it down
One thing I learned recently, which made the effect of the 2011 flood on the HD market worst: Aramco purchased a big chunk of several months of the world supply of hard drive to recover from Shamoon in 2012, by replacing every HD in the company [1]
Or they managed to fill it with hardware without having to coordinate an intricate conspiracy involving supply chain and manufacturers across the globe.
Nobody is alleging that but you. Personally I don't think either "surveillance system needs a way to store what is collected" or "high-volume customers have economic leverage to demand confidentiality" are particularly intricate concepts.
Colordrops said, "The Thailand story was a cover." This is the allegation. Not that the NSA bought a lot of drives and kept it secret, but that this purchase alone rather than the floods were responsible for the drive storage, and the flood story was fabricated. That is the alleged conspiracy.
I find that a number of conspiracy theories are 'debunked' by creating an obviously ridiculous strawman/rendition of the theory that nobody actually claimed.
The Snowden effect has worn off, it seems. Nowadays the NSA is back to being able to pull the most extreme stunts and everyone who thinks it is even remotely possible is called a crazy conspiracy theorist.
To claim that the NSA caused or faked the Thailand flooding is conspiracy theorist bullshit of the highest order.
To claim that the NSA took advantage of an existing event to help their already-planned ordering fly under the radar more easily is perfectly plausible. And, honestly, pretty boring.
Which one are you alleging? Choosing your words carefully might be wise.
The NSA has no shortage of front companies to do such jobs. For instance, when the CIA needed to place torture experts in Brazil so they better could support the Brazilian military dictatorship, companies like Ford and Coca Cola were more than happy to offer them management jobs.
Good thing NSA et al aren't also competing for parallel-computing hardware or it would be even harder to buy a new GPU. Luckily all participants in online discussion forums are real humans who can tell me what to blame :)
Because it doesn’t sound like one of those crazy conspiracy theories. They need to feel privy to some knowledge “the others” don’t have and refuse to see.
First time I hear this but I think timelines match up and at this point I think it is completely believable. (Not saying I'm sure it is actually true, only that it won't surprise me more than if it is untrue.
The best deal we could find was buying these Seagate hard disks as external drives at retail from Costco. We picked up a truck load. Then had somebody with a screw driver remove them all from the enclosures and install them in new servers.
The error rate was high, the drives didn't last long in production.