I once read that Facebook was opening 2 or 3 massive new data-centers in the US for the purpose of hosting stale content.
You may have posted a photo 7 years ago, and statistics show that basically nobody ever revisits it. However, in case you do, it needs to be there. So these enormous buildings do basically nothing, but still need to be there.
It makes me wonder how it can go on like this. Users only keep adding content and never remove it. The income per user cannot grow forever, storage cannot get infinitely cheap, the model has to break one day?
There's no meaningful benefit to dedicating any amount of DC equipment just to stale content. Those are spindles (and networks) that could be taking meaningful hot reads and writes, and colocating stale and hot data is generally a better use of capacity than concentrating hot data in fewer locations.
Exactly. And that doesn't even take into account higher res photos and 4K video. I remember the staggering statistic where just Instagram sees 100 million photos added per day, every day. And that was years ago.
That is just Instagram, with photos. Imagine Youtube.
And again I have been saying this since ~2015/16, we dont have any meaningful roadmap for cost reduction on storage, whether that is Hot as in NAND, Bulk as in HDD, or Cold as in Optical Disc. I dont see 2TB SSD dropping below $100 in next 5 years, or 10TB HDD below $120.
Remember when Google promise infinite Gmail storage?
> we dont have any meaningful roadmap for cost reduction on storage
Then maybe we could:
- stop encouraging users to post shit just so we can track them
- stop tracking them which requires many data points and a lot of processing power (for 0 benefit for the user or society at large)
- stop the copyright non-sense and actually use hyperlinks instead of reuploading the same content 500 times across platforms? maybe even do content-addressed storage (Bittorrent/IPFS) who knows?
Instagram does not guarantee photo quality. They can resize photos anytime they want. Eventually they turn 10 year old photos that have not been viewed in 9 years into thumbnails or just delete them.
The amount of ephemeral content (stories, expiring content) being created/shared has a larger impact on capacity and provisioning than you might expect.
This isn't how scaling works though. Across all applications the hot data growth outpaces the cold.
So if you're designing capacity for exponential growth, the future point at which you stop experiencing exponential growth and only have to worry about roughly linear growth is a much easier problem to solve.
You may have posted a photo 7 years ago, and statistics show that basically nobody ever revisits it. However, in case you do, it needs to be there. So these enormous buildings do basically nothing, but still need to be there.
It makes me wonder how it can go on like this. Users only keep adding content and never remove it. The income per user cannot grow forever, storage cannot get infinitely cheap, the model has to break one day?