From my experience if you have a bucket with billions of objects it will still take a few days after the objects are deleted by the lifecycle rules to be able to delete the bucket. The objects are non-recoverable but I believe due to the eventual consistency nature of S3 you can't delete the bucket until the objects are all fully deleted.
Yeah, S3 is frankly a disaster. People use it like a filesystem and if you have billions of objects, it very quickly becomes a nightmare to manage. Size of data is a complete non-issue, it's all about the number of objects.
I will point out deleting that much data from harddrives isn't a fast operation either if you were storing it on drive. Same for tape.
Sure you can destructively destroy the drives. But you could also just use KMS on the S3 bucket object and then toss the key and close the buckets. Gonezo.
It's strange that people just shrug their shoulders and accept this state of things in 2020.
Any system these days should allow near-instantaneous admin operations for just about any reasonable request. Subtree move, delete, or rename should all be instant. Similarly, any simple ACL changes should be instant.
Similarly, data movement should scale linearly with the volume of data, not the number of objects.
I'm definitely one of those who treat it like a filesystem :') . I'd love to hear a different point of view. Would you feel ok to share thoughts on how I could be thinking of s3 to fully appreciate the right way of using it?
That entirely depends on what you're using it for. Mimicking a typical block-level filesystem with object storage will lead to problems beyond a few thousand objects or shallow "folder" levels.
Object storage is best for larger files with bulk updates. Either write new files entirely (best for S3) or append/replace where supported (like Azure/GCP). Any further structure and metadata should be maintained at a higher level with pointers to the raw files in the object store.
There are products like Stablebit Cloud Drive [1], ObjectiveFS [2], and other storage companies that handle all this for you. You can also look UtahFS [3] to see open-source code on building something yourself.
Then they could set a lifecycle rule, wait a day, and it would be gone for real.