Hacker News new | past | comments | ask | show | jobs | submit login

Turn on weekly manifest to get a list of files and then you can delete based off that. Much faster and better then listing billions of files.



What’s a weekly manifest?


I believe ludjer is referring to S3 Storage Inventory. This is a daily, or weekly, file produced containing metadata on every file within a S3 Bucket. It does not use the synchronous List APIs.

> You can use Amazon S3 Inventory to help manage your storage. For example, you can use it to audit and report on the replication and encryption status of your objects for business, compliance, and regulatory needs. You can also simplify and speed up business workflows and big data jobs by using Amazon S3 Inventory, which provides a scheduled alternative to the Amazon S3 synchronous List API operations. Amazon S3 Inventory does not use the List API operations to audit your objects and does not affect the request rate of your bucket.

>

> Amazon S3 Inventory provides comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or objects with a shared prefix (that is, objects that have names that begin with a common string). If you set up a weekly inventory, a report is generated every Sunday (UTC time zone) after the initial report. For information about Amazon S3 Inventory pricing, see Amazon S3 pricing.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag...




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: