On: Moving 25TB data from one S3 bucket to another too...

[awiesenhofer]

I had a similar task once, tried it with awscli and s3cmd first, then took about 5 minutes to google for a multithreaded alternative and found s4cmd[1], problem solved. (tmux to keep the session running)

Makes me kinda doubt at least parts of this story - not one of these 7 engineers thought about googling?

Edit to add: Agree with the basic premise though - s3 can be real slow when when you actually need to work with your data.

[1] https://github.com/bloomreach/s4cmd

[gfody → awiesenhofer]

or even faster https://github.com/peak/s5cmd

[miniman1337 → gfody]

s5cmd is super fast