Did you use CloudFormation to crate the infra? If not, I'd love to hear some details on how you did this. Any API Gateway being used? Seems to be offline at the moment.
We run similarly on CloudFormation. Interesting with the ‘aws s3 cp’ command. I started using ‘aws s3 sync —delete’ nowadays after having issues with pre cleanup required for ‘cp’.
In my case since the assets have fingerprinted filenames and index.html references them, if I delete old files with each deployment, then a user who has a cached version of index.html will see a broken page. So I just leave old asset files as is.