It makes backups PITA. I migrated blobs to S3 and managed to improve backups from once a month to once a day. Database is now very slim. HDD space is no longer an issue. Can delete code which serves files. Lots of improvements with no downsides so far.
Yes, S3 is extremely reliable and versioning protects against “oopsies.” I do always recommend disallowing s3:DeleteObjectVersion for all IAM roles, and/or as a bucket policy; manage old versions twith a lifecycle policy instead. This will protect against programmatic deletion by anybody except the root account.
I second these sensible protections. Also would recommend replicating your bucket as a DR measure. Ideally to another (infrequently used and separately secured) account. Even better if you do it to another region but I think another account is more important since the chances of your account getting owned is higher than amazon suffering a total region failure.