I have never thought of that, ever. I'm not being sarcastic. Usually I think of email as existing on some secure server, I never thought that backups are kept, maybe in a different medium which is then open to vulnerabilities.
This is why, for example, Amazon S3 has a little checkbox on each bucket to encrypt the contents of the bucket.
At first glance this might seem a little silly. Amazon has the key. (You don't even get to see the key yourself.) So Amazon can read all your data. And every time you read from the bucket it's automatically decrypted, so the encryption won't protect you from anyone who has somehow achieved permission to read your data.
But that's not the point. The point is to protect against attacks like "I found this pile of dusty drives stacked in the maintenance closet at Amazon," or "I went digging in the local dump near an Amazon data center and unearthed this hard drive, and look what I found backed up on it."
Amazon S3 now supports server side encryption with customer-provided keys (SSE-C), a new encryption option for Amazon S3. When using SSE-C, Amazon S3 encrypts your objects with the custom encryption keys that you provide. Since Amazon S3 performs the encryption for you, you get the benefits of using your encryption keys without the cost of writing or executing your own encryption code.
Yeah, I go back and forth on that one. The problem is that the key management is crucial. Slip up and lose your key and your data is toast. Obviously this also applies to Amazon, but they have more resources, better incentives, and one million times the operational experience with their key management system than I'll ever have with mine. Frankly, my clients should bet on them over me.
Now, one good idea might be to redundantly back-up Amazon's backups at some other host, using GPG to encrypt those. This ensures against Amazon encryption errors, billing errors, mistyped legal injunctions, Jeff Bezos declaring you his personal enemy, et cetera.
This may be obvious, but rolling my own at-rest encryption is not going to significantly protect my data from an attacker who works inside Amazon, nor from an attacker who roots my instance. If the keys are in the cloud, the keys are in the cloud.
UPDATE: oh, yeah, I forgot the use case where you are writing to s3 from outside Amazon's datacenter over HTTPS. Okay, that is a much stronger case for GPG in advance. It wouldn't matter if we assumed that TLS always worked. But this is TLS. Does your upload client check the certificate chain? So many of them do not. I sure hope Amazon's CLI client does.....
I'm not even giving a hypothetical scenario. It's no longer fresh in the minds of most people but years ago Reddit used to store their passwords plain text for the convenience of being able to send them to the user, instead of implementing a password reset mechanism. One day their hosts backup tapes were stolen out of the back of a truck. Now they salt+hash their passwords.
It is frighteningly common to neglect backups when thinking in security. I have seen a couple of examples where the technical aspects of security with good firewalls, good access control to servers and so on were in place, but backup media could still be found laying around for everyone to grab.
