I guess taking some risks as I am not really authorized to use social media, but at AWS there is a sauron like focus on not letting internal engineers view customer data. On my service we don't even have tools to do it and the security controls are very tight and getting tighter all the time. It's a big fucking deal, if not the biggest fucking deal, besides KTLO.
Posting to get ahead of some of the comment here on security posture.
The first premise when using a vendor is trust. If you don't trust them then don't use them. So far, AWS has proven to be trustworthy in my opinion. From what little I've seen about their operations they seem to give a damn and you have to assume that iceberg goes deep.
If you don't trust them, don't use them. Provide evidence to your leadership of malicious intent and provide an alternative and they'll back you.
Executives make decisions based on a small amount of evidence and go with their gut from there. So far AWS has a good reputation but if they're not careful the mood could change quickly. I don't think something like this shows any malice and I have to just assume any cloud provider _can_ access all of my data but chooses not to in order to preserve their reputation which ultimately generates more revenue long term than any single piece of data.
This seems very binary / black-and-white to me - it’s not “you either trust them or don’t”. I may trust AWS to keep the cloud running, but I may not want to trust all their stuff with access to my private data.
If a provider puts themselves in a position that they’re entirely unable to access my data, or it being extremely difficult, that would actually increase my trust in them.
If having all customer’s private S3 data being accessible by all AWS support is happening due to a wrong checkbox being selected, it definitely hints at a bigger problem and erodes my trust in that vendor.
And how they respond to this incident could restore that trust, or continue to erode it even further.
> If a provider puts themselves in a position that they’re entirely unable to access my data, or it being extremely difficult, that would actually increase my trust in them.
This is never possible if you're using KMS / server side encryption or no encryption at all. Your data on these services is always visible if they were to try to read it, whether that be forging KMS requests to decrypt data or passively snapshotting VM memory for inspection. Technically this might be solved via recent datacenter CPU security improvements but there will always be flaws and 0-days that people with money and a will can use to bypass these protections.
Use rclone with encryption for S3 and consider anything else potentially visible.
Of course, that’s why I said “extremely difficult” — it’s never completely impossible. I was thinking about Apple, who made iCloud reasonably e2e encrypted: of course, they can still push malicious updates targeting specific devices, but that needs to be a deliberate action by the organization, not a single rogue employee.
It’s not entirely unlikely that AWS was hacked / socially engineered in order to get this privilege in there, perhaps because some specific s3 buckets were being targeted.
As an organization, you need to have a high enough level of protection that these things are just not possible.
So maybe this whole idea of managed IAM policies being installed in all customers’ accounts is just a fundamentally bad idea, and should just be given on a case by case basis.
While I recognize that's the ideal goal, and most appropriate, I think there may be gaps in practice. My team just discovered yesterday that a Kinesis Firehose Stream can still deliver to an S3 bucket that lacks a bucket policy and whose ACLs disable any access to it at all. That diminished my confidence a little bit that all teams are perfectly in compliance of the overall goal.
Kinesis Firehose uses an IAM role to deliver data, so delivery within the same account does not necessarily depend on permissions on the bucket. Removing s3:* permissions from that IAM role or adding an explicit deny statement to the bucket policy would stop the flow of data.
I thought this was true at the engineering level, but "authorized" cross customer data sharing- where there is some debate at what's customer data vs what's platform data- at the business level seems...rampant? Just curious for more perspective on posture.
When I was at Facebook (2018-2020) there were in fact few permission barriers (click through the screen warning you to follow the rules, and that's at the UI level not to mention pulling stuff directly from Tao through the old PHP interfaces still laying around, or just building a user context from scratch), but plenty of auditing and automated detection.
Posting to get ahead of some of the comment here on security posture.