Hacker News new | past | comments | ask | show | jobs | submit login
Defense Contractor Left Sensitive Pentagon Files on Unprotected AWS S3 Bucket (gizmodo.com)
98 points by danso on May 31, 2017 | hide | past | favorite | 50 comments



Retired as govt engineer. Reported a security violation, employees sharing passwords/account on a secured network. The 2 contractor ITs claimed to know nothing about it. At most only a handful of employees have access to the network and at most, but rarely 3 work on the network at the same time. If the contractor ITs were not aware, and they never are more that a few feet of the activity, they should have been fired. Management moved me to an isolated area and began the slow process of retaliation, so as not to look like retaliation. To their surprise, I walked away from my job. BTW, the supervisor I was reassigned too, told me not to report any more observations. I am not a fan of contractor support, and I see no reason to use such infrastructure such as Amazon to store any data, especially sensitive data.


The US is seriously blundering by using contractors for military operations, and instead needs to hire more military personnel and civilian employees. This fixation on not having more government staff (and on funneling money to private sector contractors) is endangering the US government's operation.

In-house staff aren't a panacea, but in-housing critical capabilities has always been the prudent decision.

Of course, much the same can be said about contractors and off-shoring in the private sector as well.

At this point, the US is cannibalizing its future for short-term gains. Of course, the people making the decisions generally won't bear the costs, and have convinced much of the populace that such a strategy will benefit them through wishes and pixie dust.


Nothing changes until we can reduce money's influence over the election process. Until then, any special interest with deep enough pockets can and will have US taxpayers over a barrel.


Nothing changes until the gov pays competitive salaries.


Original commenter is correct. This has little to do with salaries. The political power and money goes toward a government that sends tax dollars at high margins to contractors who don't even earn it in many cases. There's no major liability for malice so long as it affects taxpayers instead of those in power. The defense contractors continue to cut anything nonessential to their mission of turning tax dollars into profit for the people on top. And vicious circle continues.

Here's a nice write-up on the bribery and abuses of just one:

http://www.vanityfair.com/news/2007/03/spyagency200703

The others pull the same shit. The biggest trick is the revolving door of moving people back and forth into the private sector with them benefiting while taxpayers lose.

https://www.opensecrets.org/revolving/

And note before any counterpoint that a non-corrupt system might pay better wages in the event you're right. Most of the wages are rigged to go up and to bullshit that benefits people I'm describing. That could easily go to better tech. For instance, the first time the private market started doing strong security was a small initiative by the U.S. government mandating it with purchases promised in return. Led to the strongest systems of the time that resisted NSA pentesting. Although NSA killed that (incompetence or malice), it could be done again with a different group.

http://lukemuehlhauser.com/wp-content/uploads/Bell-Looking-B...


You might be right.

Most people who got into gov. gigs know this but the upside always been a cushy retirement, sweet pension and a Cadillac health care plan in retirement.

"Unfunded government liabilities" are now always on the chopping block and those people that took those low paying jobs for a better long term payout in the end are now getting screwed, royally. Government jobs are no longer the sweet gig people think they are and I think they have to start being more competitive with the larger markets.


This is one of the root problems in India too. At the end of the day, in a Govt job, it's just:

- a lot of bureaucracy

- and the mandatory ass-licking

- working on ancient platforms and techs

- and a lot less pay compared to the market (so either you do under the table deals, or well..)

Very few tier one engineers go for Govt jobs and almost no tier one engineers from tier one college go for the Govt jobs.

Top it wtih maddeningly absurd and archaic interview process and selection criteria.


Exactly.


For those wondering how this can happen - AWS' command line tools and SDKs by default upload files as private. Scarily, some other libraries set uploads to public by default, for example: https://www.smore.com/labs/tinys3/

For those who say cloud isn't less secure than on-prem... that may be true. However, the cloud is generally just one incorrect click or setting away from being completely open.

Anyone know how the security researcher found the public bucket? Does he just crawl for buckets made up of common words? Does AWS not limit the amount of crawling from one IP?


>For those who say cloud isn't less secure than on-prem...

I mean that's fundamentally untrue, simply from a trust, venn-diagram point of view. With cloud you have to assume an additional trust with a third party that you don't with on-prem.

Is this difference meaningful? that's arguable. But no matter which way it's argued it's still fundamentally true that in direct comparison between a cloud-hosted and self-hosted system, the cloud-hosted version inherently requires that a higher level of trust.


I agree with OP, but this is not true. You can trust Amazon to be better at security than you are. In essence, depending on how (in)competent you are, cloud allows you to convert money into security.

If I am a complete zero when it comes to securing my home, and I get a professional contractor to do it. Is that "inherently less secure" because I have to trust him, now, too? Not necessarily, if he does a much better job than I would have done. It can make up for that extra trust.

If you are as competent, or more, than Amazon: sure. You are right. But not all of us are so lucky :)


>You can trust Amazon to be better at security than you are.

You can't blindly trust them to be that, rather you determine them to be that. On an individual provider level you can definitely assess their capabilities and make that determination. But any attempt to blanket extend that to 'cloud providers' is an absurdity.

There is no common element among 'cloud providers' that makes one provider's competence at security extend to any other provider under that same banner. Yet today common groupthink is 'cloud hosting is more secure than self hosting'. Nonsense.

A simple reword reveals the absurdity: 'Allowing other people to run your IT systems is more secure than running it yourself'. The buzzword 'cloud' somehow turned a nonsense into a widely parroted myth.


I don't see how that's absurd. There is absolutely no way I could run a system myself that's more secure than what I could get by paying someone else to do it.


Again, a blanket statement that can't possibly be true.

What system are we talking about? What third party are you talking about that you're gonna hand money over to do this on your behalf? These things matter, and there are many answers to those those questions where yes, you could indeed deploy and run a system yourself and be more secure than a provider, because for what you say to be true, your relying on a false belief that because someone offers a service (whether for money or not), that inherently makes them more competent than you to operate and secure that service.

There are cloud service providers that are able to offer a more secure base service than 99% of it's consumers would be able to create independently. I am not challenging that assertion. I'm challenging the commonly-expressed belief that individual cases where that is true in any way extends to the blanket claim being made in the parent. That is 'Some cloud providers are more secure than their customers therefore all cloud providers are more secure than all their customers'. No. Absurd.

This is something that can be proved for individual suppliers and with suitable assessment/verification processes. To extend that to 'cloud providers' as a whole is absurd. If you can't see that then fair enough, I truly wish you luck because that's ultimately all you'll be relying on if you choose to play in this space.

When you say 'There is absolutely no way' what you must surely mean is 'Under certain circumstances it is the case that'.

I remind you of one simple example: Dropbox ran for a significant period of time with a system that allowed any person to log into any dropbox account, with any password.


> You can trust Amazon to be better at security than you are.

Unless you have more access to Amazon internals than everybody else, then you can't really know that for sure. Even if you did, it's only true if you've decided to let it be that way. There's nothing stopping a company from hiring great security people, and implementing their own great security.

Also, there are many scenarios, like a defense contractor handling sensitive information, where I would expect the the in-house (or in-government, I guess) security to be a lot better than Amazon's.


> There's nothing stopping a company from hiring great security people, and implementing their own great security.

Money.

We can afford to pay Amazon for their services and reap the benefits of their security investment and experience as part of the purchase.

We can't afford to hire someone to be dedicated to security.

If you're a small business, and/or security isn't one of the core components of your business, chances are you'll make better use of your security 'budget' (or lack thereof) dollar-for-dollar by paying Amazon.


You say this, but I would not be so sure about that after having worked for a defense contractor.


I've also worked for a defense contractor, and the company I worked for, and the facility I worked at, took securing classified information very seriously. The consequences for screwing it up can literally be prison time, so I'm really surprised to hear the company you worked for didn't take it more seriously.

In this specific case, it doesn't sound like the "sensitive Pentagon Files" were classified at all, though, so it's probably not as big of a deal as the article is making it out to be.


The main issue is not necessarily security in the traditional sense, but rather the proper use of the security that AWS provides. In this case, AWS more than delivered by providing plenty of options to secure the files. You can make them private, use encryption, generate temporary access URLs, etc. However the user (contractor) simply made a mistake and misconfigured the service. This is why I developed https://cloudsploit.com - to catch these kinds of misconfigurations before they bite you. It's not enough to just have security available you actually have to implement it properly.


You're right that's a huge issue also - And Amazon (to use your example) also takes great pains when engaging with customers to stress that their security competencies and certifications absolutely do not extend to a customer's use of their platforms.

For an obvious example, if Amazon has PCI DSS compliance and a customer comes along and builds a payment gateway on AWS services, that customer is not also PCI DSS compliant by extension.

But this thinking extends to all of the security assertions/certifications/measures that Amazon puts forth.

That is to say, 'we secure the platform but you can absolutely make an insecure mockery of a system on top of that platform'. In all cases you must implement your own security protections and processes on top.


The idea of crawling for S3 content has been around for a while (see this post from 2011 https://digi.ninja/blog/whats_in_amazons_buckets.php for one example).

Also google can end up indexing content in places like S3 or Azure file stores if there's a link there from another piece of content that it indexes. (for example https://community.rapid7.com/community/infosec/blog/2013/03/...)


Cloud isn't less secure than on-prem if you are your average software startup or non-tech company, which, for many reasons, is unlikely to go all out on on-prem security. Commodity cloud should be less secure than on-prem, if you are fucking Booz Allen...


>Cloud isn't less secure than on-prem if you are your average software startup or non-tech company

Even that's hard to argue, unless you rely on a fundamental incompetence within that startup or an often-unprovable competence level on the part of cloud providers.

There have been enough cloud-hosting provider security facepalms hitting the front pages now so 'cloud providers are better at security than small shops' is starting to be called out as the blind falsehood it has always been. Fundamentally it has always really been 'We're unable to see what these guys are doing behind the curtain so we'll assume good faith', and little more. The emperor may have clothes, or in many cases has been utterly balls-out naked, but up until now groupthink has been that not knowing = must be secure.

I am biased though, I spend most of my work hours analysing service providers big and small, self-hosted and cloud, and have probably developed a suitably sceptical view in that time.


The really interesting question for me is why the government would award new contracts to Booz Allen after the Snowden leaks. (While I applaud Snowden, I deplore the government's inability to prevent his leaks.) Booz Allen demonstrated that they were unable to enforce appropriate practices for handling classified data. So what's the rationale for ever awarding them another contract? What capabilities do they have that the government doesn't have or can't develop?


The question is more, "what capabilities do they have that other companies don't", and I suspect the answer is "extreme proficiency at the federal RFP process".


This!

I interviewed at Booz in 2004. I told them i was interested in going to grad school and that i heard they work their new hires to death. I told them that id rather work myself to death in grad school where i can get a phd rather than at Booz. They said none of the folks in their group work more than 40 hours so I agreed to come to for an interview.

So I arrive at my interview all suited up. I was interviewed in a lunch room by a woman first. She was really nice. I asked her how many hours she works a week and she said around 60 and that everyone does. I was a bit taken back because this is the opposite of what the hiring manager had said.

Interviewer 2 comes in and I ask the same question during the course of the meeting and I get the same answer: he works 60 hours a week and so does everyone else but everyone really loves their work. I'm angry now because I had to drive 4 hours to get to this interview and it essentially wasted a whole day.

I literally got up in the middle of the interview and shook the guys hand and said, sorry, but i dont think this will work for me. The hiring manager must of saw that something was up and came and and asked what was going on. I politely said he lied to me about the position and mentioned the hours thing and he accused me of not being a team player. I told him frankly, you wasted my time and said goodbye. He threatened to not pay my hotel and per diem as i walked out. I never even acknowledged him and just kept going. I walked out past security on the way out and dropped off my badge, I didnt even sign up and no one even bothered to stop me.

They ended up paying for everything and there were no consequences. I ended up getting a well paying job at raytheon the following day. I worked there for 5 years and never worked a day over 8 hours AND they paid for my graduate work (they offered).

A friend mentioned it best, "When you Booz, you lose."


Similar Booz story: I took an interview with them while with a competitor. I had no real interest in the firm, but was interested in what they had to say and offer for competitive intel. Ended up having a great interview with the guy who ran one of their larger accounts, who happened to be one of my clients at the time. Got a phone call from their recruiter that evening who was excited to offer me a position at a 30% pay cut and a more junior title. She didn't understand my confusion, tried to convince me that "our managers are basically partners at all the other firms", and eventually got the picture when I started laughing.

I think it was also Booz who tried to get me to tell them confidential compensation information because, quote, "C'mon, I'm not going to tell anyone."


> He threatened to not pay my hotel and per diem as i walked out

Wow. Not only is that deeply unprofessional, I can't even imagine what he hoped to gain by that.


Your answer would be correct, at least in my experience... the amount of times my DARPA RFP's have been shot down by some minor box that didn't get ticked in the RFP process is a bit absurd. We even had a dedicated contract officer who used to work there to help with the process.


How would you have them decide who is allowed to deviate from the documented RFP process and by how much?

Also not checking all the boxes during the RFP process sends a signal about your ability to do so once the work is awarded.


>Also not checking all the boxes during the RFP process sends a signal about your ability to do so once the work is awarded.

A false signal that people would be wise to ignore. There are a multitude of examples showing that companies that learn how to check all the right RFP boxes pay no mind to checking any of the 'do a good job and product a good product' boxes. It shows they are good at checking only the boxes that make them money, and the current process is that doing a good job is not a box one's pay depends upon.

Is there any similar evidence that people who fail to check all the RFP boxes also fail to check the "do a good job and deliver a good product" boxes?


In the specific cases I'm talking about the 'ticking all the boxes' was rather subjective and not adequately explained in the RFP.


That, plus the same motte-and-bailey PR that in fairness most companies employ:

Terrible bad thing that happened? It was that one individual's actions and they've of course been let go! (see: VW emissions scandal, but this plays out in B2B privately all the time).

Good outcomes due to individual heroics? Our company has an established history of delivery! (ignore the fact that those folks quit years ago). The Ship of Theseus sails ever onwards ;)


This is true of firms getting government contracts in general, especially true of federal contracts, and, from everything I've heard (especially from people who've worked on defense contracting from either side) epically true of defense contracting.

Working the contracting system is a completely different competency than doing the work that is contracted for, and the people that get contracts are the ones specialized in getting contracts.

They probably have some competence in the actual work, but they often aren't competing with the best in the field for the work, because the best in the field for the work often have no or less competence in the relevant area ofmgovernment contracting.


How would Booz Allen Hamilton enforce security policy that is managed by the government client by one of its consultants?

Snowden was a consultant working on the client site. Booz Allen Hamilton's contract scope didn't include security policy. The only realistic way BAH could have enforced this policy is by having a manager follow each consultant around. This position would be added to the contract and increase the cost.

> What capabilities do they have that the government doesn't have or can't develop?

Absolutely nothing; permanent hires are deeply unpalatable by Congress, yet they encourage using contractors at higher cost to the government to perform the same work. One of the major pillars of the President's election platform was reducing the number of government employees. One of his first actions in office was to implement a hiring freeze for federal employees.


Good point about headcount. So the federal government hires contractors to handle classified data even though doing so makes it structurally impossible to secure classified data. And they do so over and over again because they're caught between headcount restrictions and mission requirements. And because, as noted elsewhere in the thread, Booz is a massive proposal-writing machine, they're well positioned to receive these contracts.


> So the federal government hires contractors to handle classified data even though doing so makes it structurally impossible to secure classified data.

Why? What's the difference between a contract employee and a government employee at this level? They all sign literally the same NDA, the same laws apply to both, the same polices apply to both. They both go through the same security screening process and they both pass the same (roughly) technical competency bar to do their jobs.

You probably see more contractor fuckups just because there are more contractors. There are also many government staff fuckups (sure, Snowden was a contractor but every big spy prior was a civillian or military).


>hires contractors to handle classified data even though doing so makes it structurally impossible to secure classified data.

Why? Ames and Hanssen were government employees.


It's really confusing how they ended up in this predicament. S3 buckets are private by default. You have to manually configure them to be public-read. Given the scope of Booz Allen's work, it doesn't seem like they needed any of the S3 buckets to ever be just public. I wonder if they're using software or maybe even just copied a script in which the default assumption for S3 configuration and uploads is to set files to be public?


We had this issue previously as one of our clients was using cyberduck to upload files and it had default read access for public for uploads.

[1] https://forums.aws.amazon.com/thread.jspa?messageID=213248


Yeah, this is also something that can be set using Transmit (my OS X GUI of choice). I guess I assumed that, given the size of the data they have to manage, they have something more sophisticated than file-management-by-click-and-drag but...maybe I'm giving them too much credit in terms of dev ops.


tinyS3 used to do this as well:

https://github.com/smore-inc/tinys3/issues/44


I'm wondering if they were bitten by the semi-common "any authenticated user" misunderstanding. That maps to any user in all of S3, not just your account.


Much more likely for it to be the ole "Download it quickly and I'll make it private"

Edit: After reading the article, I have no idea what could explain this. They had a dozen passwords in there as well..


Who knows what dumb GUI they used that configures the bucket with public access by default.


This happens surprisingly often.


Sometimes when I read such things, I really hope that it was a planted leak to seed disinformation to The Russians (tm). But then I realize that this is the US government we are talking about...


Kids these days have it easy. You young whippersnappers! We didn't have google dorks you have today! Hell we didn't even have google! Back in my day if you wanted to hack the Pentagon you actually had to hack the Pentagon!!! And there was no tor hideurass neither... times like this makes me think the internet is just one big honeypot...


<Tin-Foil-hat>

Well that could also be construed as a 'oups my bad' kind of way of leaking documents, instead of going the (painful) whistle-blowing route ...

</Tin-Foil-hat>


So docker registry credentials and auth to a "datacenter operating system". I wonder if they use Mesos and Mesosphere's DC/OS :)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: