Is there any evidence there has been a breach of CircleCI directly, or just that MongoHQ suspended all access to their database(s) and so the service went down?
At this point it doesn't seem like they were 'hacked'[0], but simply involved in a larger security incident.
[EDIT] seems the linked page doesn't mention 'hacked' anywhere (literally - I searched for it!) Seems like the title[0] is a classic case of editorialising
[EDIT 2] As per below, would be very interested to see if their database being 'compromised' means that data was actually accessed by the attackers, or if they are just fall out from the bigger attack
[0] original submitted title was "CircleCI has been hacked"
>We are contacting you to inform you of an ongoing security incident affecting CircleCI customers, as a result of the compromise of our database (http://security.mongohq.com/notice).
Seems like a 'security incident' is code for 'hacked'.
I think I agree with you, however there is a big difference between having your service stopped as a precaution and having data lost/accessed by a 3rd party.
From that line it does seem more likely that they might be one of MongoHQ's clients who MongoHQ noticed had had their data accessed as part of the bigger breach.
In that context, saying they have been hacked is probably fair. Would love more details.
This is very likely. I'd guess that the likely target was a MongoHQ customer, and that MongoHQ getting hacked was collateral damage as part of an attempt on one of their users.
This is why I'm so mindful about using third-party services that require access to our code to perform their service. It's not that I don't trust the third-parties not to abuse their power, but by trusting them with my code, I also have to trust their security measures.
As such, I always very carefully weigh the benefits such a service can provide against how much work it would be to do it myself (which will likely produce a shittier result, but which will be fully under my control).
It's not even that I think I can do security better than $SERVICE, but I'm a much smaller target. I'm not as afraid of a targeted attack than I'm afraid of somebody compromising one of these focal points and then just lifting everything.
Github has a feature to allow access to a singular repository via a key. It would be logical for CircleCI to use that feature, although I'm not sure they actually did.
That could be irresponsible as a company which uses MongoDB for non-sensitive data (say data in publicly available pages) would be shamed when they didn't have a data breach of sensitive data. Or worse, a company that has signed-up but only played around with MongoHQ without using it in production. Each company should announce its own breach (and the scope of the data exploited) to its customers.
I understand all the implications you mention, and that's why I phrased it the way I did.
Though, the assumption I'm making right now is that the hackers know exactly what sites they've compromised. The longer they have this information in advance of MongoHQ's customers' customers, they more damage they can do. That sucks terribly for those people (potentially)
All this said, a serious lesson about PaaS/SaaS/DBaaS--whatever you want to call it--has been learned today.
[edit] - I'm reminded here of the Epsilon breach. Epsilon is magnitudes larger than MongoHQ and they apparently did decide to announce the names of affected companies. I am not sure how they handled that internally. [1]
In Epsilon's case those end users had direct consequences - their names and emails were compromised.
The end users of sites/apps/services using MongoHQ are probably unaffected in many cases since "a database" doesn't imply credit cards or even credentials or emails. The average database is probably just holding content for a site or app.
In probably close to all cases the end users would have no clue how they may be affected since they won't know what's in the databases, or what a database is.
This is one of the reasons why we have shutdown our own CI solution. We didn't have enough manpower to develop the product, audit the whole stack and monitor against attacks.
CircleCI's response to the incident appear adequate. In the best case, their system was shutdown before any keys were compromised. If so, that level of monitoring is certainly better than what the average business would be able to roll on their own.
Identified - We have notified all users and recommended appropriate action:
"We are contacting you to inform you of an ongoing security incident affecting CircleCI customers, as a result of the compromise of our database (http://security.mongohq.com/notice).
We are taking aggressive action to protect your data and systems. At this time, we have suspended all CircleCI account access, and all builds & workers have been suspended. In addition we have revoked all access to Heroku and GitHub OAuth tokens and API keys uploaded to CircleCI.
We do not yet know the scope and impact of the intrusion and are therefore treating this event as if all data has been compromised. While we have no evidence that these credentials have been compromised, we urge you to revoke the following:
SSH keys that were uploaded to CircleCI
API tokens added to CircleCI as environment variables
secrets stored in GitHub repositories
We will be keeping you informed at http://status.circleci.com and will update you at regular intervals as the situation progresses.
We deeply regret that this has happened and are working around the clock to resolve this incident and protect your data and systems."
22:25 PDT
Update - We are still investigating the issue. The full team is engaged and we are working with upstream providers to diagnose and respond to the issue, and protect all of our users. We will keep you informed.
21:17 PDT
Update - We are currently investigating an ongoing issue with our database service. At this time, we have suspended all account access to our service. All builds & workers have been suspended. We will have another update in the next 30 minutes.
20:20 PDT
Investigating - "CircleCI is experiencing technical problems. We're investigating and should have an update within 30 minutes."
19:30 PDT
I'm curious why they mention uploaded SSH keys. I presume they mean cases where SSH keypairs have been uploaded? The public key is public, and in fact anyone can use the github API to pull the verified public keys for any user in the system, http://developer.github.com/v3/users/keys/. If there is code uploaded with private keys in it, then it's likely there are other security problems in the organization.
I do not think so. There is currently no evidence that CircleCI itself had its data directly compromised as part of the MongoHQ hack.
They are just being cautious and assuming the worst and recommending their clients do the same (quite right too).
However the fact that so many apps can be screwed by a breach in a "Database as a service" style setup will make me wonder "How is this SaaS storing my data? internal or oursourced?" When evaulating new ones.
[EDIT] - Just noticed that I did not actually give opinion on the guess. It seems a reasonable guess that this could have been part of an attack on a MongoHQ customer.
I don't think it's relevant that it's a database as a service over all the other hosted services we put our trust in - if someone hacked Heroku, RackSpace or whatever other service provider they would get some db access, aws keys, source code etc too.
Private repos on github and bitbucket are probably a goldmine of accidents waiting to happen considering how many api keys etc slip into public ones!
Not really sure what the takeaway should be for developers building on [anything] other than to do your best not to store stuff in a way that can hurt your users if a platform you use is compromised.
http://security.mongohq.com/notice
So this really is mongohq's fault...
Demonstrates the dangerous domino effect if you think of all the code/keys/secrets...
Since circleCI users upload the private heroku SSH keys to automate deploys, how hard would it be for the intruders to get access to the environment variables stored in heroku?
Customers: stripe, Red Bull, Kickstarter.. and MongoHQ. How meta. This hack could get a lot more interesting depending on how much code they could have obtained from those companies.
I started testing out CircleCI a bit ago and this really doesn't help seal the deal for me. What am I supposed to do now? CircleCI was connected to my github account...are my SSH keys not safe now? If so, not clear on what I'm supposed to do.
Revoke the CircleCI key that you added to your repo on Github. In the worst case scenario, that key has been compromised and could be used to access your source code.
Is there any general sense of the techniques being used in these cracks? Just from a defensive coding perspective, I'd like to know if there are programming errors, weak links in frameworks, or what.
If you read MongoHQ's extremely detailed report[0], you'll see that the original intrusion was based on shared credentials (that were cracked on another system). This was then exploited in a number of ways to compromise various clients of MongoHQ.
Lots more discussion over at [1].
The main takeaway is to take security seriously, and employ multiple levels of security. The MongoHQ team are doing things like 2 factor auth, and restricting customer service tools to a vpn. As far as I can see, no framework or coding bugs.
At this point it doesn't seem like they were 'hacked'[0], but simply involved in a larger security incident.
[EDIT] seems the linked page doesn't mention 'hacked' anywhere (literally - I searched for it!) Seems like the title[0] is a classic case of editorialising
[EDIT 2] As per below, would be very interested to see if their database being 'compromised' means that data was actually accessed by the attackers, or if they are just fall out from the bigger attack
[0] original submitted title was "CircleCI has been hacked"