"We are experiencing massive demand on our support capacity, we are going to get to everyone it will just take time.
Code Spaces : Is Down!
Dear Customers,
On Tuesday the 17th of June 2014 we received a well orchestrated DDOS against our servers, this happens quite often and we normally overcome them in a way that is transparent to the Code Spaces community. On this occasion however the DDOS was just the start.
An unauthorised person who at this point who is still unknown (All we can say is that we have no reason to think its anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a hotmail address
Reaching out to the address started a chain of events that revolved arount the person trying to extort a large fee in order to resolve the DDOS.
Upon realisation that somebody had access to our control panel we started to investigate how access had been gained and what access that person had to the data in our systems, it became clear that so far no machine access had been achieved due to the intruder not having our Private Keys.
At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel. We finally managed to get our panel access back but not before he had removed all EBS snapshots, S3 buckets, all AMI's, some EBS instances and several machine instances.
In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted.
This took place over a 12 hour period which I have condensed into this very brief explanation, which I will elaborate on more once we have managed our customers needs.
Data Status
All svn repositories that had the following url structure have been deleted from our live EBS's and all backups and snapshots have been deleted:
https://[ACCOUNT].codesapces.com/svn/[REPONAME]
All Git repositories are available for export but all backups and snapshots have been deleted
All Code Spaces machines have been deleted except some old svn nodes and one git node.
All EBS volumes containing database files have been deleted as have all snapshots and backups.
Code Spaces Status
Code Spaces will not be able to operate beyond this point, the cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in a irreversible position both financially and in terms of on going credibility.
As such at this point in time we have no alternative but to cease trading and concentrate on supporting our affected customers in exporting any remaining data they have left with us.
All that we can say at this point is how sorry we are to both our customers and to the people who make a living at Code Spaces for the chain of events that lead us here.
In order to get any remaining data exported please email us at support[at]codespaces.com with your account url and we will endeavour to process the request as soon as possible.
On behalf of everyone at Code Spaces, please accept our sincere apologies for the inconvenience this has caused to you, and ask for your understanding during this time! We hope that one day we will be able to and reinstate the service and credibility that Code Spaces once had!"
Edited because someone didn't like my original tone. Was a bit rushed to be honest.
Few things seem off about this:
- Offsite backups were also deleted, I don't think they had offsite backups, or at least backups you could legitimately say were "off site."
- EC2 has two factor auth, why you wouldn't use this for your business I don't know. [1]
- Corresponding with extortionist is a really dumb move. It would be better time spent locking things down - contacting amazon directly to get an account lock / getting your ducks in a row.
Hindsight is a bitch. Of course using 2-factor auth was the way to go, of course offsite backups have to really be "off site" (and not available to anyone with access to AWS control panel to delete), etc, etc, etc.
Now there are many "of courses" for the owners (that external people already knew, but it doesn't help their situation). It seems that for them these things weren't so obvious as they are now... the unknown unknowns.
Sad story but I'd call lessons learned for them, no news for the rest of the Internet.
Probably because of the part where they say this isn't the whole story: "This took place over a 12 hour period which I have condensed into this very brief explanation, which I will elaborate on more once we have managed our customers needs."
That could be it. But there is a certain dissonance about this whole thing, I try to imagine myself in the same situation and the whole thing weirds me out. How could this mysterious hacker have known they had no other backups? Have they talked to LE at all at this point? Why not string the guy along, buy time, immediately alert amazon to lock the account completely?
So many questions. Anyway, they'll be updating this sooner or later, I just can't help but feel a bit weirded out by some of the things in there (and things that should be in there that are not).
This is most likely just my professional paranoia acting up. And of course it is easy enough to be back-seat driver here, I'd hate to be in their shoes, no matter how they got there.
> How could this mysterious hacker have known they had no other backups?
I don't think we can infer that he knew that. It seems more likely to me that he expected the outcome of deleting all their Amazon stuff he could reach would be that they would be down for a day or two as they reconfigured everything and then restored from offsite backups, costing them overtime or comp time for their IT guys, a few disgruntled customers who leave, a few more disgruntled customers they have to placate with freebies, and making them more likely to pay next time an extortionist comes around.
I would not at all be surprised if the extortionist is very surprised that they did not have other backups and his actions have probably killed the company.
He's probably also somewhat worried, as this probably knocks the monetary damages up enough to (1) make it much more likely that this will get some serious law enforcement attention, and (2) if he is ever caught and convicted greatly increase his sentence and/or fine by moving the severity level of the offense way up.
For instance, here are some examples for 18 USC 1030(a)(5), which covers causing damage or loss on a computer via unauthorized access, assuming no other factors that increase the sentence:
Trying to cost someone a few thousand dollars worth of damage and instead killing their $10 million dollar company, for instance, changes it from 6 months tops to 5 years minimum. Ouch.
> I would not at all be surprised if the extortionist is very surprised that they did not have other backups and his actions have probably killed the company.
> He's probably also somewhat worried, as this probably knocks the monetary damages up enough to (1) make it much more likely that this will get some serious law enforcement attention, and (2) if he is ever caught and convicted greatly increase his sentence and/or fine by moving the severity level of the offense way up.
That's plausible. It makes some sense that if you destroy something that you should be responsible for that. At the same time, even for a hacker the assumption that there would be back-ups would be a fairly logical one, though I'd hate to be in a position of fielding that defense.
"We are experiencing massive demand on our support capacity, we are going to get to everyone it will just take time. Code Spaces : Is Down!
Dear Customers,
On Tuesday the 17th of June 2014 we received a well orchestrated DDOS against our servers, this happens quite often and we normally overcome them in a way that is transparent to the Code Spaces community. On this occasion however the DDOS was just the start.
An unauthorised person who at this point who is still unknown (All we can say is that we have no reason to think its anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a hotmail address
Reaching out to the address started a chain of events that revolved arount the person trying to extort a large fee in order to resolve the DDOS.
Upon realisation that somebody had access to our control panel we started to investigate how access had been gained and what access that person had to the data in our systems, it became clear that so far no machine access had been achieved due to the intruder not having our Private Keys.
At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel. We finally managed to get our panel access back but not before he had removed all EBS snapshots, S3 buckets, all AMI's, some EBS instances and several machine instances.
In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted.
This took place over a 12 hour period which I have condensed into this very brief explanation, which I will elaborate on more once we have managed our customers needs.
Data Status
All svn repositories that had the following url structure have been deleted from our live EBS's and all backups and snapshots have been deleted: https://[ACCOUNT].codesapces.com/svn/[REPONAME]
All Svn repositoies using the following url format are still available for export but all backups and snapshots have been deleted: https://svn.codespaces.com/[ACCOUNT]/[REPONAME]
All Git repositories are available for export but all backups and snapshots have been deleted
All Code Spaces machines have been deleted except some old svn nodes and one git node.
All EBS volumes containing database files have been deleted as have all snapshots and backups.
Code Spaces Status
Code Spaces will not be able to operate beyond this point, the cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in a irreversible position both financially and in terms of on going credibility.
As such at this point in time we have no alternative but to cease trading and concentrate on supporting our affected customers in exporting any remaining data they have left with us.
All that we can say at this point is how sorry we are to both our customers and to the people who make a living at Code Spaces for the chain of events that lead us here.
In order to get any remaining data exported please email us at support[at]codespaces.com with your account url and we will endeavour to process the request as soon as possible.
On behalf of everyone at Code Spaces, please accept our sincere apologies for the inconvenience this has caused to you, and ask for your understanding during this time! We hope that one day we will be able to and reinstate the service and credibility that Code Spaces once had!"