Only 8 accounts were affected. Do not worry. Minor breach. Not much harm done.
It seems to me the truth is the attacker looked for bitcoin wallets and emptied them. The fact he could identify 8 accounts and access them suggests the attacker could have accessed far more accounts if they wished. I think this is the most worrying thing about the breach.
I don't really understand how bitcoin works but it seems that people with wallets need to set up multiple wallets on multiple providers and limit the amount of bit coins in each wallet to limit any losses from breaches like this.
If I was a linode customer I would be thinking about moving. This message, while fairly open, doesn't give me much confidence there aren't other security issues with the platform.
Just imagine that BitCoin is like having cash in your wallet, because that's more or less its intended model. There are a lot of 'anti-counterfeiting' measures because computers are very good at copying, and you don't want people to be able to copy BitCoins the way they can copy music -- and when you ask "what is BitCoin?" people basically start to tell you about the anticounterfeiting technology, and the limits on printing uncontrollable amounts of money. But it's essentially stamped paper in your wallet in any other sense, worth whatever people using it on the Internet will pay for it, not backed by anything in particular but its usefulness.
Basically a lot of people were renting storage rooms in an apartment complex run by Linode, you get your own key to enter the door and retrieve and store things -- whatever. Some people left their wallets inside these buildings, with cash therein. Someone else used some unidentified systematic security flaw, but we don't yet know what it was. Maybe there is a ventilation system which is easily navigable once you know how to get in; or maybe all of the rooms have unlocked windows for no good reason; we haven't been told yet. (There are some suggestions that they stole a key from one of the janitors who cleans these rooms up.)
What we have been told is that some burglar stole eight wallets, and that "All activity by the intruder was limited to a total of eight customers, all of which had references to 'bitcoin'." That suggests that the burglar did indeed peek in the windows beforehand somehow, to find out that these 8 rooms had wallets inside. Otherwise, presumably they would say something like, "The intruder broke into many of our customers' accounts but didn't actually do anything in 99% of cases." In that sense I think the scary bit isn't that he accessed the 8 accounts, it is the fact that he identified them in the first place.
Amortizing the loss across many points of failure may be a good idea, but it wouldn't seem to solve the central problem. Suppose I put $20 in two accounts with 5% chance of compromise, rather than $40 in one account with 5% chance of compromise -- either way, I should expect to lose $2. What I've changed is that I am more likely to lose some of my money (9.75%), but I am less likely to lose all of my money (0.25%). This may appeal more to risk-averse people but it is not fundamentally changing the situation.
Perhaps a better approach is to keep a BitCoin wallet encrypted, since that's pretty simple to do in day-to-day life. This is something that you can't do with your wallet -- you cannot turn your wallet into a steel vault with two-foot-thick walls.
> In that sense I think the scary bit isn't that he accessed the 8 accounts, it is the fact that he identified them in the first place.
This isn't all that surprising. There are basically two reasons why you would have a Bitcoin wallet on a server: if you are mining using the CPU power of that machine, or if you need to send Bitcoins from an online application. For example, one of the people who mentioned having coins stolen was from a mining pool; you need some automated system to pay out the earnings to the people who have been doing mining, and so the wallet for that automated system was on the server, and was stolen. I suppose one further reason might be as a backup, but in that case, I dearly hope that it's an encrypted backup without the encryption keys in the sever.
Given these reasons for having your wallet on the server, it's not surprising that people found them. These require network-facing services, that are easy to trace back to the server in question. The mining pool is a public service; anyone can join, and find the address of the server. Furthermore, when you make payments, you announce them to the full Bitcoin network. Someone sniffing transactions can watch where transactions originate, and target that. If they already had a compromised customer service account on Linode, they probably watched the Bitcoin network for a while, made note of transactions originating from IP addresses in the Linode range, and then targeted those accounts.
One way to protect yourself from this would be to proxy your Bitcoin transactions through a host other than the one that has the wallet, obfuscating where the transactions are actually coming from. You could even go so far as to make all of your transactions via Tor, which would probably make it fairly difficult to find where your Bitcoin wallet actually lives.
> Perhaps a better approach is to keep a BitCoin wallet encrypted, since that's pretty simple to do in day-to-day life. This is something that you can't do with your wallet -- you cannot turn your wallet into a steel vault with two-foot-thick walls.
The problem is, if you need to make payouts from your wallet, then the machine that does that needs to be able to decrypt the wallet. That machine can then be compromised to be able to steal your keys. Encryption doesn't buy you all that much, unless you are just doing a backup and don't need the machine to be able to do online transactions at all.
Perhaps another solution would be to encrypt each key in your wallet separately using a k-out-of-n encryption scheme (where produce n keys, any k of which can decrypt the wallet). You can then distribute those keys to independent hosts, which hopefully should not all be subject to the same vulnerabilities. Then any time you do a transaction, k of those hosts will need to produce their key to decrypt the key in your wallet and perform the transaction. That way you would have to compromise several different, independent hosts in order to steal the wallet.
Of course, this would drastically increase the cost and complexity of the system; and you would need to ensure that whatever system that authorized payments was likewise distributed, which if you had, say, a web-facing service would be difficult.
The easiest thing to do to reduce the risk is to only leave enough value in the wallets that are on the servers for a couple days worth of transactions. Then you transfer Bitcoins from a more secure location once a day to keep the coffers full. This is not much different than a physical store; yes, you are at risk of being robbed, but if you only have one days worth of cash there, with the rest somewhere more secure, you reduce how much risk you have.
If they had used 'unlock-at-boot'/true-crypt style disk encryption and kept the password/key off the machine they would have been safe from this attack. (They would also have to provide the password/key every restart.) It is only in hindsight that something like this seems worth implementing!
Your more general solutions would protect from an attack that rooted a live system rather than just resetting the root password while the machine was offline.
Indeed, we are a Linode customer, and this message only helps a bit. Yes, I know now that we are not affected. But little other information is given: were the user accounts compromised by a vulnerability in Linode's VM management software? If so, was this vulnerability found and fixed? Or did the attacker compromise the account of one of Linode's employees?
Yes, but how did they steal the credentials? Did they find a sticky note with the username and password written down? Or was there a vulnerability in the admin interface that allowed someone to sniff credentials? Or did they hack into the personal computer of someone with admin privileges?
Basically, this announcement gives me no confidence that they've done due diligence in fixing this problem. They haven't explained what the vulnerability actually was, nor what they have done to avoid it in the future.
Of course, this does speak to the dangers of using hosted services for anything that needs a high level of security. Anyone with appropriate admin privileges on the host system can compromise any user. That increases the attack area considerably; you don't need to attack the system directly, nor the users of the system in question, you just need to find one person who has admin privileges who is vulnerable, steal their credentials, then attack any users at your leisure.
It wasn't clear from the memo, but how did the attacker know which 8 specific accounts had 'references to bitcoin' if they didn't access other accounts too?
I doubt it's that: I would have worked out the target sites hosts (looking up the IP), made a list, worked out the most popular by value - probably worked out it was linode and gone from there...once in, I'd bet it's a simple task of searching for the IP in their interface.
Ok, asking the other way around, did Linode check out the non-attacked VPSs to determine whether they didn't have bitcoin wallets on them?
I'm not disagreeing with anything else said here - it's just that it seems that the memo was very conclusive in stating something that couldn't be known unless there was more VPS introspection than they claim...
I'd like to understand the what actions will be taken to prevent similar attacks in the future. Also, what can I as a linode customer to prevent my host from being compromised in a similar fashion.
Implementation of two factor authentication for your customers and requiring it for a root password reset would go a ways to preventing similar attacks.
The only excuse for this incident is if they did have 2-factor for their admin portal (which, from the discussion, is presumably separate from manager.linode.com) and someone conducted a targeted attack on a linode employee to compromise/steal both her token (either separate or a totp oath app like googleauthenticator or similar) AND her password.
If someone went to that much effort just to steal some bitcoins, they set their sights too low. Linode must host more valuable stuff.
Yes they are free a day after the event. There are bound to be logs and with that the possibility of capture. The person or people who did this are not quite resting easy, well unless it was a foreign entity that did it... That would change things considerably.
Given the nature of how bitcoin works, however, they can easily move the money through exchanges in foreign jurisdictions, effectively laundering it. There'd be a trail, with logs, but it'd be inaccessible to investigators.
Such as? bitcoins are valuable and easy to run away with. Stolen credit card numbers are such a hassle to monetize that they can be bought with only $2 or $3 of e-currency.
Also, is this customer service portal available via a public URL? Shouldn't some sort of VPN access be required to even get on the network hosting these things?
Can additionally restrict by IP which. That is also the way Verisign protects the registry system that registrars use (as well as two factor authentication).
The title of their blog post is: "Linode Manager Security Incident" and that's exactly the name of the customer website where you can manage your instance, billing, etc.
I think someone found a way to gain access to any Linode customer account through the customer website and from there shut down the instance, changed root password and rebooted (you can do that from there).
I think it's just poorly worded - from the OP and what I've read elsewhere someone accessed the portal used for customer service employees that has access to options/all hosts.
Yes the notes in the original pastebin post kind of indicated it was an issue with a "customer support" control panel, maybe it's just another method or area using the Linode Manager.
I just wish they posted a little more, it feels vague.
All this talk about banks being safe yada yada and cloud hosting not safe for US50k. Real banking companies (with billions of dollars on hand) do use commodity cloud hosting including Linode, for even sensitive parts. Take for example Natwest online banking login. On initial login page they load a cookie via an image from www.advanced-web-analytics.com and then once you enter a customer number the next page loads a ...drum roll... javascript file from www.omni-traffic.com. Now who can tell me what one can do when you have control over the Javascript on a banking login page?
Ah crap. It looks they have been moved to Amazon EC2, ~8 months ago they were hosted on conventional Linode VPSs. Points still stands though.
In my experience working on a US financial website, a bank would never consider using a VPS like Linode to store actual banking and customer data. It's not even close to Level 1 PCI compliant.
Not sure about this PCI complaint stuff, but perhaps this is why major banking companies jumped from Linode to EC2? Much improvement? Although I must say I have friends working on banking websites in the UK that dont know the whole picture, its not unreasonable to assume that these things are fucked up.
From what I know, PCI compliance is firstly, just a guideline. It's not like OSHA, but IANAL.
I also know there are "levels" of PCI compliance. The highest one, which reputable banks should be following, is very strict AFAIK, and includes provisions for controlling who has access to the physical hardware, encryption levels, etc. The fact that a Linode VPS can be 'rooted' via their management software by a sysadmin working for Linode would, from what I can tell, make them unqualified to be used to store banking transaction & customer data, though perhaps I am wrong.
Well tomg, when I researched this ~8 months ago there were at least 2 US financial websites that were using the same specialized analytic company that injected JS into banking login pages that were hosted on Linode VPSes.
Oh I don't doubt you. I'm not an expert on this, heck, I wasn't even allowed on to the actual servers (because of said compliance). I don't know the guidelines for login pages or what kind of security third party JS libs are supposed to have (also PCI is not a law, afaik).
What I'm asserting is that the servers that store the actual banking and customer data have very high security standards. It's one thing to store front end website code on a VPS, it's a totally other thing to store your database with customer & bank data on Linode.
The bitcoin breach seems analogous to Bank of America storing your account information on Linode and trusting it as the Real Data. Does that make sense?
[quote]
The bitcoin breach seems analogous to Bank of America storing your account information on Linode and trusting it as the Real Data. Does that make sense?
[/quote]
//reply to tomg, but seem HN stops nested replies beyond a certain level
At the end of day you can have millions of dollar of security, auditing, PCI compliance tests passing, developers that celebrate every Friday that everything is secure, data is hosted on premise etc... But if you leave the login page javascript to a third party hosted on Linode then you might as well be BoA storing your data on a mySQL linode instance. So in a nutshell it kind of undermines the work you guys do.
That's very true, and TBH I'm a bit surprised these banks are allowing that. IME doing frontend code for banks is that they're very strict on third party libs, even ones hosted by the bank itself, right down to only approving certain versions of the lib.
The user who was affected by the incident quoted an email from linode that stated "Our investigation has revealed a customer support interface was used to access your account.", based on that and all the information of that post you get the impression that through the 'interface' the attacker was able to change the vps root password.
Now a reply from linode comes and says "The portal does not have access to credit card information or Linode Manager user passwords". So if the portal doesn't have access to Linode Manager how the attacker gained ability to change the root passwords ?
Thy should give more details on the incident, i do have a certain trust in the ability of linode to have a secure environment & i can understand that things like that will happen at some point to everyone. However its one thing for someone to get access in your system because you had your roots password to 'password' and another if there was a bug that got exploited.(yea this is an extreme example)
> So if the portal doesn't have access to Linode Manager
They didn't say that, they said it doesn't have access to the passwords. They have an interface to change details, they just can't read them. So they can reset your password to "hunter2" but they can't see if it's "hunter2".
You are right, my bad on that. Still this looks like a Public relations post by them than giving out facts. They should be explaining what the attacker could do by gaining access on that interface, the ability of the attacker to change the password has the same consequences.
The point is that exploited interface had a backdoor access to the virtual machines (to be able to change passwords or w/e)
I understand how this might be confusing to a third party, but Linode's response thus far makes perfect sense to those of us who have been customers for a while. We're pretty aware of the general parameters of Linode's internal systems.
I run a web app. I built an administrative interface for managing it. This interface includes the ability to log me in to any individual user's account (by appropriately initializing a session with that user's ID, the same as logging in normally would do), and to reset any individual user's password. This interface does not let me view any user's passwords; it's both technically impossible, as I don't store them in plain text or encrypted form, and unnecessary.
Unless you believe they're lying, then Linode has the same thing. Some interface where they can access their users' Linode Manager accounts, but that interface does not show Linode Manager passwords or credit card information.
Suspicious events prompted an immediate investigation and the compromised credentials used by this intruder were then restricted
Does that mean the credentials were gained outside of Linode and used to change the root passwords of the accounts for the purposes of the theft, or were those credentials used as part of an exploit in Linodes systems?
It sounds like Linode's user system is different from their server management system. They probably have some administration tools for resetting VPS passwords, but don't have access to sensitive user details.
I've got to wonder too. If you rent out an apartment and someone breaks in with a key they stole from the landlord and takes your TV, can you sue the landlord? On the other hand, if you rent a security deposit box from a bank and it gets broken into, is the bank liable?
I think the threshold would be, did Linode take "reasonable" precautions in protecting the servers in question. Just like the landlord or bank. So long as they take reasonable precautions, they can't be held liable.
You can expect for your landlord to keep their copy of the key in some sort of lockbox, but you aren't going to expect them to keep it in a pressure-sensitive safe, guarded by movie-style lasers and a German Shepard.
You expect a little more security out of your bank.
The real question is: why were services that act like bitcoin banks storing their coins on Linode in the first place.
Reasonable is subjective, I believe all hosts should use an NIDS and a HIDS if they don't I in my opinion consider them amateur. Regarding resource utilization sampling connections periodically would not take much.
The only (realistic) way to generate bitcoins today is with GPU or other specialized hardware that doesn't exist on webservers.
These servers that were compromised were used to manage generated bitcoins. One was used by a pooled mining service (mined coins were sent to the server then payed out to miners) and the other was a faucet service which would give a little bit of bitcoins to new users. The other 6 servers that were compromised are unknown to me.
It all comes down to insurance. No one generally pays out of their pocket, they pay with insurance money. The reason you pay so much at a bank for a tiny box, where as, space wise, you pay a fraction for your appartment, or essentially change for a storage locker, is that the bank expects you to put valualbles in that space and so pays lots of that money to inssurance so if they do get robbed, they are covered and can pay you. The other places... dont. And usually have clauses in their contract saying as much: "We aren't insured for valueables, so don't store them here, it may not be safe and we won't (and can't) pay you back if they get stolen.
Linode falls into the second category, cheap, uninsurred storage space. You want storage space insurred against digital robbery, a) good luck b) expect to pay a lot more than $20 or $30 a month
These bitcoin servers trusted their currency exchange to cloud servers--VPSes, really--and they want Linode to compensate them for the money lost? Insane. I don't claim to know anything about the bitcoin infrastructure hosted here but I'm going to go out on a limb here and say that there was no dedicated hardware firewall in front of it, no IDS, no WAF, nothing but some Linux instances running iptables.
The payment card industry wouldn't certify this hardware to process credit cards for a mom-and-pop online business, yet these guys use it like their bank? Come again?
PCI-DSS is actually quite easy to get certified in. I would say the issue is Linode not protecting their customers but as you say it's completely optional and would incur higher costs.
So you are saying that you can not use cloud hosting for anything serious? Even if they don't keep cash around, most web apps might lose a lot of value by being hacked. user data stolen, reputation lost and so on. So according to you, the cloud is only for worthless toy projects.
If you put $12k of value on their service, why do they take on that risk? Why wouldn't the risk fall squarely on your shoulders?
Why would a hosting provider take on the liability of what you host on it? If the underlying filesystem had an error that Linode could avoid and you lost data, would you expect them to replace the value that was lost? Why is it not limited (at most) to the value of the VPS itself? (I can't imagine them compensating for hardware failure either).
Any business that would be running on linode could be worth potentially far more than that. What your saying here is that no one should put anything of value into a VPS, which I imagine is not something linode itself would say.
OP's expectation was that Linode should have unlimited liability where no guarantee on their part has been made. If you expect Linode to take on the risk then you would want (and probably need) that in writing.
Otherwise, you take on the risk: which would be true wherever you hosted it without any guarantees regardless of whether it's a VPS, dedicated server, or your basement.
One user (slush) lost $12k worth of bitcoins, but another (bitcoinica) lost upwards of [$40k-$50k - wrong].
EDIT: first report from bitcoinica was "over 10k btc". Most recent report is 43,554 BTC, which would be worth almost $200k if liquidated on MtGox at the moment.
Not entirely sure, but I believe this represents someone willing to sell 1 BTC for $1,337,000,000,000.00, where all sell orders in the order book add up to 272981 BTC and USD$1,405,085,033,491
Best thing to do would be for Linode to generously reimburse these people, but they have no obligation to do so.
As a policy, it would neither be good business nor appropriate for Linode to assume all of the risk in a situation like this.
Also, if Linode did put a policy in place to assume some of the risk (some sort of insurance policy) they open themselves up to scams (just get your friend to rob your bitcoins and cash in on Linode's good will insurance policy).
The scam you mention would be difficult to pull off as there would have to be evidence that the account was compromised on Linode's end for the customer to make a claim.
In Linode's FAQ, they mention that their virtualization management software was developed in-house. They seem quite proud of this, quipping: "The Linode Manager is custom software, written in house, and is not for sale (although others have tried to mimic it)."
It seems to me that this is a classic example of security failures following inevitably from a lack of peer review. Maybe Linode didn't consider its LM software to be peer-reviewable, but I bet the victims of the bitcoin thefts wish that someone else had tested the code (and human systems surrounding it) for vulnerabilities.
Is this not exactly what Bruce Schneier frequently points out? Anything that must withstand attacks to protect the valuables within should be tested by attacking it. A lot. My hunch is that the vulnerability exploited by this attacker would have been found and fixed already if the LM software were more open.
I think you may be a little off here. The statements in the thread seem to indicate that the compromise was not based on a vulnerability in custom software, but compromised credentials. You can certainly argue that the management console should be protected by two-factor (and it should be), but their software doesn't seem to be at fault here.
I would be willing to bet that they have had the system tested by external security contractors and scanned with automated scanning tools. This seems to be a people problem and features problem not a vulnerability problem.
I guess we just don't know at this point. You may very well be correct. I guess if you want to use an open source provider, just make sure they are running OpenStack (openstack.org).
I really hope the various bitcoin related incidents don't poison the well for online currencies in general. Yes, you need a greater level of security when dealing with transferrable, relatively anonymous or pseudonymous (and rapidly extractable) online assets that you don't need for book-entry accounting with an audit trail and reversible transactions (credit cards, ACH, etc.). Yes, this is beyond what even most banks currently use. No, it's not beyond current technological state of the art.
Gaming (i.e. casinos), at least some of them, do a reasonable job with some very similar security problems.
> I really hope the various bitcoin related incidents don't poison the well for online currencies in general.
It shouldn't do significantly. In the public's mind the well is some distance off, many not even knowing it exists, so there is plenty of time to get security better sorted before the average man on the street has his money invested.
Also, everyone knows that cash and other forms of investment are far from safe anyway. Hopefully people will eventually see online currency as being no less safe than other forms, and if the security is done right they'll see it as more safe (as it could be).
But this seems to me to be a general security issue with decentralised anything, not a bitcoin specific problem. If you remove central control, and take as full control yourself as possible, then you remove responsibility from other people and that is something you need to seriously think about. Providers like Linode will not be taking responsibility for financial loss in these cases and paying for hefty insurance policies to underwrite the possibility of said loss: they'll just add clauses to their contract disavowing themselves of responsibility if such clauses don't already exist.
One way to protect yourself is to spread the money around multiple places, then one hacked provider doesn't put all your resource at risk. Again this isn't bitcoin specific: if you have more then 50K to invest over here (in the UK) then you split it between multiple organisations as only 50K per registered organisation is guaranteed protected by national safety nets provided by government.
Back onto "poising the well" this could of course work the other way around: if the virtual currency is worth the effort of stealing then it might be seen as more valuable in the eyes of the public - much like a sign you have a good product is that knock-off copies start appearing.
Yeah, kinda. Although E-Gold's problem was force majeure, not technical (did they ever get hacked? people stole passwords all the time, but since it was double entry, it was pretty easy to reverse).
osGold, etc. were probably better parallels. But yes, this whole field has had a long series of technical, business model, governance, and government related failures.
e-gold was shut down by the government for running an unlicensed money service business. They were officially Nevis based, but actually run out of Melbourne, FL (where they started originally); criminal charges were involved. The main reason was a particularly motivated federal prosecutor and the specific uses e-Gold users found -- child porn, paying for stolen cc's, etc. It wasn't a general thing about PATRIOT/MSB laws.
osGold was just an internal fraud; digital currency, give me your (gold) and issue tokens -- then one day it disappeared, taking the original value with it.
There were hack attacks by third parties against some other gold currency systems at the time, but I don't remember the details.
While the incident is unfortunate, I do give them credit for being up-front, honest, and relatively speedy in their response. Sad to say, I'm not sure a lot of other hosting providers would be as quick to admit culpability.
That said, there's no way that resetting the root password should be something a customer service rep can do. Particularly when Linode are very explicit about not being a service for newbs - and are in general unwilling to provide help setting up a Linux system.
Liability might be unlimited, but most of the insurance products for other categories cap it based on premium. So that may not be much of a problem. I agree with the difficulty of modelling the events though... they need more data for any kind of modelling.. I'm guessing amazon already has some kind of insurance to cover its risks under AWS service agreement..
I wonder about that. It seems to me that the bitcoin wallet shouldn't have been accessible after reboot, at least until someone came by to give the agent managing it the passphrase that would allow it to decrypt it's state. From the sound of things the wallet was just laying around unencrypted?
Not if he entered it manually on every reboot. There'd still be attacks around snooping the contents of the wallet from memory, but the ability to do a root password reset that involves rebooting the box wouldn't be enough (and it seems like that may have been all the access that the attackers here had).
Agreed. This is in sharp contrast to the Media Temple / Plesk debacle. Which affected how many thousands of sites? How long did it take them to admit it? Meanwhile, Linode reports full details of a hack affecting no more than 8 accounts the SAME day.
It's good the fast acknowledgement, but I find it lacks in detail. I'd love to know exactly what kind of measures are they going to take so the chances of this happening again are truly reduced.
I share a Linode with a friend and at work we deal with 4 so this is a concern.
Further comment is indeed warranted, but that part can come in a few days, when there's more clarity on the scope of the incident and a fully-hashed plan of attack for closing the vulnerability.
I hope so. They said they will be "reviewing our policies and procedures to prevent this from ever recurring."
That doesn't imply they will be posting a public update. As a Linode customer, I would like to know details of how this breach occurred and what actions will be taken to prevent it in the future.
The breach originated on the Lionode customer service system. Which the attacker used to reset user passwords. Unlikely the incidents are directly related.
Only 8 accounts were affected. Do not worry. Minor breach. Not much harm done.
It seems to me the truth is the attacker looked for bitcoin wallets and emptied them. The fact he could identify 8 accounts and access them suggests the attacker could have accessed far more accounts if they wished. I think this is the most worrying thing about the breach.
I don't really understand how bitcoin works but it seems that people with wallets need to set up multiple wallets on multiple providers and limit the amount of bit coins in each wallet to limit any losses from breaches like this.
If I was a linode customer I would be thinking about moving. This message, while fairly open, doesn't give me much confidence there aren't other security issues with the platform.