Have they figured out how the attackers got in? Have they competed the forensic analysis to determine they can't get back in through the same or similar vectors?
There's nothing really here about what happened or how they have mitigated it. Perhaps it's too early and they don't really know, but then isn't going back live fairly risky?
Aside from a persistent firmware threat, which has nothing to do with the attack vector, I don't see why "yay we're on AWS now" is any different unless they know the compromise was due to a specific piece of infra which is no longer present. Isn't it much more likely to be a bug in their software logic which would certainly not be fixed by restoring a backup...
The investigation continues in parallel. First priority was safety, which meant shutting everything down to prevent any additional issues or destruction of evidence. Second was giving people the ability to access their funds and to trade - a complete redeploy of the infrastructure which took us working 22/7 since Monday until just now.
An investigation is underway and we have some internal speculation from the first bits of information - the real statements will come after we aren't guessing. This is the first minute we've had to breathe this week.
Could you talk a little about how you covered the first priority, safety? A restart on a fresh server doesn't really inspire the notion that whatever exploit they used can't be used again?
i.e. why did you decide to restart when you haven't determined what caused the safety issue?
Was the volume on bitstamp sufficiently large that a 19k BTC hot wallet was actually required? I only know enough about this stuff to ask the odd ( possibly daft ) question, but $5.2 Million strikes me as an awful lot to have "at hand".
Hi. Can you comment on the attack vector used to infiltrate bitstamp? was it a zeroday, misconfiguration? logic problem with their code? How did you discover and fix the problem before relaunching the server so quickly?
I'm confused - "an investigation is underway" - and investigation as to how they got in? or WHO did it?
I wish I knew more specifics about this situation but I'm starting to feel that you guys rushed the relaunch of the service post-intrusion.
The real question is whether Bitstamp is now insolvent. They lost $5 million. Did they get new funding to replace that, have enough capital of their own, or are they dipping into customer funds? Their statements have carefully avoided mentioning this issue.
If a real stockbroker lost funds like that, and became insolvent (debts > assets), they would have to stop operations immediately. In some jurisdictions it is a felony for a broker to continue to accept funds once insolvent. They don't get to "fix it later". That's because, historically, the temptation to fix it by speculating with customer funds has been a big problem.
Bitstamp now needs a full audit by an outside auditing firm.
So Bitfinex, pretty similar volume to Bitstamp, reports about 200m in volume in the past 30 days. A run rate of 2.4b.
Their fees are roughly 0.4% on a transaction. But note, the fees are incurred twice per unit of volume. i.e. they pair a buyer and a seller, who both pay a fee, thus you get a 0.8% fee on any unit of volume.
So we end up with a very rough revenue of $20m a year. Of course there will be various costs, but I've seen most exchanges run with teams of 6 core people, with another 6 or so support staff. I wouldn't expect to see much larger teams than 20 at Bitstamp, and given their cost-center for development is in Slovenia, I doubt you'll see them average more than $150k, which would be $3m per year.
So, together with $10m in funding not so long ago, together with substantial revenues ($10-20m per year), and together with the fact it's quite likely we're talking about early adopters (Company was founded in 2011, the year of 30 cent bitcoins, 1000x cheaper) who likely could pay this out of their own pockets, do they have what it takes to cover $5m? I think so. On that last point, if they had $5k of bitcoin when they founded the company, that'd be worth $5m today, and over $15m a year ago, for perspective. It's not at all unlikely they're sitting on $30m, but rather build a company where they offload their entire risk and give up 30% of equity. Who knows.
Does that mean an audit isn't necessary, not at all. It should be standard practice. Just speculating here on whether I think an audit would show a positive result or not, I think it would.
Agreed. Unless a tier 1 accounting firm (Deloitte, PwC, etc) comes in and audits their books, I wouldn't touch them with a 10 foot pole. Bitstamp is large enough and has been in business long enough that they may indeed have been able to absorb this loss. If that's actually the case though, then they should have no problem proving it. On top of that, the publicly released audit results would be a huge marketing tool for them.
The fact that they didn't do something like this and release it along with this statement makes me rather suspicious. It's not like they don't know that this is the first question on every single customer's mind.
An outside security audit wouldn't hurt either as they've lost the benefit of the doubt on that point now and their mentioned list of security improvements is all very hand-wavey.
Bitcoin exchanges aren't very profitable. The numbers for Mt. Gox are available now. [http://www.businessinsider.com/mt-gox-financials-2014-2] The numbers for 2013 are actuals; the future-year numbers were pure fantasy. For the year ending March 31, 2013, Mt. Gox, the biggest Bitcoin exchange in the world at the time, made $286,000. That's all. For comparison, the average profit for a single McDonald's location is about $200,000.
The Bitcoin exchange business has low commissions, and they're heavily discounted for big traders. Sometimes all the way to zero. It's hard for an exchange to raise prices. So exchanges tend to look for other ways to make money. Those ways usually aren't good for customers. Trading on one's own exchange and front-running are tempting. Mt. Gox probably did that; many users observed that, during busy periods, trades did not appear to be first in, first out.
That's why no major financial firm has entered the Bitcoin exchange business. It's not very profitable.
They generated $1.3m in revenues in a year with a 20% growth rate.Their growth rate was for 2013? 400%+, putting them at a runrate of about $5.5m at the start of 2014.
And the amount of bitcoin transactions has only grown since 2013. e.g. the record highest number happened just yesterday. Or compare wallets at the top bitcoin company right now which verifies users and bank accounts, Coinbase, which was at 700k wallets this time last year, now at roughly 2 million. Another 300 million of VC money was invested since 2013 into the bitcoin space.
I mostly agree on your larger point regardless, big traders get discounted (any source for that from any particular bitcoin exchange, btw?) and exchanges are definitely not gold mines right now. Look at Vault of Satoshi, just closed. It ran well but it just wasn't bringing in the type of revenue they could with the amount of expertise they had on board. They've switched focus to a Netflix app. But comparing 2012 Mt. Gox to 2014 says quite little, we're talking about a trainwreck of a company here in an industry that's seen 1, almost 2 orders of magnitude growth since 2012. If price is a good proxy (not really great, but sorta ok), the fiscal year in which Mt. Gox made the $1.3m revenue / $300k profit started with a bitcoin price of less than $5, for perspective. Things have grown somewhat since then.
I appreciate that Bitstamp made a statement, but anyone who trusts this is a complete fool. Trusting Mt. Gox was precisely how I lost money in its demise.
I was also called a complete fool by someone on HN just before I lost my money. The comment made me angry. Me, a fool? Yet it was true, and I should have listened.
Run away, don't walk. They lost 5 million dollars. Let that sink in before deciding to trust them with any money you care about.
5 million dollars is about the same as a Series A round. They lost the equivalent of an entire company of programmers. You could employ 50 programmers for one year with that amount of cash. And it was precisely your money that they lost.
For all you know, this could now be a game of "musical chairs" where the last person to withdraw money from bitstamp will be left wanting. Don't let that be you. Guard yourself.
By the way, Mt. Gox lied through their teeth to me, all the way up until their service shut off. Leading up to the disaster, I was in constant communication with their support staff, and they were very reassuring. "No, of course no one will lose money. Withdraws will reopen soon." Too bad the support people were themselves being lied to by Mt. Gox management.
So, no, a statement by a bitcoin service isn't worth the weight of the paper it's printed on.
EDIT: It's distressing that HN downvoted the parent comment to the point where they felt like deleting it. Please try to downvote less in general. The parent comment was simply relaying Bitstamp's statement.
As I read it, that's a promise to users, not a statement about their balance sheet. It does not preclude the situation 'Animats is talking about, where they currently have less BTC than they credit their customers with, but they plan to make it up somehow before it becomes a problem.
That doesn't formally answer the question, though the fact that they will respect all withdrawals does suggest that they are unlikely to be insolvent. For example, they could be insolvent but allow withdrawals up to the last 19k bitcoins.
I don't mean to suggest that they are not solvent; I am only pointing out that the language doesn't seem to perfectly answer the question.
Honestly, to all in long Bitcoin investors 5 million USD is practically chump change.
Real talk.
Simply put, it's pretty much assumed based on volumes and fees that they make that much profit anyway, but there are also things such as recapitalization.
5 million is not a nail in a coffin for a Bitcoin exchange.
Lastly and honestly, if these assumptions are wrong then assuming Bitstamp wants to continue business it would be easy to raise the money on slightly less favorable terms given the distress/desperation (assuming the previous assumptions prove false).
Unless you're speaking as a founder in the field, I'm finding this hard to believe. Five million dollars is about an entire Series A round. Or at least half of one. And a Series A round is supposed to last multiple years. Or at least a whole year. It's not chump change.
And I've heard you're going to have no luck raising money at all if investors catch wind that you need it to survive. "Slightly less favorable terms" probably means the deal doesn't happen. See e.g. http://paulgraham.com/pinch.html
But I have no experience at that, so I admit I'm going off of hypotheticals and what other people say.
EDIT: Also, that's five million dollars at current BTC prices. Bitcoin has only recently fallen to $300/coin. It was $600/coin not to long ago.
> We can assure customers that any bitcoins held with us prior to temporary suspension of services on January 5th at 9am UTC are completely safe and will be fully honored.
I, for one, am glad to see a Bitcoin story turn out with a somewhat positive outcome, at least from the users' perspective.
That said, the article says a breach resulted in the loss of ~19,000 BTC, or around $5.6 million. If they're honoring all user funds, does that mean they are just eating that loss out of their own profits, and perhaps try to file an insurance claim? Or were they actually responsible enough to set enough profits aside in cold storage self-insure the total amount floating in the "hot" wallets? On that note, I wonder if profits and hot wallet demands scale proportionately or not, or if their margins are just at a scale where this is a non-issue?
It is probably much higher. If we take a conservative estimate of 10k btc traded per day, on average, and a trade commission of 0.008% (0.004 two ways), we get to $5.6m in little over half a year. Now yes, the have some operational costs etc, but they've been in business for several (?) years.
tldr: back of the envelope calculations show that they must have made more in profit to-date.
For comparison, Bitfinex did 27k in the past 24h. And an average of 22k per day in the past 30 days. So 10k is indeed somewhat conservative. I'd personally guesstimate their annual fees at around $10m, including discounts for larger traders.
What isn't mentioned often is that I suspect they're personally somewhat rich, too. i.e. they were two people into bitcoin in 2010, started the company in 2011. Even if they didn't invest in 2010 which they likely did, in 2011 the price was still in the pennies. A $5k investment then would've been worth $5m today, or $15m just a year ago. Between the two of them, I don't think it's unlikely they've got quite some millions stashed away. And given they've had a $10m investment not too long ago (with a valuation in the tens of millions at least. e.g. Coinbase and Bitpay both have $150m valuations at $30m invested) it's not unlikely they're burning VC money or their own money just to cover these issues and keep an otherwise potentially very valuable (long-term) company afloat.
Mix that with 2 years of not unsubstantial fee collecting and a cheap cost center in Slovenia for the vast majority of their development work, and I think it makes a lot of sense that they can pay this off.
Of course all of this is speculation. The crazy thing is that bitcoin is so young and nascent that we don't even have basic oversight. For all we know the two co-founders could have stolen $5m from their customers, said it was a hack, and continued happily knowing they don't have to work again for the rest of their lives if something happens. Of course this makes no sense seeing as they fronted the loss, but this notion that we are assured by that, rather than say an audit, oversight, blockchain analysis etc, is a reminder of how far bitcoin has to go still.
The trading commission is 0,8% (0,4% two ways), and it is lower the more you have volume. In addition, they have withdrawal fees.
In addition to operational costs probably fraud is also a big cost, but it is very difficult to estimate how big it is for them.
And, they don't exactly need to take it from their profits, they can just do fractional reserve... They are a bank, as are any bitcoin services that hold money for the users.
They are not regulated as a bank, they cant "just do fractional reserve" which you entirely misunderstand. They would be stealing from their customers.
Doesn't just have to be retained earnings from past profitability, it could also be investor cash that would presumably be recuperated with future profitability.
To me this marks a turning point in the maturity of bitcoin - major breaches may still be a cost of doing business, but end users can finally hope for some limited insulation from such failures.
Perhaps not this particular event, but definitely this period was a turning point. Multi-sig is a great example. Here's a chart on the amount of bitcoin stored in multi-sig addresses:
Nine months ago it was sub 0.1%. Three months ago it was sub 1%. Today it's over 6% and growing rapidly.
Multi-signature is just one example of potentially radically improved security versus the single key methods we had before.
We've also seen in the past 6 months 3 companies come out with proper insurance. Circle, Xapo, Coinbase. All at a sub 1% cost. (Coinbase and Circle free, Xapo I think 0.12% annually). Combined with multi-sig vault products from Coinbase, which has done a lot of interesting security stuff. (the basic 2fa stuff of course, but I think also things like finterprinting devices to detect stolen credentials from users.)
And with that, we're also seeing enterprise-products come out which offer bitcoin management in a secure way. I wouldn't be surprised if such a company like BitGo starts marketing insured warm wallets to exchanges on a percentage cost basis, and let exchanges offer users to pay for the option of this insurance.
And we've seen more steps taken regarding the bitcoin ETF on the NASDAQ, which settles manually twice a day and allows (industrial) trading and liquidity but gets rid of the issue that Bitstamp and everyone else has: running a warm wallet on a server that automates transactions. The vast majority of Bitstamp's coins were in cold storage, but if you want to run automated transactions you can't do it on cold storage.
So this period has definitely been a turning point I think. Will be interesting to see things develop the next few years.
Apparently they were the ones moving the famous transaction of 200k bitcoins. So, yes, for them this breach could look like: "Meh, we've lost some pennies".
Wow they have multi-sig support now, that is pretty amazing. Would easily be worthy of its own article in other circumstances.
If I understand it correctly it means money can not be withdrawn without both the account holder and bitstamp signing the transaction. This means that neither can the money be stolen by hackers, nor could bitstamp run off with it. There are actually some bitcoins in the real blockchain "with your name on it", as opposed to you just having a claim on bitstamp.
I doubt that. Having the funds jointly controlled by the exchange and the user, and outside the full control of the exchange, would require each fund transfer to be represented as a transaction on the Bitcoin network, which is simply impossible - the volume of transactions on an exchange is a few magnitudes larger than what Bitcoin can currently handle. To keep up with that volume, exchanges must use an internal off-chain accounting system, which precludes the use of multi-signature (or any other smart contract based on Bitcoin scripting).
In addition, this would require the user to be online and actively sign transactions when a matching order is found.
I think that they most likely meant that they're using multisig internally to protect their funds, in a setup where all the keys are controlled by them.
Not necessarily. Bitgo uses a 2-of-3 scheme, where they provide retail account holders with a primary key, and a backup key. In the usual flow, you and Bitgo sign a transaction. If you need to recover, you can use your backup key.
Source: I am a Bitgo retail customer, but I don't have access to an enterprise account and cannot comment on their Bitstamp integration.
How would they turn on such a feature without each account holder generating keys specifically for those TxOuts and moving the funds into appropriate P2SH addresses?
It sounds more like, it means it's possible given the necessary end-user configuration to enable multi-sig, not that all coins are already benefitting from that level of protection.
Multi-sig transactions work as you describe but many darknet markets have had that option for some time. It's actually recommended you use multi-sig for every darknet purchase.
They have always been django/python and have never used PHP, please don't spread this kind of bullshit (unless you have solid facts to back it up. I remember checking the setup in 2012 and it was already django back then).
I think the conclusion from all this heists is that virtual money is much easier to steal. Next time you will see physical break-ins and so on. We've watched all those action movies where people risk their lives to get hold of $5M. Nowadays, you can do this remotely from the convenience of your home... but it will be easier for the old crooks to steal a key vs bags of cash, so, let's see how this goes forward.
It's trivially easy to secure bitcoin, it's just data after all, a private key, one you could write on a dollar bill if you wanted and secure no different from a dollar bill.
The difference is securing bitcoins in high-risk environments: live automated servers. But that only goes for a small minority of bitcoins as only about 0.5% of bitcoins are traded on a daily basis, the rest can be in what is called cold-storage which is way, way easier to secure than a bunch of physical money, gold, diamonds, whatever. The 0.5% is a bit like the security of wallets in people's pockets: not very good and not comparable to bank-grade security of those same people's savings in a bank. Or comparable to lone ATMs which can be raided.
Beyond that, bitcoins exist on an open ledger. While it may be hard to trace that now, we're seeing blockchain analysis tools develop all the time. Who knows what they might uncover later? I certainly wouldn't be excited as a criminal to leave a permanently recorded trace of every single transaction ever since the moment of the theft, with the prospect of decades of cheap supercomputers doing analysis of this open source data set which might reveal yourself. Bitcoin is great for pseudo-anonymity when buying por n online, but it probably won't stand the test of time in protecting you from every criminal investigation, a bit like a randomly generated pseudonymous email address gives a large extent of privacy, but not if you're a criminal.
It's not easy to secure it. You have to do two things very well:
1) Don't let anyone find the private key
2) Don't let yourself lose the private key
The more of 2 you do, the harder 1 gets and the more of 1 you do, the harder 2 gets. If you have only a single copy on paper, it's hard for anyone to find, but also easily damaged or lost. If you make a backup on a computer, a virus might find it, making the paper copy worthless.
Not only that, but from the moment you generate the key, you have to be completely confident about the security of the computer you're using. People recommend a clean install of OS from the original CD and never connecting to the internet. That's not impossible but not easy, furthermore you have to carefully destroy all record of it left on that computer. So unlike gold, not only do you have to keep it secure in the future, but you have to have made sure that it was always secure throughout the past. You can't transfer ownership to anyone else without them going through the same rigmarole too, unlike gold which you can physically give to someone and they can be confident it won't evaporate in front of their eyes one day.
I agree with you about blockchain analysis. Who knows what new techniques might be able to dig up. They might not have much certainty but perhaps enough to put suspicion on someone.
The thing is, I can smear $5M across say 5 physical locations in a way I can't with actual money. With actual money, the best I could do is put $1M at each physical location. With bitcoin, I could put a portion of the key at each physical location, requiring that you break in to all 5 to get any money.
BTW, in that title, the word 'spent' is used in the formal sense for blockchain transactions. It doesn't mean the money is gone or ''used up'. It really just means 'moved'.
After reading that, it looks like the person who stole this money is actually panicking and making a lot of dangerous mistakes. I guess they are in a state that's between terror and ecstasy. I doubt they had any sleep recently, and they're going to be looking over their shoulder for a very long time.
There's nothing really here about what happened or how they have mitigated it. Perhaps it's too early and they don't really know, but then isn't going back live fairly risky?
Aside from a persistent firmware threat, which has nothing to do with the attack vector, I don't see why "yay we're on AWS now" is any different unless they know the compromise was due to a specific piece of infra which is no longer present. Isn't it much more likely to be a bug in their software logic which would certainly not be fixed by restoring a backup...