I made a comment on an earlier thread about the security properties of hot/cold wallet and the security properties of separating the matching and settlement systems. (https://news.ycombinator.com/item?id=7340505) This incident is an example of the Bitcoin community's best practices "working." They lost 12.3% rather than 100%. That's actually a considerable accomplishment in Bitcoin, but not a success condition for most people who deal with money.
If they had the settlement processed asynchronously on a different system, likely with human interaction, they'd be able to say "Hmm, it seems like account #944325 has convinced us to schedule a withdraw of $250,000 more money than he has on deposit. Well that would be a really bad idea, now wouldn't it. Denied. Now I think I'll sip a cocoa while leisurely planning my review of our withdraw scheduling code."
This is the most fascinating part of the Bitcoin story: watching a group of people who are philosophically opposed to most elements of the modern economy discover, one by one, why all those elements exist.
I wish I could upvote this comment more than once.
Many complex systems are complex because they solve complex problems, not just because they have evolved over time from different simple conditions. You can't always (if ever) replace a complex system with a simple one. You can't escape complexity by coding around it. This is the attitude that killed Netscape and kills many Enterprise projects.
Totally agree, but would also add this general concept applies to many non-bitcoin startups (especially those that deal in "real world" services as opposed to pure software) as well, eg. Airbnb, Uber, etc.
The natural path of these "disruptive" startups is to start small and lean, ignoring a lot of the complexity of whatever market they are disrupting, and then a slow and steady snowballing as they begin to re-accumulate a lot of the baggage of the industry they came in to disrupt. In the end if you're lucky you wind up with something that is still better than the old thing, but not nearly as much better as the original hype around the new thing would suggest.
I hear this a lot, but I don't see much (if any) evidence this was ever a prevailing view. The impression I get is a general distrust of any centralised service, whether it's a traditional bank or a Bitcoin web-wallet.
I wonder if perhaps a similar thing is going to happen to the meal replacement products, such as Soylent and Ambro and others. Maybe in their attempt to "hack" human nutrition they, too, will rediscover why human nutrition research exists. Except perhaps with more serious consequences involving health issues.
> This is the most fascinating part of the USA story: watching a group of people who are philosophically opposed to most elements of the modern social-democracy discover, one by one, why all those elements exist.
The people who are interested in cryptocurrencies come from all walks of life and share different views on many topics, just like the citizens of a country. Nothing's ever black or white.
Perhaps "all" is an overstatement. An overhaul starting from zero may not be a very realistic or efficient answer, but it seems a little optimistic to suggest that there is nothing our financial systems do that couldn't be changed or dropped.
So you think we should never tear down anything because there's no possible way we could ever rebuild it better than it is? Our financial system couldn't ever be any better than right now?
Those are two different propositions. Tearing things down inevitably breaks vital things and should only be a last resort. But that says nothing about whether incremental improvement is possible.
However, nobody wants to put the effort into organising politically to cause improvements to the existing system. Because they don't trust anyone, and only care for the increase of their own wallet.
To be fair, most people rallying against fraud, over-regulation, corruption, <insert-pet-issue-here> in the financial industry generally aren't really hoping that they'll replace all of that with poorly written software.
> This incident is an example of the Bitcoin community's best practices "working."
It's also an example of all the best engineering practices missing.
I thought that checking the account balances and wrapping the withdrawal in a transaction was an introductory example of concurrency issues. What were they thinking, handling money while allowing negative balances?!
I know everyone is gleefully using the failure of three poorly-coded Bitcoin exchanges to crow about the many wonders of regulations.
However, has anyone stopped to ask why so many exchanges are poorly-coded? No, it's not because everyone in Bitcoin adores PHP. I've met some of the most capable coders among cryptocurrency enthusiasts. Go check out Conformal's btcd, or any of Jeff Garzik or Warren Togami's projects, or the amazing talents of the Bitcoin core dev team. (If you decide to denigrate the abilities of any of these Bitcoin/cryptocurrency developers in a response, please be sure to include a link to your own github)
No, it's not because there's no talent. Rather, it's because nobody who is competent in the Bitcoin world is willing to risk their hides or the welfare of their families on something they know the authorities will eventually crack down on. You see, competent people actually learn about the space, and the relevant regulations before jumping in. And they understand that -- absent several million in start-up funding -- there is no way to legally open an exchange at this point.
So by process of elimination, the only ones left to open exchanges are either scammers, or ignorant, incompetent coders who can barely code up a PHP site, or a very few brave, mostly-competent individuals who have the money and lawyers to at least (hopefully!) keep them out of jail when the inevitable crackdown occurs.
I know personally of several extremely competent entrepreneur-developers who have abandoned Bitcoin projects out of regulatory concerns. And I'm sure that's just a small sample.
So that's the reason for the prevalence of poor coders among Bitcoin exchanges. Your solution to this problem will of course be a function of your worldview.
My suspicion is that simply removing barriers to entry for more cautious folks would not effectively resolve this sort of problem. Methodical people who take their time in engineering ironclad software systems backed by rock-solid accounting practices don't get first mover advantage, and they might not have bottom lines capable of supporting the kind of fee structures that would allow them to compete effectively with faster, sloppier businesses.
An unstated major premise of the "wonders of regulations" argument is that regulations exist because sometimes the hand of government is needed to handle situations where the invisible hand is a demonstrable failure.
Perhaps, in the case of the most cautious developers. But right now, I'd be happy with just competent developers.
And to me, at least, it's pretty clear why competent, responsible developers are not participating in this space -- we're too frightened by possible government sanctions.
it's pretty clear why competent, responsible developers are not participating in this space -- we're too frightened by possible government sanctions
Where's 'rayiner? He'll love this.
Writing bitcoin software is like writing crypto. You need to get it exactly right.
But instead of starting with a spec written down that the crypto community tears to pieces, instead the developers eat their own dogfood. No, change that: they build critical infrastructure out of their own dogfood. All before it's ever been vetted by the really smart people.
I'd say competent people don't write Bitcoin marketplaces that handle real money for the same reason competent people don't write their own home-grown crypto and then make it a single-point-of-failure for their entire business.
The arrogance here is unfortunate. First, you start out with a remark reminiscent of a high school bully. Do you really need a co-conspirator to share in your attempt to ridicule someone?
Also, you're pretty uninformed about the history of Bitcoin. Satoshi did start with a spec, or at least a white paper (yes, the lack of a proper spec for Bitcoin has been a major problem). Satoshi presented the white paper to the cryptography e-mail list several months before releasing the software. He got some feedback at that time, and then he got significant feedback when upon his initial release, which he integrated into the protocol.
This list included many of the top cryptographers in the cryptocurrency space, including Hal Finney and Adam Back.
Finally, with the exception of the transaction malleability issue, the failures in exchanges have had nothing to do with cryptography, but rather with basic secure software development practices and architectures.
I'm talking about exchanges, not the protocol itself.
Every exchange seems to be a bespoke system and that is just begging for trouble.
They should publish their architecture, then publish their source code, and then make sure that smart people have tried as hard as they can to find all the weaknesses in their stuff, and then turn it into a business.
Of course, there are business reasons not to do that: someone else can use your verified source, someone else can get a first-mover advantage, you might not get the smart people to pay attention, you might not know when it's ever good enough, and (the more pernicious idea) is the worry that seeing the source will give the attackers ways of attacking your stuff.
They fail for the same reasons that secret crypto systems fail. I fully understand why they are doing it, but it's still doomed to failure.
Is it really just government sanctions? Or do regulatory cost and exposure to liability have something to do with it, too?
I think that in the USA at least, government sanctions are far from guaranteed - the courts have confirmed that BTC is a currency. What that means is that there is no longer a question of if you can legally start a Bitcoin exchange in the USA - it's a question of what hoops you have to jump through to do it. (And liability. . . I'm curious if the Flexcoin situation means we'll soon be finding whether a clickthrough EULA that says, "Not responsible for lost or stolen bank accounts" is enforceable.)
Now, perhaps the hurdles are prohibitively high. I'm more than willing to believe that's true. But if so, then you're being rather melodramatic to frame this in terms of "government sanctions". That makes it sound like people aren't doing it because they don't want to knowingly engage in illegal activity. It'd be more realistic to just say that competent people are staying out because the regulatory environment results in an excessive cost of doing business for anyone who wants to make sure they're operating on the up-and-up.
The hurdles may be high for a couple of guys in a garage, but since the US courts have agreed BTC is a currency, the hurdles are well known and well understood by existing players in the finance world. I suspect their reticence to get into BTC is a combination of limited customer interest, underworld associations (SilkRoad), and volatility. If those three things would go away (i.e. Lots of people want it to use for mundane things, and the price remains relatively stable) I'd bet Bank of America would be happy to get into BTC.
>It'd be more realistic to just say that competent people are staying out because the regulatory environment results in an excessive cost of doing business for anyone who wants to make sure they're operating on the up-and-up
That's exactly what I meant. Apologies if I made it sound like anything else. It costs millions to start up a money transmission business in all 50 states (like an online exchange would need to), and the federal government has been explicit that they consider a Bitcoin exchange a money transmitter.
> And to me, at least, it's pretty clear why competent, responsible developers are not participating in this space -- we're too frightened by possible government sanctions.
I don't think that's clear at all.
In fact I think bunderbunder has it right. Those who make the best products will require a large enough expense for formal design, implementation, secure hardware acquisition, physical security (i.e. no simply running your exchange on someone else's cloud without a lot of oversight), the works. This requires tons of time and resource investment.
Because if any part of that chain is improperly coded, designed, implemented, etc. it will eventually be exploited and you'll be no better than the Poloniex type exchanges of the world.
In the meantime there will be those "incompetent" developers you mention with a shipping product already on the market. And theirs will be much cheaper as they don't need to devote "Space Shuttle computer software" levels of development design and implementation effort. So you'll be both late to market and more expensive.
If you're talking about financial security here then you effectively need to be building a Bentley instead of a Pinto. But you'll be competing in that unregulated market with Pintos with a consumer base full of people willing to take the risk of driving in a Pinto instead of a Bentley they can't afford anyways.
And this has nothing to do with the government yet, either sanctions or regulations.
In fact this type of "tragedy of the commons" is exactly why there is government regulation. They help ameliorate the inevitable "race to the bottom" by artificially limiting where the bottom may be.
But government regulation probably won't help too much here since you can always run your exchange out of a country that doesn't care and people can make their transactions with whatever identity they wish.
Who knows, maybe the industry will self-create and self-adopt appropriate regulation as a market differentiator. But that still would open the question of who does the enforcement; if competitors discover their competition isn't actually following the regs then they'd be forced to "streamline" themselves and then the whole thing goes to pot again.
But here's where I disagree with you: it costs literally millions of dollars to get the needed state money licenses to legally operate a Bitcoin exchange[1]. FinCEN has been very explicit that they consider Bitcoin exchanges to be money transmitters.
So those millions are money that could go toward building the infrastructure you describe. Bitcoin companies like Coinbase who have finished their Series A have to spend their runway on licenses instead of developers or infrastructure.
It's a huge problem, whether or not people are willing to admit it.
But thank you for arguing in a rational and non-bullying/non-contemptuous manner.
Well as long as the cost is fairly applied to new entrants then I don't see that as being problematic per se (as after all, any new entrant will have to deal with the same selectivity). In fact that's practically the point, to ensure that the "fly by night" Bitcoin exchange shops that couldn't survive a theft of 50 Bitcoins without going under don't actually make it into legit business.
Yes, this is unfortunate for those devs out there who could solve the problem of making a good Bitcoin exchange if only the cost-of-entry were cheaper, but that happens in tons of other industries too (and not always due to the government), and it already has an answer.
I think deposit insurance is a better solution to the problem. Part of underwriting Bitcoin deposit insurance would be a code and design review and proof of reserves etc, and it would also allow reimbursement if it does get hacked.
For whatever reason, decentralised cryptocurrencies seem to be a very polarized subject.
As somebody in the financial markets regulatory space, your comment aligns with my own thoughts - there is a middle-ground: a balanced but essentially prudent view of Bitcoin and altcoins that is seldom seen in these threads full of naive ideals and cynical strawmen.
Another aspect of this middle-ground viewpoint: both of these polarized sides seem to be focused exclusively on fully automated Bitcoin-based systems. I find this highly unlikely.
If Bitcoin-based systems do become popular, there is no reason to imagine they won't involve some human (e.g. back office settlement systems with STP rules, payment tests and 4-eye reviews of breaches) and legal elements (e.g. declare your BTC addresses to your government tax office, KYC/AML compliance for exchanges, etc.) to solve certain problems that have been in the news lately.
"the auditing and security features were not explicitly looking for negative balances".
WTF, who are these clowns that purport to be running the equivalent of a bank?
It's like everyone running a BTC exchange either is corrupt or slept through the part of Databases 101 where they explained "this is why transactions are important, here is banking as an example". Seems likely both.
The people who typically make and run these things are sitting on the sidelines until the legal issues are worked out. It's not exactly surprising that the people willing to risk jail time for first mover advantages aren't the best at finance/development/risk.
Gosh, it would be nice to have some sort of assurance that the group holding on to my monetary reserves met some sort of minimum level of competency in dealing with quantities of currency. Or, lacking competency, some guarantee that any bank screw-up wouldn't result in loss of funds from my account...I wonder why no one has thought of this before?
Oh, wait...
Seriously, if I was an economics or law professor today, I don't think I could come up with a better classroom to teach financial regulation than the mess that is the Bitcoin economy.
A percent of every card transaction goes to cover system's losses on fraud. There's value in the costs being predictable, stated up front, etc, but it's not fundamentally different.
IMO that predictability makes it fundamentally different. I can handle a regular 2% fee, but I can't really handle a sizable chance that I'll be completely wiped out.
Exactly! I know how to fire a gun, tie a tourniquet, extinguish a structure fire, and set aside an emergency fund. I also know that paying someone else to relieve me of having to worry about these things is money well spent. You call it regulation, I call it peace of mind.
"We fucked up, so we're deducting the losses from everybody's accounts so that people don't immediately withdraw all their money."
If my bank sent me this notice, the FIRST thing I'd do is withdraw all my money - there's every chance it may be "an absolute necessity" to steal more of it at any time...
A better way to phrase it would be as a "Bank Holiday" as FDR did, or as a 12.3% "temporary withdrawal floor." It's a lot easier for a government to survive such an event than a small IT company.
Those things are called crises because they actually happen quite rarely. When such episodes become a regular occurrence people tend to abandon the system in question.
I keep seeing this attitude among Bitcoin fans. Of course, there have been many financial crises with regular currencies and banking systems. But that doesn't mean they're the norm. When you look at the number of crisis against the number of different financial systems and years-without-a-crisis it's obvious that they're exceptional. Sadly, the number of Bitcoin shocks seems to be increasing linearly with scale.
I don't think this is inherent so much as a problem of overconfidence and magical thinking. Right now Bitcoin service providers (in the aggregate) are like builders whose structures catch fire distressingly frequently, but who excuse this by pointing to famous fires of the past. The fact that fires have historically been a problem doesn't mean fire codes are useless; quite the opposite, in fact.
I agree that most Bitcoin service providers are total bullshit. Satoshi does backflips to create a decentralized system, and the first thing these idiots do it to try to centralize it in all sorts of ways. If the people doing the transaction can agree on a trusted 3rd party, then you don't need Bitcoin. But credit card fraud happens every day, large bankruptcies happen every year, and sovereign default happens every decade, because people are unreliable.
You're quite right, but we should view the incidence of fraud, bankruptcy, and default in the context of the economy within which they take place. Like there may be thousands of credit card fraud attempts every day within the US, but the number is fairly insignificant compared to the number of legitimate transactions by honest users. Likewise there are a few bank failures every month, but institutional failures where large chunks of depositors' money disappears are fortunately quite rare).
Which I'd imagine would be illegal for more than one reason... but then this is one of the costs of an unregulated currency (not saying that makes it good or bad, just that it's there).
It's also why I don't understand maintaining a balance with a service like this. Unless you're doing very small trades, the 0.2% fee plus the network charge for a withdrawal seems like a small price to pay to be mostly safe from this kind of thing.
> If you have 2 BTC, withdraw 10 BTC, and are left with -8 BTC, the software would see that you deposited 2, withdrew 10, and have exactly what you should: -8.
Nice, I wish my bank was so chill about having a negative balance. :)
This reminds me of the early Amazon bug, where you could add negative numbers of items to your cart, and it would credit your account, and then wait for you to ship them the book. ;)
My bank is completely OK with that. Most banks probably are. The problem is that then they charge me even more money for it rather than going bankrupt themselves. Talk about a bad response...
The comments on the bitcointalk thread are interesting. Here you have a bunch of people who have just lost 12% of their funds and most of them are totally ok with that. If you are looking for evidence that bitcoin supporters are driven more by ideological reasons than economical ones, here is a pretty good example.
Eh, there's a lot to be suspicious of, but people have gotten locked out of a portion of their funds, with a promise by this guy who they previously trusted to make it up.
Apparently it's only $50,000 ("only"), meaning no individual lost tens of thousands of dollars.
It's unfortunate that this guy being upfront with everyone puts him above average, but it does, so his business will probably continue, such as it is.
If BTC is going to survive there needs to be insurance against loss. An FDIC for Crypto.
The exchanges take a cut on every transaction, so Poloniex should have self insured for the first 3% (or what ever their transaction fee is). After all they made that money on the transaction.
They should carry insurance for the rest.
The 12.3% deducted from everyone's account is "wrong" in my view because Poloniex absorbed none of the loss, and kept its cut of the transaction.
The "right" thing in my eyes is for Poloniex to adjust minus their transaction fees.
A better solution would be for the Bitcoin Foundation (or some other entity) to certify exchanges and online wallets as being compliant with a well defined code of practice and audit them regularly.
Consumers could look for the mark as indicating that the exchange is a well run outfit rather than a bunch of cowboys.
Then you end up with what happened with the housing market in 08, ratings agencies rubber stamping things for profit. Why should I trust what they say?
I don't know enough about the current economics of exchanges.
A Currency exchange would typically not be a bank. They would have a lot more in transactions than they would have cash on hand. $4M in transactions on $500k cash on hand. They would have little ability to lose a customers money because the customer would have to be in the store when it was robbed.
A bank would not have all of its money in a place that could be robbed, so they don't have 100% risk (or 12.3%).
So for a bank the insurance is about .2% of transactions. For BTC it would likely be closer to 3% because the systems are audited.
This is not something that can realistically happen by itself - just as for 'traditional' banking, it would happen only because government regulation would force it to happen.
Look: every startup gets owned up somehow in its first year or so. If you think your company hasn't, I have an adage about the sucker at the poker table for you. I'm not sure how many people on HN really understand this, because every thread I read about Bitcoin companies having security problems features highly-voted comments expressing shock at how bad their security must have been. No: this problem is universal.
The difference between Bitcoin companies and cat sharing companies isn't fly-by-night operational practices (although in some cases, sure).
The difference is that when a cat sharing company gets owned up, you don't hear about it. Only a small subset of security compromises involve password hashes dumped to pastebin. A quiet security incident at a cat sharing company doesn't end that company; they patch the bug, (hopefully) reimage their servers, and get on with their lives.
Even payment companies have incidents. But (perhaps counterintuitively) incidents at payment companies aren't company-ending events. Payment companies don't hold bearer-bonds, for magic cards or anything else, on their servers. Attackers might get a few thousand credit cards out of the rolling transaction feed of a payment company. Why bother? Those attackers can just acquire massive dumps of credit card numbers from major retail compromises. Attackers report flaws to payment companies! They're better off trying to score a $500 payday from a bug bounty than trying to monetize those breakins illegally.
Not so at a Bitcoin company. When Bitcoin companies get owned up, whatever "hot" assets they have get taken. Bitcoin companies aren't stepping stones for attackers the way cat sharing companies sometimes are; they're the intended target.
It bothers me when people caution that Bitcoin companies are scary because they're "targets", because I think the people saying that only grok 75% of what's happening. It's not that Bitcoin companies get owned up because attackers spend more effort targeting them. No. Attackers expend effort on everyone's companies, and are usually successful. The difference is what happens after the attacker succeeds.
People considering starting (or funding) Bitcoin companies need to understand this. I feel like there may be a memetic belief that competent security teams can reliably stave off security flaws if they're just careful with two-factor auth and parameterized SQL queries. That meme is false. When you start a new software service, you need to build it on the assumption that you are going to get owned up, if not by a stupid password compromise than by a memory corruption bug deep in V8 or MRI or CPython or nginx that only a few dozen people in the world know about. It is going to happen. If you give custody of cash-equivalents to a startup, you should be doing it with that in mind.
Catsmolo: A social network to share your cat with your local neighbors. Mobile app enables you to browse current inventory or list your cat. Members can up and downvote cats, based on cuteness or difficulty.
Bitcoin companies create additional attention from criminals because people engaged in illegal activity are a targeted market segment. Near the median there's just ordinary getting away with something; an AirBnB rental in Milwaukee that takes bitcoin to facilitate reporting less taxable income. The long tail of bitcoin use probably leverages anonymity move money that facilitates the most extreme nastiness.
Using bitcoin is a strong nexus between an individual and the long tail and it puts the individual on their radar. The long tail is being nice when they only steal my bitcoin.
Perhaps any online Bitcoin service is going to have to expect a certain loss rate due to hacking and budget accordingly. Just as credit card issuers expect to charge off some percentage due to fraud. The trick is to keep that loss rate manageable. It's clear that many of the same features that Bitcoin users like about it also make it attractive to criminals.
I don't think you should handwave this. Companies have hot bitcoins because they need them to cover transaction volume. Their "hot" liability scales with their business. When you're six months old, a 50k loss (more accurately: the requirement to redeem BTC that start with a market value of 50k) kills your company (or the rational incentive to continue pursuing your company). When you're a year old, 100k does the trick.
This isn't a problem bitcoin companies outgrow; it's a problem that festers as the company gets more successful. Do you go out like Flexcoin did, or like MtGox? Either way: you eventually do get taken out.
I didn't mean that to come off as hand-waving. I think this is a very serious issue that is going to require a complete rethinking of security practices. Perhaps it won't ever be solved satisfactorily, but I wouldn't rule out ingenuity of developers to at least reduce the risk to manageable levels.
Well I think the problem is that if we take the assumption of "being owned" at face value (and that seems justified by the evidence) then "reducing risk to management levels" doesn't stop these disastrous events from happening. It only makes it more disastrous once the black swan finally lands in your pond.
It seems that the only way for this line of business to be feasible in the long term is for the hot wallet :: total assets ratio to be as low as possible. Your income to build assets is proportional to transaction volume, but so is the required size of the hot wallet.
Maybe a massive up-front investment to allow for start-up assets to be suitably large in comparison to hot wallet size... but even then you'd need to be careful not to grow too quickly and to ensure that you proportionally build up your reserves for when your hot wallet gets wiped out.
But this means that you have those stored assets that you can't invest elsewhere, so are you even making a profit now? The only way to reduce assets needed is probably some kind of insurance arrangement, but why should the insurance company offer low fees for this with the risk profile we currently see?
The loss rate is extremely high; I don't think people would be willing to pay an extra 20% spread on their exchange to cover it.
Much higher levels of isolation (such as running the transaction engine in some kind of HSM) would help, but I suspect that anyone smart enough and cautious enough to do it properly is too smart and cautious to go anywhere near the prospect of running a bitcoin exchange.
Most credit card purchases, like almost everything in the financial system, are reversible. Things that aren't reversible are the exception, and those are the places where, pun intended, the buck stops. If someone gets my credit card, they can't get $50,000 out of it and disappear. It's very very hard to buy cash equivalents via credit card, and this is not an accident.
Nothing in Bitcoin is reversible. Everyone working with them has to be hyper-vigilant, always, constantly, whatever you do don't blink, blink and you're dead.
I think if I were working with Bitcoins I'd get an ulcer.
For thousands of years the only way to pay for something was with non-revertible mediums, either cash or bartering for other goods. Reversible transactions are new in the grand scheme of things.
If I ran a bank and didn't bother to lock the doors or keep the money in a safe and someone comes in and steals the money, I don't get to do a chargeback and get the cash back.
> If I ran a bank and didn't bother to lock the doors or keep the money in a safe and someone comes in and steals the money, I don't get to do a chargeback and get the cash back.
That would be one thing if physical security were just as hard (or harder) than cyber security.
But it's in fact the complete and polar opposite; physical security is much easier and much better understood by the actors who need to engage in defense.
Seems to be a recurring theme recently. Evidently too many developers inexperienced in proper security are building these things. You would have thought after everything that has happened so far that people would take a long hard look at their security measures.
My (completely pulled out of thin air) opinion is that there is no way that startups populated by young, hip developers can build a proper system for handling money. You need to have a baggage of a thousand bizarre things how these systems can be violated, unless you want to repeat the same mistakes - and that comes with time and experience.
There are literally at least hundred thousand developers who have worked for 20+ years on financial systems - the industry employs a lot of them. If you're going to be storing money of other people, picking up a random such guy - even completely mediocre, boring one - would at least bring up the many issues that are taken for granted in 'that world' but nonobvious if you're not from the financial industry.
> My (completely pulled out of thin air) opinion is that there is no way that startups populated by young, hip developers can build a proper system for handling money.
At least not if they are populted solely by young, hip developers. Those developers can probably build the system, but what they can't do is specify and validate the system. So, those young, hip developers need to at least spend some time talking to some experienced domain experts.
Or, you know, keep repeating every failure in the history of finance and banking that has led up to the industry practices and government regulations they are ignoring. Sure, you can likely find better solutions to some of those problems, but it would be better if you at least "anticipated" the well-known, obvious problems and solved them, rather than repeating them first.
Moreover, these things should be built upon the certainty that somebody smarter than you, the coder, might be interested in free money, and that the software will be broken.
Only with that mindset can one build a system that doesn't screw over every legit customer when it happens.
Ahh, the default finance mindset is a bit different - first, assume that your own employees, managers, sysadmins and developers would be interested in free money. Design the system, processes, checks and audits according to that - and it covers most of the precautions against outside hackers as a natural consequence.
If you start a BTC exchange, write half of the initial code yourself, have access to the servers and own the company - then you should ask a simple question: could I myself steal funds undetected? If you're an investor, could the CEO/founder steal funds undetected? If the answer is yes, you have work to do.
There are some theft options by privileged people that can't be realistically prevented, but you can make sure that those scenarios would be detected within a day, and thus those privileged people simply wouldn't do it to avoid jail.
Security is hard, especially computer security in the absence of trusted/trustable systems. (Yes, it's a deliberate devil's advocacy, but it should be said.) We should ask ourselves why the existing government banking systems are secure. In part, it's because nothing is simply trusted and everything is double checked. In part it's because criminals have tried all manner of things, giving the system a chance to develop collective experience dealing with all manner of attacks. In part, it's because the law makes it a poor economic choice to engage in many kinds of criminal activity.
Then ask, how many of these things are true for bitcoin? Then perhaps ask, could I provide these things, and make money while doing so?
It's not just "inexperienced in proper security." People very experienced in security would still have problems with this, because the irreversibility means the slightest mistake is doom.
Writing bitcoin software should be like writing crypto: you aren't smart enough, so don't try.
The missing link for bitcoin is coverage under something like the Uniform Commercial Code- a system that specifies general principles governing transactions and the framework for their completion and resolution of conflicting claims. Caveat emptor is not a foundation for a currency.In practice bitcoin's very anonymity makes it a more attractive target for theft- bitcoins come pre-fenced. As Patio points out, the bitcoin community is inclined to build magic bullet software that assumes theft can be prevented and policies whereby the victims of theft are shit out of luck as in this case and Mt Gox. For an ordinary good citizen bitcoin is a bet that the people holding the funds are not only honest but the smartest people in the world or at least smarter than all the criminals.
Saying that a company has the most secure bitcoin system in the industry, even after proof it was the case, doesn't change the fundamental design tradeoff in bitcoin. Bitcoin offers anonymous possession in exchange for risk. Because the risk is high relative to normal forms of commercial exchange, bitcoin attracts rational actors for whom the rewards outweigh the risks and the distribution curve of interested parties has a tail that skews criminal.
The missing link for bitcoin is coverage under something like the Uniform Commercial Code- a system that specifies general principles governing transactions and the framework for their completion and resolution of conflicting claims. Caveat emptor is not a solid foundation for a banking system..
Eh, I'm not sure they are incompetent. They are, however, swimming in a sea where the slightest mistake is lethal.
Some people see these stories and think "ha, I could do better than that!" Other people see these stories and think "can I really be sure that I haven't made even one fatal mistake?"
All software engineers write bugs. All software engineers write security holes. For most of us, the fatal flaw doesn't irrevocably wipe out a bunch of people's life savings.
Some of these guys are indisputably incompetent with regard to the software and services they are building. While it's true that all software can have bugs, these are extraordinarily unsophisticated attacks that any non-hacker can exploit by, say, refreshing his browser in rapid succession.
That is, there is a sliding scale with regard to the level of competence imputed to, say, a certain type of bug. And, this wasn't a simple coding error (which, can be more easily forgiven). This was a fundamental oversight in the overall approach to the software, with regard to a critical operation. And it involves such basic concepts as transactions and race conditions. While eliminating the latter can be difficult to get right, it appears that they didn't even consider the fact that they could occur.
I am one of the guys in your group who thinks "have I considered everything?" In fact, I would be inclined to believe that I haven't. Perhaps that's pessimism or just realism, given what I've seen from determined hackers attacking my business over the years. So, I can certainly give a pass to oversights or errors. But, at a certain point, the nature of some oversights or errors are indicative of the fact that the developer(s) are not competent, at least with regard to the domain.
Any incompetence is not so much technical as commercial. The problem isn't bad code, it's lack of a backup plan- e.g. insurance and reinsurance.
Among bitcoin holders there seems to be a tendency to assume the technical sophistication of bitcoin mining and blockchains somehow perfects operations involving the buying and selling of bitcoin. The analogy I would draw is to notes which can be exchanged for gold and backed by a central government.
It can be the case that notes are reliably exchangeable for gold and also the case that a bank takes these notes for deposit without writing down the deposit in a ledger. Likewise the bank allows withdrawal of notes, again without writing them down. If An individual makes the decision to deposit notes with that bank based on the certainty the notes may be exchanged for gold, they have used the wrong set of facts as the basis of their decision.
At best the robustness of bitcoin is independent of the robustness of bitcoin businesses. My suspicion is that in practice their is a negative correlation because stolen bitcoin are largely unrecoverable and do not trade at a discount.
The financial system at larger scales is not terribly different -- you electronically transfer money and in the blink of an eye it is gone, and is only recoverable with the cooperation of the counter-party. Of course most banks have identity requirements, but identity proofs in most countries are wholly insecure.
If someone with ill intent has the ability to wire money from your account, they can do so with ease. Bitcoin really isn't that unique in that sense.
The real story in the bitcoin world is a recurring tale of gross, epic levels of incompetence, largely because a bunch of people who know some PHP have become the ones building financial systems. People who lack even the most rudimentary of knowledge necessary (this particular "exploit" for instance....good God. This is literally usually the very first "learning databases" lesson because it is so obvious of a weakness).
What they have in common: they deal in Bitcoins, they're not banks, they have a "o" and an "x" in their name.
Now if you're trying to bash PHP (or Ubuntu?) for banking software that does not take an exclusive lock on rows it reads then modifies and a daemon that does not check balances on withdrawal, try again.
It's a strawman, though. The problem isn't PHP developers per se, it seems to be a lack of awareness that dealing with money is in any different than dealing with, say, any other CRUD operation (see - Mt.Gox just casting things to floats, it's just numbers, right?.)
Just wait until people start announcing their exchanges and business written in Rails or Python, or "anything but PHP" but still turning out to have been written by webapp developers with no idea how to actually deal with currency transactions or basic security.
What they have in common is a bunch of keen individuals rushing in to cash in on the hype but in the process releasing some of the worst financial systems (if you can call them that) which I've ever seen because they don't follow the same strict regulations and testing processes that the rest of the financial industry have to go through.
The hacker discovered that if you place several withdrawals all in practically the same instant, they will get processed at more or less the same time. This will result in a negative balance, but valid insertions into the database, which then get picked up by the withdrawal daemon.
When I was a kid I imagined that banks were incredibly vulnerable to this.
Later I discovered transactions and thought my kid-self was just silly.
I've learned that my kid-self understood things pretty well, sometimes.
It's the same kind of exploit that plagued many web games in the past; it's just something most programmers do wrong, because it's not a requirement in most cases.
It's good that these guys appear to be very transparent and accept full responsibility for their error.
But, this is an outlandishly amateurish oversight. Race conditions and atomicity where balance-affecting transactions are concerned would seem to be one of the first considerations that pops to mind. That, along with sanity checks just before final execution of the transaction.
It's clear that there are a lot of people building exchanges with little to no experience in the financial/transactional software field. But, I am beginning to wonder how much experience they have building any software.
Let's not suggest this is just a race condition. Sure, it's a race condition, but whats the maximum damage you can do? c * your current balance, where c is a low number (10 is a very optimistic guess), since at some point the race concludes and you can't exploit it further.
But apparently they didn't even check or put a proper constraint into the database that your balance should be positive (and if you do find a negative balance, shut the system and investigate). Now that's not a "pretty stupid race condition", it's reckless.
This is exactly what Erlang would excel at implementing. Not that one couldn't do it wrong in that language as well. Same goes for Go and Clojure and a number of other languages with advanced concurrency features.
On the plus side, Bitcoin isn't regulated like a nation-state's currency, and the vast bulk of fiduciary institution law that nations have doesn't apply (or simply isn't applied) to Bitcoin exchanges. So when a mistake like this happens, nobody has to go to jail.
On the minus side, $50,000 of innocent people's money just got stolen, and nobody is going to jail.
I don't understand how so many people in that thread can be OK with this happening. Saying "at least you didn't lie to us" when your money is stolen is hilarious. I'd rather keep my money under my mattress than use bitcoin for anything right now.
The major problem here is that the auditing and security features were not explicitly looking for negative balances. They add deposits and withdrawals and check that accounts are in balance. If you have 2 BTC, withdraw 10 BTC, and are left with -8 BTC, the software would see that you deposited 2, withdrew 10, and have exactly what you should: -8.
Another design flaw is that withdrawals should be queued at every step of the way. This could not have happened if withdrawals requests were processed sequentially instead of simultaneously.
As some one who's almost entirely oblivious to the trappings of cryptocurrency... is it not possible to trade on sites like this with a local wallet? Or do you have to keep a balance with them separate from your wallet?
It's invalid logic to assume that because some bitcoin sites have been hacked, all of them will be, but at this point it seems pretty clear the entire community needs to be a little more careful. Until that happens I don't think keeping a balance with a service like this makes sense.
They need to have your money before they can execute a trade from you, otherwise they're opening themselves up for significant fraud. If it's trade-then-send, I could sell some bitcoin, then wait a few minutes to see whether the price went up or down before I send them. If the price went up, just refuse to send the bitcoin, and execute a new sale at the higher price.
You can certainly put money in right before you execute a trade, and pull it back out right after. But that introduces significant lag, up to an hour for bitcoin, depending on how many confirmations you wait for, and several days for USD.
That said, I don't understand why people keep significant balances in these places. Keep a small amount to trade with and save the rest yourself!
That said, I don't understand why people keep significant balances in these places. Keep a small amount to trade with and save the rest yourself!
Yeah, that's what I mean.
You can certainly put money in right before you execute a trade, and pull it back out right after. But that introduces significant lag, up to an hour for bitcoin, depending on how many confirmations you wait for, and several days for USD.
Thanks, that's definitely something to consider. I can see it being very inconvenient, though I still agree that the idea of keeping a balance larger than your trade volume in a service like this is hard to fathom.
I can only guess that people have become so used to solid financial institutions that present little risk for deposited money. Even non-FDIC institutions, like stock brokerages, usually have extremely low risk for deposited cash. If you're thinking like that, why not let them hold onto it? It's less effort. Of course, we see the answer to "why not?" here.
I am only so familiar with Bitcoin, but why not have an additional wallet per user of the exchange? That way its not possible to exceed the funds of that wallet.
Hold on, did anyone read their proposed "solution"?
"Right now, all markets and withdrawals are still frozen, and they will remain that way until the negative balance watcher is written and in place and balance deductions are calculated."
A "negative balance watcher" sounds like a horrible idea. Isn't this a solved problem? Atomic database transactions.
"not looking for negative balances."
If this caused the problem here, this shows that this is just the tip of the iceberg.
Someone should start an open source project for BTC Exchange's back-end.
If they had the settlement processed asynchronously on a different system, likely with human interaction, they'd be able to say "Hmm, it seems like account #944325 has convinced us to schedule a withdraw of $250,000 more money than he has on deposit. Well that would be a really bad idea, now wouldn't it. Denied. Now I think I'll sip a cocoa while leisurely planning my review of our withdraw scheduling code."