I told the Coinbase people to use Postgres for this two years ago, and they treated me like a crazy person for it. Assuming they arrived at this through my advice and not the inevitable catastrophe, glad they finally took my advice to heart.
Incase anyone's interested in being a developer at Coinbase, I'll save you some time: they're more interested in pedigree and whiteboard puzzles than science and code. Most disrespectful interview I have ever had in my 13 years of coding, and probably ever will have.
I think you've been downvoted because that comment has little to do with the topic at hand and seems like a shameless promotion of BitGo. Alternatively, you could create a separate "Show HN : BitGO -- one of the few multisig web wallets available" thread and that would be totally acceptable.
You have an issue right now. By running EventMachine, you are blocking Ruby's ability to do concurrent IO using Thread, which means that you've reduced the entire ruby infrastructure (except the EM async code style that nobody programs for anymore) to a single blocking IO thread, dependent on 10-15k+ lines of awful C code that haven't been under active development in over four years, which breaks the premise of ruby's synchronous development methodology, is extremely difficult to debug, and which everyone agrees (including the maintainers) needs to be obsoleted.
Indeed, there are a few maintainers on EM, for the sole purpose of preventing it from breaking every single time the ruby team releases an update, because if it broke, it would cause a problem with legacy code in the ruby space. If the EM maintainers walked away, Coinbase would be stuck on an old version of ruby until they themselves patched EM to solve it. EventMachine is not something people should be basing new projects on, especially projects that involve money.
Tony Arcieri wrote nio4r (https://github.com/celluloid/nio4r), which solves this problem in a far better way, and then used it to implement cellluloid-io (https://github.com/celluloid/celluloid-io), which doesn't choke ruby's ability to do concurrent IO using it's internal threading mechanism. Combined with JRuby, you get even more than concurrent IO here like with MRI, you can also get real threading, which is important in a space where CPUs aren't getting faster, they're getting more parallel. Node.js has the same problem, but unlike Ruby/EM, JavaScript is actually designed to be async, so at least everything is designed to expect it.
Because Sinatra (like all non-EM ruby code) uses the threading style for concurrency, as soon as you start running EventMachine, everything will block for a single request, even if it's slow IO bound. You can demonstrate this very easily by putting this into the code:
get '/blocking' do
sleep(10)
end
get '/notblocking' do
'ok'
end
Hit /blocking with curl, and then hit /nonblocking with a second curl command. The second request is blocked until the first one completes. Now try the same thing without eventmachine, and it serves the second request concurrently without a problem. That's what you lose in this exchange, and it's a big thing to lose. I'm using sleep here, but the same blocking will occur for database queries, remote cache hits, HTTPS requests (to get things like, say, the exchange rate), everything involving IO.
The only way to get around this problem and still use ruby synchronously is to resort to crazy hacks, including a fairly popular project I created a while ago called sinatra-synchrony (https://github.com/kyledrake/sinatra-synchrony), which wraps async code using fibers. But in order for that to work, every library you're using has to support it, so this path would require a massive refactor of the entire ruby ecosystem, and that's simply not going to happen. It's also incredibly difficult to debug, and there's a lot of bugs with EM - sometimes things would just lock up or segfault, or ruby would spit out an error stack that had nothing to do with the actual exception that occurred.
I tried very politely to explain this problem to Coinbase about two years ago during an interview, and their reaction was to ask about my college degree, give me a bunch of weird whiteboard problems that had nothing to do with actual programming, and then show me the door, one day after I helped fix all of their client libraries when they switched SSL CA providers and didn't even realize it, and then provided a pull request to coinbase-ruby providing concurrency support using this very same threading mechanism, on the two-hour flight there (proof: https://github.com/coinbase/coinbase-ruby/pull/6)
I tried very politely to explain this problem to Coinbase about two years ago during an interview, and their reaction was to ask about my college degree, give me a bunch of weird whiteboard problems that had nothing to do with actual programming, and then show me the door, one day after I helped fix all of their client libraries when they switched SSL CA providers and didn't even realize it, and then provided a pull request to coinbase-ruby providing concurrency support using this very same threading mechanism, on the two-hour flight there (proof: https://github.com/coinbase/coinbase-ruby/pull/6).
Are you being fair to Coinbase? There are two sides to every story.
But if this is true at face value, then it's disappointing to hear that a YC startup would care at all about a degree or about whiteboard problems, of all things.
If a company cares about what a programming candidate can do, then caring about a degree or whiteboard problems is caring about the wrong thing. Thought experiment: How much better or worse would YC's results be if they had candidates solve whiteboard problems during the interview? (Admittedly, YC cares about degrees though.)
Not surprising to me about Coinbase. A friend interviewed there and decided not to continue with the process as he was so turned off by their team. In particular he made a suggestion for improvement and they got very defensive.
I sure hope you landed a better gig than what they could have offered you. Thanks for the free education-- I'd be kicking myself if I was the dude who didn't hire you!
I feel like this pissed you off a lot, but a lot of hiring in startups has nothing to do with people's competency, and more to do with culture fit which is generally totally out of your hands and not indicative at all of your ability.
Also, people screw it up big time, often it's totally random. Maybe they ate a bad sandwich for lunch and didn't enjoy the interview as a result. Facebook famously turned down WhatsApp co-founder Brian Acton back in 2009, even though they went on to acquire his company for 19 Billion 5 years later.[1] Talk about making lemonade from lemons.
I think all you can do with these things is use it as a pause to think about how you can do better, and if you can't think of anything just chalk it up as a random event, like a bird crapping on your windshield.
Sometimes I feel "fit" is really important in hiring someone - other times I reckon it's a good excuse for my in built prejudices that I cannot be bothered to overcome.
A good HR manager will be tracking "fit" by rejection and skin colour, gender and university.
That said, coinbase etc should be looking at the technology used - but we should not. In the same way we do not look at multi national banks by their tech stack but by their regulatory regiemes, their liability to asset ratio and so on
In other words, if bitcoin is going to take it's place amount the worlds currencies, it will be handled by things we think of as banks with asset backing and more. Probably gaming companies and telcos.
If multisig applies to a future product and not the one that was released then it sounds like he was being factual with you. Not sure what you expect - for him to be non-factual?
As for high risk cancellations, do you expect them to approve transactions that they expect will actively damage them? Of course the cancellation serves Coinbase - nothing else is reasonable. (It's also funny how you only hear about complaints of that when the price is going up - when the price is declining and they cancel due to high risk, nobody complains because Coinbase ended up saving them money.)
It's funny how everybody who didn't get a job there comes out of the woodwork to post critical comments on something completely unrelated.
>As for high risk cancellations, do you expect them to approve transactions that they expect will actively damage them?
The point is to make clear is that Coinbase is not an exchange like Bitstamp et al. Coinbase will cancel your transaction /only when it benefits Coinbase/. Buy bitcoin and then USD/BTC crashes? Coinbase is Johnny on the spot to complete your transaction. But, when the opposite happens? Sorry: You are High-Risk.
Regardless. Olaf did not know about Vault including multi-sig. This is /after Coinbase announced publicly that Vault would include multi-sig in the near future./ To me that signifies a disconnect between those in the company.
That doesn't really signify anything. If you made it sound like multisig was in the current version, and he rightfully didn't know anything about multisig in the current version - because, in fact, it was not the case - then you are wrong. Obviously no one can validate your statements but you, so we've got a one-sided story from a 1-hour old account from a person who didn't get hired. Clearly this account will represent the pinacle of objectivity.
Also, since when does every employee know about all details of all products? If "olaf" is "head of risk" then he's probably not an eng and in the know of all that. At Dell a couple months ago people were like, "Oh, we make that tablet? I didn't even know." And that's hardware.
Everyone knows Coinbase isn't an exchange because there are not limit orders and an open book. This is dummy-obvious and they don't try to obfuscate it.
> Buy bitcoin and then USD/BTC crashes? Coinbase is Johnny on the spot to complete your transaction.
This is factually false. I'm not sure if you read my comment. Coinbase has an equal amount of high-risk cancelations when the price is falling. You never hear about them because people don't have a reason to complain about it.
I mean if you feel jilted because you didn't land a spot, go be grumpy about it if you want. Not sure why they would hire someone for a support role who is coming in with a grudge about their cancelation policy. How would such a person be able to be trusted helping others with that issue? Bitterness doesn't justify FUD.
That's completely fair. Consensus systems are very difficult to (re-)implement correctly but I would argue there is value in doing so. There is also great risk and we appreciate that.
This time we took a slightly different approach. Whenever possible, we directly ported the validation code from bitcoind. Granted, the script runner is the most fragile part and for that we are relying on bitcoin-ruby which does not directly port it from bitcoind.
While we are passing Matt Corallo's brilliant test suite, it is not currently recommended that anyone use Toshi in production for the purpose of managing bitcoin. And if you are going to run it in your environment we highly suggest running it behind trusted bitcoind nodes. There are many pieces still missing -- like all of the DoS penalty code. We currently have the demo running in the wild for the very purpose of giving helpful people the opportunity to break it.
The only instances of it we having running are the ones listed in the blog post (and just perhaps a Litecoin version that we're trying to get to sync.) They are in no way currently integrated with the web site.
I would encourage people to really read this post and try to reason about the implication for the Bitcoin "protocol" the claim that the Bitcoin Core software project is, and must always be, the only usable instantiation of it. (n.b. While this claim is extraordinary I don't believe it is actually wrong.)
Currently we're saving every block we see. And if we find an orphan's parent which is on the main chain we try to connect them both. We aren't doing anything to explicitly avoid them.
If you're referring to side chain blocks (which some people refer to as orphan blocks) we'll extend them to be the main chain should they win or leave them forever marked as side chain blocks. We aren't doing anything to explicitly avoid these either.
We're mostly a ruby shop here - so using what we know is more efficient and allows more engineers to contribute (both internally, and from an open source perspective).
What advantages does running a Toshi node hold over using one of the many cloud services that allow free public API access to basically the same endpoints?
This is another cloud service that allows free public API access, but it's the first one I'm aware of that is also open source. Much better to build a service or business on something you are in control of.
The next problem is that they are still using EventMachine. Still think I'm a crazy person? Oh look, it's like I'm a well-received field expert on this topic or something http://www.slideshare.net/mobile/KyleDrake/hybrid-concurrenc...
Incase anyone's interested in being a developer at Coinbase, I'll save you some time: they're more interested in pedigree and whiteboard puzzles than science and code. Most disrespectful interview I have ever had in my 13 years of coding, and probably ever will have.