Good work Patrick. I'd point out another, tricky source of failure: crappy dns hosting. It's worth it to buy dedicated dns hosting from two different providers. The cost is about 50 bucks per year.
That is one I wouldn't have thought of. My personal experience -- and the standard disclaimer applies -- is that when I can't reach DNS it is because of a networking issue that would make the server unreachable even if I knew the correct IP address.
Still, thanks, I'll pencil that onto the "thing through when not 3 AM" list.
It's more of a chronic problem than a sitedown problem. You might be surprised how crappy some hosts and resolvers are.
Using dual 3rd-party DNS does give you one very good advantage in acute situations, though. When your host goes down hard you can still repoint domains, receive email, etc.
In general I recommend dispersing services: put your company's blog on a subdomain hosted somewhere else like Weebly, and use something like Google Apps to handle your domain's email routing.
I think having two DNS providers is too much of a hassle.
I use dnsmadeeasy.com (which is very reliable and inexpensive) and it works very good for my web sites.
If you think having two DNS providers is a hassle then you are probably doing it wrong — if you setup a hidden primary configuration then adding an additional secondary is just a matter of giving the new provider your zone name and IP address and then updating your delegation.
In case you're not familiar with it, a hidden primary configuration is just a master setup to allow zone transfers with the zones themselves not including a delegation to the master. This means the only queries your master serves are zone transfers requested by the secondaries.
The only other smallbiz-grade service I've used is DNSPark. They are decent but their management interface is painful and has not changed in 7 years (!).
been happy with it, and they have an easy to use API so if you are doing ~cloud~ things, you can script adding DNS entries etc quite easily from most programing languages.
I really liked the concept of this article. I've been impressed by your other posts and just wanted to thank you for the insights you provide. As I get closer to launching my own ideas, I can't tell you how helpful it is to learn (or attempt to learn) from the experience of others like yourself.
A side note - I noticed you're using slicehost. With the "cloud-y" providers available (Heroku, AWS, etc) would you ever consider a move to them? Could that help eliminate any of the moving parts, or just relocate them?
I don't have personal experience with Heroku or EC2.
My feeling is that, coming from my background (system engineer: I can do basic Linux administration but I don't breathe it) and at my scaling requirements, EC2 is more complexity versus "as simple as dedicated" VPS providers like Slicehost without any compensating advantage. Heroku seems like it would potentially be amazingly useful for folks who swing more programmer than systems engineer, and quite possibly for me as well, but I just have never gotten around to trying it out for a test drive.
For historical reasons, most databases are made/configured around the assumption that it is better to fail loudly and completely rather than fudge the ACID guarantees silently.
It's the opposite, actually. The default install of modern databases (Oracle, MySQL, and Postgres are all guilty) is to silently corrupt data rather than to abort transactions. If you want data integrity, you have to explicitly ask for it:
Presumably this is because people get really scared when transactions fail. (Except that's the whole point of transactions -- optimize for success, and clean up a few failures here and there.)
Also, most NoSQL databases are fully ACID. OK, CouchDB isn't, but nobody uses that anymore, right? What people "give up" with NoSQL is the strict structure of the relational model. Things like sets and transactions still exist -- it's just that sets can be polymorphic, or you can use some other collection structure!
Your last claim is not true. Most NoSQLs don't have transactions thus can't have any ACID properties. ACID is all about transactions in the first place.
BDB and Tokyo Cabinet have transactions, and so does everything built on top of these (KiokuDB, Memcachedb, etc.)
Mongo, Couch, and Cassandra don't have transactions over the entire database, but documents must be self-contained and must not refer to other documents. Following those rules, operations are atomic, consistent, isolated, and durable. "Useful" is not one of the ACID qualifiers, however :)
Some organizations get checklist happy and make checklists for procedures
which essentially can’t fail and which require no judgement. Those shouldn’t
be checklists — they should be shell scripts.
That's in principle the right thing to do, but shell scripts have this nasty habit of accumulating complexity, to the point where you end up with, say, a backup-and-restore process that everyone is afraid to change the slightest thing about, because nobody actually remembers exactly how it works (even if you wrote it yourself!), and the bash script has grown to be some opaque monster.
Maybe it could work to insist on very well-commented shell scripts, so well-commented that someone could use it as a checklist just by reading the comments, if they needed to. But then the comments have to stay in sync with what it's actually doing!
If you do this, you can escape the time or notification limitations which the notification services use to segment their customers into “hobbyists” and "enterprises who have an awful lot of money to spend to make sure nothing breaks."
Another free alternative to Mon.itor.us and Pingdom is my own http://wasitup.com - monitors up to 10 of your sites in 5 minute intervals, takes 10 seconds to setup.
Great article Patrick. In particular, I liked where you mentioned doing external API calls asynchronously. I hadn't thought of that in the past and I know I'll be doing that in the future.
