(Sadly, I did discover the outage when I opened my morning read of Fail Blog / Graph Jam / I can haz cheezeburger / Oddly Specific / Photobomb / WTF Pictures / Ugliest Tattoos / Failbooking. Does admitting that mean I'm no longer qualified for HN?)
Why would any website which gets few million hits a month not have a redundant system is beyond me. If you consider the amount of money they make the cost of a redundant setup is really nothing.
I was under the impression that people used Wordpress VIP hosting so that they wouldn't have to worry about using a redundant system (meaning that Wordpress VIP should take care of it).
At a certain point (a point which TC passed quite a long time ago) you simply have to own your own infrastructure. That does not make it bullet proof, but it puts it under your own control and you can respond accordingly and better control the outcome of any outage.
I know that in Web2.0 everybody wants to think that they can outsource all the messy bits of running a website to the lowest bidder, but that approach has never worked.
When you get REALLY big (like facebook and google big) you move from a self-managed server in a carrier-neutral datacenter to building and maintaining your own datacenters.
For some reason people think it's perfectly normal that facebook builds its own datacenter, and yet also perfectly normal that a large blog (Techcrunch) take no ownership of the very equipment that makes their site go.
Really? You'd be hard pressed to own all of your own infrastructure if you're Techcrunch, especially for anything approaching the cost of the Wordpress VIP thing (or even Rackspace Cloud for that matter). You know that lovely self managed server in the carrier neutral datacenter? It's still going to go down when power in one of the datacenter's grids fail and their backup generators don't come on properly. The network's still going to fail when someone screws up a config and routes your traffic improperly.
Once you own it all, you're still banking on the fact that you can do a better job than some company that specializes in what you're trying to do.
You'd be quite a bit better off sticking with some "cloud" provider and putting up a static failover cache on EC2 or something. Even then you're at the mercy of your DNS provider (unless you're going to run your own anycasted DNS) and any misbehaving DNS caches.
Why would TC want to spend the money on a datacenter, staffing, bandwidth, etc for their blog when there are services out there that will do all of that stuff for them at a fraction of the cost? Private datacenters have outages as well, they wouldn't have been immune from outages if they were hosted internally. If anything, they are somewhat shielded from liability because everyone knew Wordpress was down.
It makes sense for Facebook and Google to build and own their own data centers because they deal with volume on an hourly basis that is magnitudes larger than TC probably sees in a week (or possibly even a month). Techcrunch is a blog. That's it. That doesn't require owning your own datacenter. It requires a hosting company that can keep the lights on. Wordpress is about as good as you're going to get in that situation.
Why would TC want to spend the money on a datacenter, staffing, bandwidth
I don't think they should/would do that. They are nowhere near large enough in traffic volume to justify that. But they SHOULD have more direct control over the servers that run their blog so that WHEN something happens (again, I never claimed that maintaining your own server made you immune) they can directly control the outcome.
Wordpress serves traffic live out of three datacenters. That is the redundant system. And I can say from experience that it's really, really hard to do that. Doing so well is probably the reason they've had such a good record for the last four years. An unexpected condition on the network that keeps the datacenters from talking to each other (I'm speculating based on their report) might have cascaded into a wide-scale outage. It's also worth considering how hard it is to cold-start a large site, if the outage had any impact on their MySQL servers or caches. We can't cold-start Wikispaces anymore (nights and weekends included) and we're only doing ~10M unique visitors a month -- WP has something like 300+ million.
It's not really critical to anyone for some blog to have perfect uptime. TechCrunch's advertisers don't pay on a CPM basis except possibly a few of the traditional banner slots.
I think that's how Google Reader works, or at least I hope that's how it works. They do the polling for the feed and then store it somewhere to serve to anyone who's subscribed through Reader.
This means that Techcrunch is down, and @arrington is gearing up for a nervous breakdown after his 4th or 5th provider switch in 2 years.
Eventually I would think that they would belly up to the bar and implement a redundancy solution that involves multiple solutions. Then again, I am an infrastructure architect, not a blogger, what do I know?
Multiple solutions at which level? WP is hosted in redundant datacenters, I assume they have redundancy most of the way down the stack as well. What do you expect Techcrunch to be able to do that the Wordpress people can't?
Techcrunch's problem is a far smaller and simpler one than Wordpress's. Four or five ISP switches now, and it's clear with each move that they are grasping at straws, hoping to just throw money at the problem and it will go away. Websites don't work that way.
Wordpress.com has a problem that is many magnitudes more difficult and complex than just keeping one blog available.
Techcrunch could do a far better job by keeping their TTLs low, choosing a secondary datacenter site for replication. In the frequent event of an outage they just perform a master promotion, change the dns records in order to get back online quickly.
You don't have a redundant site if all of your eggs are at layeredtech. Throw SoftLayer or Equinix into the mix.
I believe the appropriate response after multiple provider switches would be "fuck this shit, it ain't worth all the worry" and just deal with occasional downtime.
