Call me crazy, but I tend to think issues like this ultimately serve, if handled well, to improve a company's public standing.
I royally screwed up the embed script testing for our very first customer resulting in their entire site not functioning and a panicking customer on the other end. I ended up writing a 3 page post-martem and apology. We debated a bit internally to be as open as I'd been, essentially admitting basic sloppiness on my part and promising to learn from the mistake if given a chance.
Fast forward 7 months, the same customer(originally acquired from a cold call) recently introduced us--without asking--to one of the top seed funds locally.
Happened to me. I made an SQL update without specifying the conditions on a live database (project running, surprise surprise, late). Our team has sent some flowers and chocolates too. I am going to do the same thing next time I screw up, and I will.
These days it is too much about the electronic and not enough of the "physical" in my mind.
In 1982 some nutjob went around to supermarkets and pharmacies and put potassium cyanide in Tylenol bottles. Johnson & Johnson responded by issuing a recall for ALL Tylenol products nationwide. This incident is anecdotally credited with having directly led to those paper or foil seals on over-the-counter pill bottles.
This isn't crazy at all, in fact it's been proven to be true.
The Marriott hotel chain found that guests who had something bad happen during their stay but which the Marriott fixed were actually more likely to return than those who had nothing bad happen at all.
Everybody makes mistakes, and everybody experiences bad luck.
If you only experience good times with a company, then you have no idea how they respond to bad times. They might be great, or they might be awful. You just have no idea.
If you experience a problem with a company, then you know how they respond to problems. If they do poorly, you'll avoid them. But if they do well, you now know that they handle problems well.
The time then comes to choose a company for something. Which do you choose: an unknown, or a company that you know handles problems well?
Replace the word "company" in "a company's public standing" with "well-established startup that's already known for good service" and I 100% agree.
If Twilio's overall service wasn't excellent, people wouldn't be nearly as willing to view this from the positive angle. If Twilio weren't a startup - if it were, say, SalesForce.com - this also would hurt more than it would help.
Fortunately, Twilio seems to know this. They're absolutely in no way resting on their existing positive reputation. Even though they've earned the trust and respect of so many customers, they're still treating each and every customer as if they must still continue to earn their trust and respect.
Word of advice to Twilio execs: when this is all done and over with give your PR people a bonus and/or a raise. They did a fantastic job here in a nearly impossible situation, and I'm sure the recruiters are circling like sharks.
Truthfully? Our PR guy is on his honeymoon. Not kidding.
Glad to hear you got the information you needed when you needed it - still a lot of work to do to resolve this to our customers' complete satisfaction.
After reading some of the comments posted on the blog, just wanted to give you a shout-out for your having to put up with douchey comments and responding in a humble fashion to them.
You're not crazy. I've found that a small customer service interaction early on (regardless of fault) helps solidify a good first impression. Not that I intentionally create problems...
Thank you very much for that - our work is still not done. Through the night we have identified what we believe to be the root cause and have reconstructed the timeline that produced the incident.
We still owe you a full technical post mortem of the incident and we are responding to all the servicing fee reimbursement requests at help@twilio.com.
In this case, I think one can. If they cause damages due to negligence (or malice), they are liable for those damages. Only the dumbest of dumb companies would refuse to reimburse overdraft fees in a case like this, because doing so will open them up to lawsuits from every single affected person, lawsuits which those people basically cannot lose.
I don't think they are, but it's the right thing to do. One would hope the banks would be understanding enough about the situation (haha, I know, right?) to reverse the fees. But even in that case Twilio is still causing you to, at a minimum, call your bank, wait on hold, spend 15 minutes getting transferred around to someone who has the ability to reverse the fees, etc. If they want minimal impact for their customers, they'll just reimburse the fees.
I played this game with Dreamhost and SunTrust back with their whole billing debacle in Jan 2008. Dreamhost didn't respond to my support requests to reimburse "non-sufficient funds fees", The bank manager at SunTrust said that regardless of the reason funds weren't available in my account when the other transactions posted, so the NSF fees that I was incurring were legitimate.
In the end I think I paid a little over $300 for their mistake.
I would hope they have a business liability insurance policy, in which cases it should cover the costs associated with this unfortunate mess, including any reimbursed fees.
Simple, honest, open and straight to the point. Nice to see that they recognise it's not just about the money, but the trust people had in Twilio. I'm looking forward to reading the promised post-mortem, because I can't recall reading one from them before.
It has been some time since we have had an incident this disruptive for our customers. Looking forward to delivering that investigation summary to you as soon as we feel we got it right.
Congrats to Twilio on a textbook example on how to handle a billing outage. While I hope to never to through this I do admire a team that can execute like this.
Cheers, from one telco nerd to your team of hackers.
And you can see status from two years ago! Google won't even recognize that gmail is down unless 5% of requests are failing (!!!! Try getting your work done all day if 1/20 actions fails.)
Interesting. I just came back from an event here at Lone Star Ruby Conference they sponsored (movie - they used Twilio powered service to pick the movie we watched at a local brew theater). Would have made for some interesting questions had more known about this problem :-)
It's a good question. As our service is delivered through a utility pricing model, most of our API requests are billing events - each call, text message or recording ends up having a debit to the account balance associated with it. As you can imagine then, our auto-recharge feature could be called any moment to make sure the balance doesn't run out.
That said, we do have the monitoring you describe in place which is how we were alerted to the incident and limited the damage to 1.4% of the account base.
Twilio should really consider a lighter gray color for their text on the white background. Right now the text is simply too legible still! Why, I can almost read it, if I crane my neck and look at the screen at just the right angle. What could they have been thinking? Hopefully they will take this feedback to heart so that future urgent messages like these will be completely obscured.
I've been a fan of ContrastRebellion, though I've lately simply taken to modifying websites' CSS that I visit often enough that it annoys me. E.g., I've swapped the border and content background colors on Hacker News.
For anyone looking for multiple (redundant) SMS provisioning, I have had not perfect but pretty good experience with http://fortytwotele.com/ over a good few years. I am not connected to them at all, just throwing the option out there. As a bonus, they're European, which means you are better positioned for compliance with EU privacy laws for non-US clients.
Call me crazy, but I tend to think issues like this ultimately serve, if handled well, to improve a company's public standing.
Though I suppose that depends on how one defines "well"...