On: Ask HN: Best “I brought down production” story?

[alfiedotwtf]

i remember sitting next to someone who screamed after realising he just did a ’DELETE FROM company' without a WHERE clause. Our database was way too big to backup, so we only had production, but luckily I had rolled out database logging in the inyerface that recorded all UPDATE and DELETEs a few weeks before the event.

[chasd00 → alfiedotwtf]

I had a DBA call me on my way back from lunch absolutely sobbing after fat fingering a semicolon before the where clause while logged in to Prod as root.

we got things restored and back online in a couple of hours. I let her go home afterwards, heh she had suffered enough.

[rrauenza → chasd00]

I was taught to also use a transaction and then check how many rows were affected before committing.

[sodality2 → rrauenza]

As was Tom Scott: https://youtu.be/X6NJkWbM1xk

[rrauenza → sodality2]

I wonder if archive.org was around enough back then to capture any of it.

[chasd00 → rrauenza]

Many mistakes were made prior in order to get her in the situation to begin with. The semicolon was just the icing on the cake.

[kryptk → alfiedotwtf]

When your database is too big to backup that's a lot like a bank being too big to fail. Unwise long term strategy.

[alfiedotwtf → kryptk]

Terabytes across 25+ databases not co-located and all being masters replicating to each other, in 2009. Not sure how you would have done it.

[mianas → alfiedotwtf]

I got myself into the habit of opening transactions before doing anything that could alter data by hand. Huge panic-saver.

Even earlier, I got into the habit of typing the where clause before typing the from clause. Not much of a change, but it feels more fail-safe.