On: Postmortem of database outage of January 31

[gr2020]

Reading this, the thing that stuck out to me was how remarkably lucky they were to have the two snapshots. The one from 6 hours earlier was there seemingly by chance, as an engineer had created it for unrelated reasons. And for both the 6- and 24-hour snapshots, it seems just lucky that neither had any breaking changes made to them by pre-production code (they _were_ dev/staging snapshots, after all).

I'm glad it all worked out in the end!

[sytse → gr2020]

We too are glad we had those snapshots. And while it was the worst thing that ever happend at GitLab it is humbling to know that it could have been worse.

[animex → sytse]

What do you think would have happened if you had total data loss/failure?

[sb8244 → animex]

I don't work for gitlab.

Isn't that a put you out of business moment? I guess there is enterprise installations but the main website would be toast.

I guess there is the chance there were still older backups taken intentionally or by chance that would have been worse but recoverable from

[gtaylor → sb8244]

Most of their income probably comes from customers that run their own GitLab Enterprises installs. This would have really sucked for all of their non paying users, though.