On: Data Loss at GitLab

[rdslw]

Just to stress one thing, while mentioned in their report, most people don't comprehend:

Gitlab was _one_ (last!) final step from complete data loss of everything. One. At that night, there was quite a long moment there had only one copy (and 6 hours old). Every other backup was missing/notworking/deleted.

This is scary.

[crististm → rdslw]

This is the fact that deserves more attention. The operator did a manual backup before work which was the only available one.

I find this to be an eye opener. The real problem was dodged by doing the right thing in the proverbial last minute (6 hours).

[s_kilk → crististm]

In other words, there is an alternate universe in which Gitlab is now categorically dead, all because that employee didn't make that single backup.

[garaetjjte → rdslw]

Didn't they had two copies? (staging server and LVM snapshot)