We once got paged because our app was down. But… it wasn’t. Everything was green and API requests where being processed.
After some digging and a lot of luck, it turned out that BT (huge ISP in the UK) and unceremoniously added our hostname to some internal blacklist, meaning it would never resolve, specifically and only for BT ISP users.
Getting it removed was non-trivial, and it as only through complete luck that an employee had a friend who worked at BT and was able to escalate the problem internally. Without that connection we would have been screwed, as there is nothing we could find about this blacklist on the internet or how to contact them about it.
After some digging and a lot of luck, it turned out that BT (huge ISP in the UK) and unceremoniously added our hostname to some internal blacklist, meaning it would never resolve, specifically and only for BT ISP users.
Getting it removed was non-trivial, and it as only through complete luck that an employee had a friend who worked at BT and was able to escalate the problem internally. Without that connection we would have been screwed, as there is nothing we could find about this blacklist on the internet or how to contact them about it.
Terrifying.
Sort of a bug, I guess? Maybe with BT?