9. We had difficulty accessing our own systems because of the outage and the bypass procedure wasn’t well trained on.
Suggestion for future, learned from bitter experience: separate your control plane from your data plane. In this case, make sure that the tools you use to manage your infrastructure don't depend on that infrastructure being functional.
That way you won't have to remember how to use a bypass procedure -- it will just be your normal procedure.
Well, as with all things based in technical nuance, it depends on your definitions. Sure, control planes and data planes should be logically separated. But as you build and ship compelling products, your developers will gravitate to using well-built products’ (data plane) resources to build new products.
Imagine an IaaS cloud. It starts will Compute, Networking, Storage (block) and maybe Object Storage/S3. Next comes a fully-managed database product. The Database team may want to leverage the Object Storage data plane in the Database control plane. A year or two down the road, a team building a SaaS application will probably look to use the fully-managed database as it’s one less piece of infrastructure to manage.
To avoid or eliminate these types of delays in resolution, it’s imperative that the product team have a strong understanding of failure modes and dependencies. There’s a lot to be said for building completely isolated foundational services — it’s also a very expensive undertaking. Lastly, it’s possible to build out-of-band/break glass access without compromising security.
(I work at a global cloud but have no familiarity with CloudFlare’s internals.)
Suggestion for future, learned from bitter experience: separate your control plane from your data plane. In this case, make sure that the tools you use to manage your infrastructure don't depend on that infrastructure being functional.
That way you won't have to remember how to use a bypass procedure -- it will just be your normal procedure.