We should have a Medium post about this soon, but we built a set of tools called Fleet, Atlas, and Vessel. The simple premise is that Fleet generates code for 1..n clusters. By code I just mean yaml files. Atlas is a git repo per cluster that holds the declarative state for that cluster. At any time it should be able to be applied to the cluster and result in the ideal state. And then vessel runs in each cluster and does a (I am oversimplifying a bit) git pull and kubectl apply -f. We can use these tools to do canary deployments (generate x percent this way, y percent that way in ingress rules) across a wide range of clusters. The goal here is really to minimize configuration drift and keep a large number of fairly ubiquitous (but not always so) clusters in order.
