Re: 7, if you use a separate discovery like zookeeper, then it will have its own idea of which jobs are alive and ready, and k8s will have its own idea of the same thing - they may not necessarily agree with each other. They don't even know the existence of each other. Maybe it would work, but it seems like more hassle than rewriting the offending part.
From what I've seen, Kubernetes is a quintessential Google product - it has a very particular idea of how jobs should be run, and the farther you stray away from it, the more it will cost your sanity. If that curtails the benefits of k8s for you, then yes, I think you should revisit whether you really need k8s, based on that limitation.
If we're talking about zookeeper for service discovery. The registering application can maintain a connection to zookeeper and ping every few seconds. So zookeeper has a very good idea of what's running or not, usually more accurate than kubernetes.
Actually, zookeeper has API to register only when ready and to listen to events/changes. Not sure kubernetes has equivalent stable API so might be hard to port over.
A rewrite might be a solution, except it's not because it's doomed to fail. We're discussing service discovery, which implies multiple clients and servers, probably managed by different teams and written in different languages. The odds of completing a coordinated rewrite effort are abysmal. ^^
Well. I am thinking out loud. It's a real problem my company was facing. We've got kubernetes clusters that are supposed to run applications and we've got apps using zookeeper that can't run in kubernetes because it breaks the service discovery. I will eventually have to hint people how to make it happen, after one year of kubernetes hardly going anywhere.
From what I've seen, Kubernetes is a quintessential Google product - it has a very particular idea of how jobs should be run, and the farther you stray away from it, the more it will cost your sanity. If that curtails the benefits of k8s for you, then yes, I think you should revisit whether you really need k8s, based on that limitation.
Just my two cents.