I've just quit my job at Stripe and founded a startup to tackle K8S hard multi-tenancy, particularly for companies who want to "build a cloud" for their customers but aren't sure where to start. If anyone is interested please feel free to drop me an email! https://kubesail.com
Heya! Yes, bigger and more flexible plans coming soon! Templates are a brand new feature and we're working hard to enable Helm template support there - but eventually the "template search" will make its way to the main deployments page (and will get a looooot more templates!)
Nice-looking landing page. Please consider adding a newsletter sign-up so individuals can be made aware when email registration is added. If you're looking for other SSO providers take a peek at what Gitea has done.
Our public shared tier (Heroku-like offering) is on AWS for now - but we have a few advantages in that we can run reserved spot instances and use far less overhead than we would with virtual machines. We will more than likely move this to dedicated hardware eventually since we use almost nothing AWS has to offer other than compute, network and NVMe. Our free tier is better than AWS's ever will be, because although we might offer fewer resources (CPU/memory), our offer is automatically highly-available out of the box, and you share a server with _much_ better network and diskIO than AWS can offer, which means different but typically similar performance. We also don't up charge for services like Postgres and Redis (they use the same resources as any other "app" or container). Also we have an open API (kubernetes!) you can use on your laptop or dedicated gear!
But the more important "no in practice" is for companies who offer some value add - say for example Intragram wanted to allow you to run some code that manipulated your images server-side (or more realistically, think PagerDuty allowing custom web-hooks from their servers based on code you write on their site). Hopefully one day large companies that need a public cloud but cannot use Amazon for Reasons(tm) also see us as an option as well.
One final "no in practice" is due to our tool's ultimate goal to be to nicely encourage developers to build cloud native apps - and we believe high-available applications are actually "cheaper" than commodity-priced single-point-of-failure applications. See any website you use your credit card on that crashes all the time.
I'd like to think of it as "isn't this just a better Heroku?" instead of a more expensive AWS :P
Thanks Alex! Pau here, please do not hesitate to contact us if you are interested in our platform.
Our mission is to allow users to benefit of kubernetes whithin seconds. We take care of ingress, SSL certificates, backups... so you can focus on what really matters.
We are going to bill by resources/minutes wich is really interesting for CI/CD where you need a real cluster intead of KinD.
We are going live in a few days, join our slack or email me for feedback :)
What's the story of multitenancy in Kubernetes? Is it true that it does not provide sufficient isolation between services by default? Is this something that could be handled by configuration only or is it a bigger issue with Kubernetes design?
What's the best way to make Kubernetes the layer that isolates tenants in a multitenant platform?
As some of the other comments have mentioned, kubernetes is “multitenant capable”, but the questions come down to:
1. Do you understand all the primitives and how they interact
2. Do your use cases require some or all of these primitives be bypassed (host network, root containers, custom extensions / operators), which may invalidate most security
3. Are the human costs to manage 1 and 2 less than the human/infra costs to manage multiple clusters?
Many applications have a few assumptions that violate “safety on a single machine” - databases use raw block devices, processes run as root by default. The harder it becomes to isolate an app on a single machine, the harder it is to isolate that workload on a cluster.
In addition, the moment you install something above kube (helm, ingress controllers, operators that can read or write secrets across namespaces) you start to blur the security boundaries at a cluster level and thus make it harder to reason about who can do what. It’s a lot like the Linux kernel / userspace separation - everything secure is in the kernel, but if a driver or subsystem has a vulnerability that tends to compromise the whole kernel.
I would suggest most people run multiple clusters until you get to the point where you have cluster sprawl (and are paying for a ton of redundant / idle infra), and then start either with an opinionated distro designed for picking all the defaults for you (OpenShift, a few of the other vendors) or hire someone who can design a top to bottom policy (and even then you’ll want multiple clusters for failure isolation).
I mostly agree with you that using current tooling seems like patching holes constantly but it's getting much better with sandboxing and automation is key and of course, it's not for everyone but can provide lots of benefits with "low" effort.
There is a Kubernetes Working Group where the definition and implementation of soft multi-tenancy and any feedback is welcome.
I would agree with this. Soft multi-tenancy can work for some setups, until the tenant has a business need for something which could introduce a vulnerability and compromise nodes.
The answers you'll get will be some combination of "namespaces" plus an ever-expanding laundry list of configurations and knobs and dials and features and webhooks and options ... which you will need to apply uniformly and correctly.
To all your workloads.
Kubernetes was simply not designed as a multitenant system. Namespaces aren't the basis of multitenancy, for pete's sake, they're namespaces. Piling ever more extensions on top to try and plug all the holes just highlights it more completely.
The concept given above is where I expect Kubernetes will go, either explicitly (by creating a hard separation between scheduling and everything else) or by convention and tooling (we all adopt layers of indirection like Virtual Kubernetes). The latter seems most likely to me.
This will, as others are pointing out, only get more and more necessary as CRDs and Operators continue their long march to gather all the knowledge of your systems into their embrace. But it turns out that storing all the truths of dozens or hundreds of distinct systems, some of which conflict, many of which can't be fully walled off from others, means that your security posture may include a certain amount of humming loudly.
Something like Virtual Kubernetes splits the difference: you can retain Kubernetes-as-scheduler, reaping all the benefits accruing from workload consolidation. But you also get hard tenancy boundaries between workspaces, between Kubernetes masters being used as a skin over etcd.
Kubernetes is amazing. But this is one thing it got wrong. Just wrong and we'll all be wrestling with it for years. It would be like setting out to write a single-user OS, database or filesystem.
Disclosure: I work for Pivotal, we do some Kubernetes stuff, but these are personal views.
Pretty much everything you need to do today is done. Multitenancy was designed in from the start and is available in several levels of hardness (orgs/spaces gives you user-level multitenancy, isolation segments give you component-level multitenancy, multi-foundation gives you the equivalent of multi-cluster multitenancy).
As time goes on it will adopt more and more of Kubernetes, Istio and other systems as they emerge and become able to take over from its own original components. But you'll get the same experience throughout. And Kubernetes will converge upwards as people realise that "It Just Works" is actually, you know, a pretty nice developer experience.
But I am, having worked on and around Cloud Foundry for several years, hilariously biased.
Instead of K8s or Cloud Foundry, a practical “alternative, to really achieve multi-tenancy” might be:
- Since essentially no container platform was designed for hard multi-tenancy, use automation to stand up separate copies of a platform at scale, using a platform with low overhead in controller to worker nodes.
Consider what would be the “simplest thing that could possibly work” e.g. offering tenants an AWS autoscale group (with appropriate IAM account, user, role, security group perms) behind an AWS PrivateLink endpoint:
I say that, because most of these platforms weren’t intended for and are not marketed for hard multi-tenancy, until their engineers started getting asked the question by enterprises attempting it for regulatory/compliance/security reasons.
> Cloud Foundry. Multitenancy was designed in from the start...
This comment gives me pause, as CloudFoundry’s security model, approach, and engineering lacked hardness as of 3 - 4 years ago. We explored the multi-tenancy design, and found no guarantees, or even strong assurances, were brought by the platform.
The larger issue was we found CF’s security engineering assertions rather overconfident (flavor of the comment above). Warranted relative to other solutions without security features, but not relative to being hardened enough to trust in the face of bad actors, particularly malicious insiders.
My opinion is also “biased”, in the sense that unless a platform is doing work outside of “the server”, I don’t believe it can offer hard multi-tenancy.
Well, 3 years is a long time (though some of your concerns may still be valid). That said, the point of comparison is between the two platforms as they are today.
> unless a platform is doing work outside of “the server”, I don’t believe it can offer hard multi-tenancy.
I admire the work AWS and others are doing to harden the hardware base, but I'm not sure how software would contribute or not contribute to that line of defence.
I'll go with separate Kubernetes clusters. The cost is more systems to manage, but they will be simpler and pretty much identical, and some low but not zero resource waste with each cluster you add.
Ill add that there are few well established companies that have adopted this approach, AWS, Google and Azure to name a few.
In this approach, is a single VM node reused by multiple kubernetes clusters or is a node exclusive to a single cluster? If only the latter is possible then what is the best way to maximize resource utilization in cluster-per-tenant scenario? What happens if new customer wants his tenant - is automating of provisioning of an entire new cluster feasible?
Each vm belongs to a single cluster.
The best way to maximimize resource utilization is to size the vms according to the tenant load, generally you want to minimize the number of vms, as each of them add some overhead, but you don't want to waste unnecessary resources. You must have a minimum set of vms in each cluster to ensure high availability.
Automating the provisioning of an entire new cluster is very easy, at least for some people with the required resources. But even for a small shop it is possible to automate it using stuff like Jenkins and Ansible. It is still relatively sophisticated stuff. It is not rocket science but not an elementary school project level either.
Thanks, that was informative. Do you have a reference that you'd recommend about the details of how the big names that you've mentioned are doing multitenant Kubernetes? I'd love to read more about it.
Oh, sorry, that was tongue in the cheek. They are cloud providers, and as such they provide costumers with separate cluster.
The aks-engine repo let you peep to what goes behind the scenes, but this is not the kind of beast you will grow at home - it is a public cloud provider with global scale, with the expected level of complexity and engineering. It is available on: https://github.com/Azure/aks-engine
btw. in hard multitenant k8s you can even go as wide as using a "master" cluster that contains the control plane of another cluster. actually you need the right network cni for that.
It provides soft multi tenancy via namespaces. You can then use NetworkPolicy if the CNI supports it to manage what other stuff can communicate with the Pods in that namespace.
You can also schedule Pods on Nodes that are for a specific purpose (eg only used by a specific team or product) by using various scheduler stuff like taints and affinity/anti-affinity.
Or you can write a custom scheduler.
In general I feel like all the tools are there in Kubernetes to build a good multitenant experience but it's also just easier to run a number of clusters, especially if you're cloud hosting where the control plane is usually free and you pay only for the workers.
I'd say the amount of isolation to be sufficient, depends on the threat model. If you want some isolation but moderately trusted workloads (e.g. different departments in the same company) that's easier than the isolation required for "arbitrary customers can run any workload on a given cluster"
Multi-tenant, single cluster is tricky. You either need node pools per tenant, at which point why not run one cluster per tenant, or you need really good isolation of the workloads from the nodes, which requires PodSecurityPolicy or similar to prevent a breakout to the underlying node.
Also RBAC can be tricky in a multi-tenant setup as you need to be very careful with anything that would allow breakout to the underlying node or any cluster level rights.
File descriptors and inotify watches can still be exhausted. In somewhat older versions also pids.
If you allocate memory fast enough(if you're a best effort pod), oomkiller instead of kubelet evictions happen, which can be kind of unpredictable. Similarly you can fill up storage.
From a service point of view, you should be able to provide isolation between services using namespaces, istio and Calico. You can read about "Soft multi-tenancy" here:
That works well to isolate services/apps/teams from each other, but also allows services to be shared where required.
I think if you wanted true multitenancy it would be simpler to spin up a second cluster, especially if you are in gcp and controlling everything using a infrastructure as code solution such as terraform.
I'm not a K8s wizard, but have a little experience with it. I found there to be a steep learning curve with simultaneously learning RBAC, side cards, Helm, YAML templating, storage, logging, operators, service discovery, etc. There's so much hype around Kubernetes that it can be hard to sift through the noise to figure out what you really need to know. I found many people wanted to sell you on how "how all you have to do is...", but before you even get a basic service deployed, you can easily end up with a complex problem that is unique to your situation. And unique can mean you ain't gonna find an answer to your problem on the Google. That becomes even more of an adventure when you're trying to do something any a little non-standard because K8s doesn't expose an API endpoint to solve a problem you have. Now throw slicing up one cluster into N virtual clusters and... debugging that sounds like an adventure.
May I ask about the sources you would propose to go through for someone starting out with Kubernetes ?
I was able to (with examples) create a simple K8s cluster with dashboard and Im learning now about external dns. What else would be good to get to after ?
I mostly learned from the K8s docs and brute force trial & error.
My first recommendation would be to learn Kubernetes on the platform to which you'll be deploying, if that is possible & practical. For example, you can create a lot of work for yourself trying to debug the differences between Docker, Minikube, and Google Kubernetes Engine. Every provider supports Kubernetes in a slightly different way (in my experience).
My 2nd recommendation would be to put in as much time as you can learning to read logs/errors/messages. You'll learn a lot about your cluster(s) and how it is laid out by tracking down the errors the system is generating. They aren't always going to be where you expect them to be.
My last recommendation would be to not allow yourself to become intimidated by people who sound like they know more than yourself. The K8s hype machine is in full growth mode, and while K8s is very cool in many ways, there's a pretty darn good chance that someone who insists "all you gotta do is..." doesn't actually know what they are talking about and/or how what they understand maps to the nuances of your particular challenge.
Be prepared to grind on hard problems. Clusters of containers are still very much in like sailing to The New World. Few know it all that well, and those who do probably don't work with you :)
Create a cluster in local minikube, and/or set up k3s. Deploy an application and get comfortable with how to write and structure resource files. Deploy an Ingress controller from scratch (such as ingress-nginx), installing/managing Helm application charts; I found it mostly trial-and-error but learning concepts will come naturally once you need to implement them. Read lots of docs, and keep in mind various things change between Kubernetes versions. It's not magic or particularly difficult, just a few concepts + YAML files on top.
See also: SAP's Gardener. Their website reminds me of a spam landing page more than a valuable, robust project, so I'll link an official Kubernetes blog post about it: https://kubernetes.io/blog/2018/05/17/gardener/
EDIT: per the replies, it does look like a fairly different focus, my apologies, just got excited when I saw the virtualized control plane. (To save you reading the blog, Gardener launches a control plane in an existing cluster to manage workloads on a different, dedicated set of worker nodes.) It will be fun to think about k3v and multi-tenancy.
Gardener project focus is orchestration of multiple Kubernetes clusters on IaaS cloud providers, whereas k3v focus on running a dedicated control plane over an existing Kubernetes cluster.
Some overlaps but different project's goals.
Yeah, Gardener looks like it's meant to address one tenant on multiple k8s clusters, and k3v looks like it's meant to address multitenancy in a single k8s cluster.
> This allows one to take one physical Kubernetes cluster and chop it up into smaller virtual clusters.
What is the benefit, or difference of this over namespaces in Kubernetes? Namespaces provide resource quotas, and they work with RBAC too, the benefits listed in the README and their descriptions, they sound exactly what namespaces already provide.
The point is that it's unsafe to allow tenants ClusterRole / admin on a shared cluster, but this is needed for many CRDs and Operators.
The Operator pattern is getting more and more popular, and most of then need ClusterRole.
As the service provider (internal team, or SaaS provider), this is a liability. The aim, from reading the README.md, is to provide the ability for each tenant to be ClusterRole / admin within their own cluster, hosted in a larger real cluster.
k3s is on the way to becoming "production-ready", this appears to use k3s in the code. Perhaps it could be once it's proven out as something the community can use?
Why? Kubernetes is already a virtualization layer on top of a virtualization layer. Do we need more virtualization layers? All it does is add complexity and degrade performance.
- https://www.youtube.com/watch?v=ptOLV2wjQUw
- https://static.sched.com/hosted_files/kccnceu19/3f/kubecon-s...