Kubernetes is one of those rare cases where the industry is rallying around a technologically sound platform that emerged from the trenches, and not something that was designed to create jobs for consultants. It's great. Right now, there's a bunch of fairly traditional players such as Red Hat, but none of them are in a position to mess it up.
What I'm waiting for, though, is for a big player to do a modern, clean "Kubernetes first" cloud offering. We're currently on Google Kubernetes Engine, and I'm disappointed in the lack of integration throughout Cloud Platform. GCP is great in many areas, but the Kubernetes offering is clearly an afterthought. As an example, if you create an ingress, this creates a bunch of load balancer objects (forwarding rules, URL maps, backends). But these are all given obscure names (k8s-um-production-zippediboop-f8da5e6c92f38300), and none of the Kubernetes labels transfer over. Same with disks; GCE disks don't even have labels, so they put a bunch of JSON in the "description", which of course cannot be filtered or queried. Similar things happen with VMs and other things; the "mapping" of Kubernetes to GCP concepts is basically either lossy or non-existent. Many other similar weaknesses exist: Cloud SQL (if you want to use Cloud SQL, you have to manually run the Cloud SQL Proxy as a sidecar), GCP service accounts (different from Kubernetes service accounts!), etc. GKE is solid, but everything seems like it's been put together without any over-arching design; GKE is just another thing buried in a somewhat ramshackle jungle of different services.
There's an opportunity for a vendor to come in and offer a platform where you start with Kubernetes. In particular, this means that the platform should be manageable from Kubernetes, through the use of CRDs and pluggable controllers. For example, I should be able to define a persistent disk using a Kubernetes manifest, instead of having to go through some web UI or CLI ("gcloud compute disk create" or whatever).
That said, it's hard to compete with GCP at this point. Whoever wants to compete in this space have to compete not just with GKE, but with the technical awesomeness of GCP.
Kubernetes came out of the engineering team at Google. It was created by engineers for engineers. The fact that it's being exploited by enterprise consulting types doesn't change that premise.
I like to believe that nobody creates anything for sole intentions of extracting consulting bucks (there probably are cases of this happening in tech, but I like to be benevolent). So I don't believe Kubernetes was created for that and I don't think that OpenStack was either. But regardless of what the creators want, opportunists see new platforms and want to attach themselves to them.
Kubernetes isn't simple. Some of that is due to the problem its trying to solve, some of it is because the design, some if it is because the ecosystem that has evolved around it with regard to meshes/load balancers/control planes and overlay networks. We are after all replying on an article about a startup ecosystem being created around Kubernetes.
Tools like Kubernetes gaining adoption are partly based on technical merit/need and part based on fashion. Developers like toys and for some people the toys are what gets them through the day. Most of the people adopting Kubernetes probably don't need it. Developer Advocates and consulting shops are the ones making people think they need Kubernetes when they don't. This is an industry problem and that's all well and good. Let's just not pretend there isn't some influence from startups (typically backed by VC, which exacerbates the pressure) and consulting shops.
> I like to believe that nobody creates anything for sole intentions of extracting consulting bucks
Sharepoint and BizTalk spring to mind... MS Dynamics (or whatever), too.
SAP and other "configuration heavy" systems bring it to another level.
> Most of the people adopting Kubernetes probably don't need it
If they geniunely don't need it then it will bite the in the ass.
On the other side: the absolute nightmare hodge-podge of half-baked non-integrated solutions that Kubernetes has murdered, and the promises it delivers on for cloud-native and cloud-first systems, are major strategic wins with clear ROI...
MS does not profile Google tech without cause. The largest virtualization supplier does not integrate competing Google tech without cause. Amazon is an easier mark here, but: the very fact there is a common, turn-key, orchestration platform across all major cloud providers is massive, and very hard to attribute to developer advocacy.
It's a quick read, but to summarise: it's almost as complex as google's internal borg system but the benefit isn't even close (partially because nobody else has google problems). I can't down K8s personally, as I've never used it myself. But I wonder if there's the possibility of a system that's more 80/20 of google's borg rather than the seeming 20/80 coming from k8s. And I wonder if Grab will release it next year.
I don't think that's true. If you believe the team that designed Kubernetes, Kubernetes is an attempt to improve on Borg.
Borg is an accumulation of a decade's work with containers at Google, and has been described by googlers as a rich but a little messy, having been designed incrementally over many years as needs have surfaced. Borg could never be open-sourced because it's so specific to Google; for example, it uses Google's own cgroups-based container tech, not Docker/OCI/etc. Omega, as I understand, was an effort to clean up Borg and modernizing it, but apparently it was never put into production; instead, some of the innovations ended up being backported to Borg [1].
More importantly, Kubernetes is based roughly on the same design as Borg: A declarative, consistent object store, with controllers, schedulers and other bits and pieces orchestrating changes to the store, mediated by a node-local controller (Borglet/Kubelet). A major difference between Borg and Kubernetes is that with Borg, the object store is exposed to clients, whereas Kubernetes hides it behind an API. Another difference is the structure of containers; Borg's "allocs" are coarser-grained than pods and Borg is less strict about where things go, which googlers have described as a shortcoming compared to Kubernetes' strict pod/container structure. Another difference, also seen as a shortcoming, is that Borg lacks Kubernetes' one-IP-per-pod system; all apps on Borg apparently share the host's network interface. Kubernetes also innovates on Borg in several ways; for example, Borg doesn't have labels [2].
Borg, from what I gather, scales much further than Kubernetes at this point, but it's really not related to the design. The design is fundamentally the same.
Yegge's criticisms are too handwavy ("overcomplicated") to counter, but I don't think Yegge knows what he's talking about here. As for "benefit": Not sure what you mean by this, but Kubernetes arguably comes with benefits — declarative ops, platform abstractions, container isolation — even if you're just running a single node. The notion that you only need Kubernetes if you have "Google-scale problems" is just nonsense.
All of the things you described as improvements are more complexity and layers of indirection. Kubernetes may be an attempt to improve on Borg but adding on a bunch of features and plugin architectures to solve more use cases isn't necessarily an improvement.
Kubernetes is actually the third system. In-between, there's Omega. Yegge must have never set up a GSLB or GFE service to appreciate the extra stuff that Kubernetes features.
Thanks for the detailed insight. I've been following k8s for a while now but never had a need, and Yegge's post put an end to it in my mind.
Grab is the company Yegge left google for. He always complained about google's inability to platformize, so random hunch is he instills this desire into Grab? But entirely random. I also don't know how influential he was inside google vs outside.
Kubernetes does suffer from complexity right now, but that also gives it it's flexibility.
I think turnkey kubernetes solutions like Rancher will dominate for a lot of use cases, especially for individual devs and small teams that can't have a dedicated DevOps resource to manage kubernetes.
As someone who has been running kubernetes clusters for the last 1.5-2 years. It's quickly becoming boring (in a good way). I'm at a fairly large enterprise and we're running five clusters (two gke managed, which .. is limiting, three with our own bootstrap). I'd say we spend about 10%-20% of our time actively managing clusters (upgrades, troubleshooting.. some of that time is about maintaining some value adds like automatic log shipping.. that kind of thing), most of our work is writing value-adds, helping customers onboard and consulting with teams on how to best deploy and work with kubernetes (many teams at my company aren't really familiar with containerization).
Granted we run our own bootstrap (when we started none of the bootstraps running around we're ready, we started with a terraform/make implementation of hightower's hardway and we've just kept adding. We're thinking about revisiting the space again.
I highly doubt it. Even most of Google doesn't have "google problems". (Having to design things for "google scale" when you have no reason to was a popular gripe while I was there.)
You don't go Google scale because it's cool to. You either do it because you absolutely have to, or you don't because, thankfully, you don't have to.
I'm currently supporting ops of some "eventually consistent, globally distributed" (if you can fathom through the previous engineer's algorithms) built-from-scratch system designed to be resilient in the face of multi-megaX-transactions-per-second, that currently contains all of 8K records.
Whereas stackoverflow.com runs on SQL. Their own hardware at that.
Companies don't go google scale, companies are google scale. It's not a choice, it's a fact they have to deal with.
Pick any large tech company, they have thousands of servers and countless customized software running. That's what is meant by google scale.
They need to manage that and they suck at it most of the time, they don't know what resources they own and they can't figure out what's running or where.
>Kubernetes came out of the engineering team at Google.
Attaching the Google name to projects stopped having meaning ~5 years ago. Their hiring process optimizes for fresh graduates who know nothing about engineering. Something coming out of Google means nothing, especially when Google itself doesn't even use it!
>It was created by engineers for engineers.
So was openstack. Then the feature creep happened.
Sure, there are plenty of simpler alternatives... as long as you're willing to consider ones that don't do quite as much. (Beware, you'll also need more of them.)
The complaints that Kubernetes is "complex" are obviously comparing it to more targeted solutions and ignoring the aggregate system that arises in its place.
If you find a tool with a better fit to the problem, then yay for you do the smart and simple thing. In practice, though, for many shops there's a major hodge-podge of almost-there solutions which require much more specialization on much less popular frameworks with much less vendor support to compete with Kubernetes.
Kubernetes did not come into a vacuum: it's disrupted some mature solution ecosystems by being much, much, better.
Kube is too complex. To derive any kind of business value from it you need highly skilled people. The winner of the container orchestration battle will be the one that simplifies, not complicates.
To me this is just another hype train, like TensorFlow/Caffe before it, and like Hadoop/Spark before that.
These tools all have their uses in specific, non-common cases, but for your average business they are a net loss in time and money.
Too complex compared to what? All other commonly used web infrastructure is complex in various ways.
In my experience building and running Kubernetes clusters for Fortune 500 companies is that actually it's much simpler, it's just different from traditional infrastructure.
I guess we worked at different Fortune 500 companies. What I see are companies adding complexity to their web infrastructure where none needs to exist.
A vanishingly small number of use cases require 100's of nodes, and the accompanying orchestration. Stack Overflow runs a top 25 website on 9 webservers, and most of those are just redundancy. Nothing Ansible/Bash or Powershell/MSDeploy can't solve in orders of magnitude less time. And debugging/diagnosing problems is much easier.
Were the Fortune 500 companies hosting k8s on-prem or on a cloud provider? From an operational standpoint, I can see where GP is coming from, especially if you are hosting on-prem, for non-greenfield projects.
Trying to integrate a k8s cluster into legacy infrastructure is non-trivial, and requires knowledgeable operations teams to deploy. Troubleshooting k8s requires extensive domain knowledge, in addition to foundational understanding of fairly advanced networking and distributed systems concepts. In my experience, working with k8s also requires much more vigilant watching of the project space for (breaking) changes, updated docs, SIG news, and github issues. Even if Google or Amazon is managing the k8s management, troubleshooting your app using kubectl, etc. is a new skill to learn.
I actually like a lot about using k8s, but I understand when people complain about the complexity.
Singley rooted Debs or RPMS with a nice single symlink init system and a log shipping system that is unified across the app stack will get you 90% of the way there. Add in a build system that ships "the world" inside your deb and you are 99% of the way there. Add in a cluster wide config system and you are 100% of the way there.
K8S forces people to use an algebra, the expression of that algebra can take many forms. Containers are _just_ trees of processes with ... contained separation boundaries.
I wonder if the "It's simple, just use kops!" etc, have tried to deploy kubernetes to anything other than a cloud service provider. Setting up a production grade Kubernetes cluster integrated on-prem with in-house development and operations teams is no small feat for anything other than a toy deployment. Kubernetes has benefits for some use cases, but I think some of the HN crowd underestimate the rather large investment of time and energy it can take to get there.
k8s is not complex. You basically can spin up a cluster in no time on CoreOS via pxe/ipxe. You do not even need all these fancy stuff around k8s. You only need to ssh into your machine to start bootkube (on a single machine) or later to update the kubelet (can be done with ansible). Everything else can be managed via kubectl.
This thread is extremely refreshing. The number of times I've heard "Kubernetes is sooo simple!!!" here and elsewhere is super frustrating.
"You just take your app and put it in a container, then put that container in a pod, which is in a replica set with some other apps in their own containers, which is on a node in a cluster in a datacenter somewhere! Oh and the replica set is amongst a deployment. And if you want to talk to the outside world you'll need a sidecar with ngnix which is an app in a container in your pod. Or your deployment. But it's different than your app. Time to get kubectl and minikube installed..." People act like all these concepts are taught in beginner CS classes.
Yeah, and trying to explain all these concepts to devs, qa folks, data scientists, etc. when you tell them they have to start working in a Kubernetes environment, then actually working with them to make changes to their apps to make that happen is a long journey - unless you happen to have a first class team on hand that really knows the ins and outs of how the entire ecosystem works.
Kubernetes has some great attributes, but being simple isn't one. I feel sometimes that Google really pulled one on us, and this is just training people to treat the entire system as a black box, locking them in to services only top tier cloud companies have the expertise to provide and maintain reliably.
Only a few people need to know all the details. We have many developers that know absolutely nothing how this stuff works. They only know that they push code, and a container falls off the other side, which can even be installed automatically.
Also, you don't need sidecars. Add a k8s ingress, or even deploy a separate nginx that will do this for you (we use the second option). Developers do not care.
I'm starting to think that minikube is an anti-pattern.
Companies often think that with k8s they can put the onus of devops on the app developers themselves. My company quickly found that that it's not nearly that simple and you still need a team of engineers dedicated to helping people get set up, debug issues that arise, and build tools for both.
If you are deploying anything that's not a static website, you are going to have to learn a lot more concepts than these.
You do not have to use CoreOS, you do not need PXE boot. That's one of the possible ways.
What you do need is a handful of containers, and an overlay network. Which, if you do not want to manage, you can sign up for something like Google Cloud and let them manage it for you.
Debian, PXE/CDs/USB sticks/whatever install medium you wanna use, ansible, systemd, ssh. And you say it's not complex? I don't think you and I will ever agree on the meaning of that word.
Currently running a application on anything is always complex in any kind of sense.
>> Whoever wants to compete in this space have to compete not just with GKE, but with the technical awesomeness of GCP.
After years of using all the major clouds - Google Cloud has the best performance and primitives, but the worst support and operational/business features. If you want the fastest VMs/storage/networking, and are fine with non-standard APIs and beta SDKs, then it's a great platform and has super simple billing.
AWS and Azure are much better suited for most companies and startups though in terms of actually getting your business running with standard tooling, large ecosystems, and a massive portfolio of services.
Funny, I had the exact opposite experience when it comes to APIs. Doesn't AWS have a PM for every product with different API each time? Tried to build something like a terraform using boto only to give up after dealing with inconsistent APIs for about 2 weeks.
It's probably AWS Fargate, part of the Elastic Container Service. But even then, it's still now a vendor lockin of the past, if you got your services deployable via containers you're 90% there to running your own Kubes cluster to launch them.
> For example, I should be able to define a persistent disk using a Kubernetes manifest, instead of having to go through some web UI or CLI ("gcloud compute disk create" or whatever).
What do you mean? You can do exactly that today, with persistent volume claims. K8s will create the storage volume for you and make it available for containers. This is "dynamic provisioning".
What is missing is being able to create other resources. For instance, define a EC2 instance as a k8s object and have K8s provision it, for legacy or anything you haven't moved to K8s yet(databases, etc). I see no technical reason why it cannot be implemented today.
Pivotal and VMware are working on this. From what I’ve seen (yet to play around too deep with this yet) looks promising. You’ve got it spot on though, just needs to be simple.
https://pivotal.io/containers/deploying
I work for Pivotal. PKS is our product in the Kubernetes space, co-developed with VMWare and Google. The opensource project we based PKS on is Cloud Foundry Container Runtime, CFCR.
We are basically bringing all the stuff we learned building Cloud Foundry to bear on Kubernetes. There's also an increasing amount of systems being shared. Right now Istio is seen as the long-term north-south and east-west routing solution for Cloud Foundry App Runtime (CFAR, what folks traditionally thought of as Cloud Foundry). We're looking at introducing Loggregator to the Kubernetes world as Oratos, we have factored buildpacks into standalone systems that can have a single image builder for any OCI-compliant platform, we led the formation of the Open Service Broker API effort that creates a common service catalogue approach for both CFAR and Kubernetes, we've contributed to runC and I lose track of all the rest.
When I say "we" I should say this isn't just Pivotal. IBM, SAP and SUSE are deeply involved as Cloud Foundry community members.
Check out also VMware's container service extension (CSE) on Github. CSE makes it easy to pop up clusters on vCloud Director, which is the VMware cloud stack used by a lot of service providers.
Disclaimer: I work on CSE and use it for K8s dev/test. It's really handy.
We were on DigitalOcean before GCP. They're okay. They're cheap, but can't really compare to GCP in terms of performance, stability, networking, object storage, etc. Their Kubernetes offering is also a GKE competitor, not a "Kubernetes first" cloud.
At least DO supports IPv6, for our application the 40ms latency penalty we encountered for half our customers when on GCP from not supporting IPv6 was not worth it. On that note, HE.net really peers with everyone over IPv6, even ISPs that won't peer with them on IPv4, its quite impressive how much better the routing can be on certain ISPs due to this!
Users on Centurylink Fiber can reach us in 4ms over IPv6 (despite it not being true IPv6) versus 45ms to 60ms with IPv4. DSL is similiar but not quite as notable, more like 30ms of saved latency.
The root issue is Centurylink has no peering with Hurricane Electric on IPv4, and ditto for peering with numerous other providers, versus on IPv6 they have a much more open peering policy. This seems to be common among a number of ISPs, though as time goes on this issue will hopefully ease.
More like the end user gets routed through bespoke connections hundreds of miles away due to ISPs like Centurylink not openly peering on IPv4 at Internet Exchanges, whereas on IPv6 they seem to have local routes to Hurricane Electric and many others on the West Coast, thus no need for traffic to go back east!
Last I checked, DO still fails to allocate as little as a /64 to droplets. They provide a shared /64 out of which you are allowed to use 16 individual IP addresses.
In IPv6 land, that can introduce a lot of pointless complexity.
It would be reasonable for customers to be able to expect the ability to get /48s or /56s. Instead they don't even get a proper /64.
Are you proposing that cloud vendors should do a better job of integrating Kubernetes with underlying infrastructure or are you saying there should be a Kubernetes-only cloud platform? If the former that's clearly desirable; you should be able for instance to set up networking between services without having to worry (much) about how it is implemented.
If the latter, I am not so sure (or else I don't understand your point.) A lot of things don't/won't run well on Kubernetes--database management systems are a good example as well as any legacy application, which includes a lot of Windows OS, so you still have to address those.
More subtly Kubernetes is not going to rewrite the whole world or implement distributed firewalling, network attached storage, VLANs, etc. So you are always going to have a non-K8s layer under there that is more or less foreign to the Kubernetes model. The best you can do is make the layering relatively efficient.
I did describe both (Google could do a much better job with the integration on GCP/GKE), but by "Kubernetes first" I meant the latter.
Databases run fine on Kubernetes and have been doing fine since 1.7, and the meme that Docker is bad for stateful apps is a getting a bit old. The challenges are mostly the same as with running databases on a VM or bare metal. In particular, you need to know how to manage HA.
The weakest point is perhaps that Kubernetes's scheduler completely ignores disk I/O, so you have to bare careful to avoid sharing a node with multiple disk-heavy apps that will compete for resources. This is includes the file system cache; for example, PostgreSQL works best when it can rely on the OS to cache pages, so you don't want other apps (including Docker and Kubernetes themselves) to compete there.
That said, I wasn't saying that a hosted solution shouldn't also offer VMs. Just that Kubernetes should be the main entrypoint and control plane. And someone figured out a way to run VMs via Kubernetes [1], which is a neat example of a solution to the lack of integration I was complaining about earlier.
> A lot of things don't/won't run well on Kubernetes--database management systems are a good example as well as any legacy application, which includes a lot of Windows OS, so you still have to address those.
Not saying I've tried it, but some people are having ok luck pinning RDBMS pods in Kubernetes to specific nodes. It goes a bit against k8s principles, but it makes operational sense.
Kubernetes is baking in windows container support, for some legacy scenarios. More interestingly (and posted on HNs front page today), are solutions that allow independent VMs to be run as though they were k8s pods. This provides hybrid models ideal for legacy packaging and maintenance while moving onto new hardware.
> Kubernetes is not going to rewrite the whole world or implement distributed firewalling, network attached storage, VLANs, etc. So you are always going to have a non-K8s layer under there that is more or less foreign to the Kubernetes model
There will always be some natural impedance between hardware and software...
I think this picture is ripe for improvements though, and we're already seeing the edges of it take shape :)
Kubernetes will be moving towards smarter networking solutions to handle more and better use-cases with better performance (BPF), and is incorporating better network abstractions slowly but surely.
From the under layers: software defined networking (SDN), and kubernetes providers for major virtualization platforms (VMware, for example), have commercial offerings that could readily support integrated or "hyperconverged" operations... Microsegmentation and microservices go hand-in-hand. Empowered by a platform connected with role based access controls top-to-bottom there's a lot of potential to harmonize those distributed firewalling/VLAN needs through the same declarative YAML.
We're not there, yet. We are closer than ever though :)
I did infrastructure work on OpenShift for the majority of last year, and I can tell you that my team's experience was absolutely horrendous. OpenShift, to me, suffers from "too early syndrome", and since it was building features on top of k8s before the underlying platform was mature, has ended up in a very weird state in relation to k8s.
Frankly, it was a supremely painful platform to work on. They obfuscated just enough of the k8s API to make it simultaneously completely unintuitive for less "orchestration minded" team members, yet severely underpowered for me and my fellow platform workers. It struck an unhappy medium for our team that no-one wanted or needed.
All my looks at OpenShift suggest that not enough has been peeled back to make it a useful platform-on-a-platform, but since they probably paved the way for certain features, those features have predominantly been implemented (and more tightly integrated) into k8s. RedHat is going to need to come up with a new value proposition for OpenShift for it to ever be a truly viable alternative to "raw" Kuberenetes, and given their buy-in to the technology, I'm not convinced they'll want to throw away the work they've done so far. Good money after bad, or something.
My team's experience is the exact opposite - we're very happy with it, especially because it _doesn't_ obscure the k8s API and is quickly rebased on top of upstream.
It's been great for us since we were productive from day one without having to worry about setting up Kubernetes or figuring out things like deployments and builds. Their documentation is great and the Ansible installer is very helpful. They have a lot of operational docs, which are invaluable (day two ops guide, among others).
Apart from the obvious PaaS features, it has a lot of small and seemingly minor improvements like "oc rsync --watch" (copy changed files to a container) or "oc rsh dc/foobar", proper auditing, and a lot of other small stuff that makes life easier.
Red Hat is one the main k8s contributors and is responsible for the RBAC implementation, among others.
You did not mention any particular pain points, but I'd love to hear about them.
Maybe. I'm skeptical of Red Hat. The openshift.com web site, which is definitely not geared toward engineers, doesn't give me any confidence that this is what I'm after.
Not that Google is better here, but I already know what GCP provides; something that wants to compete with GCP/GKE really needs to explain why they're a viable competitor.
(And to people who design these things: If you need to have a "Products" dropdown filled with unexplained product names, you're doing something wrong.)
I work for Pivotal, we directly compete with Red Hat.
Look: they're Red Hat, I mean c'mon now. If you don't think they have great engineers then I'd hate to hear what you think of the rest of us.
These websites are not usually intended for you or I. They're intended to get the interest of the cheque-signers. Enterprise software is a very particular world and it has its particular cadences and norms and approaches. Red Hat understands it well. So does Pivotal. That's why we see so much of each other at a number of places where folks are looking to adopt modern platforms.
We were talking about openshift.com, though. I didn't realize there were more than one site. Since you're going there:
- I didn't even know it was called "OpenShift Origin"!
- I don't understand what the differentiation is between the various things called "OpenShift", from having visited the Google links briefly.
- Third Google hit (https://www.redhat.com/en/technologies/cloud-computing/opens...) does not even mention Origin anywhere, but describes it as "a container application platform that brings Docker and Kubernetes to the enterprise". With that copy, you already lost me. That IBM-type business site that makes my skin crawl.
- The Github site actually introduces it as "Enterprise Kubernetes for Developers", which... I don't even know what that means! What does "Enterprise Kubernetes" even mean? The actual readme's introduction ("a distribution of Kubernetes optimized for continuous application development and multi-tenant deployment") does make sense, and the readme itself is fine. However, still no explanation of what "Origin" is compared to other things named "OpenShift".
- Openshift.org (5th link on Google) is good. That's exactly the kind of site I'd want to see as a developer/CTO. However, still no explanation of "Origin". From Openshift.org it seems like Origin is the only product.
- I go back to openshift.com and click on "Products" out of curiosity, and it's clear that there's a bunch of other OpenShifts. What are they? Why should I care? The site isn't inviting me to find out, so I leave.
Why are there are so many sites? Github is one thing, but so far we have three, and none of them really give a cohesive story.
I understand that the marketing department is targeting different people, but do they realize that people google this stuff, and will likely end up on the wrong site, with no signposts to the right thing? The fuzzy messaging leaves the impression that the company is messy and disorganized, and also pretty lost in a fog of enterprise buzzwords.
Anyway, in my original comment, what I was seeking was Kubernetes a cloud service. After sniffing around on openshift.com, I suspect the product I want is called OpenShift Online, but I'm not actually sure. Because there's also something called OpenShift Container Platform, which is also Kubernetes, but apparently in the cloud of your choice? Unfortunately, the site doesn't explain what the relationship is between this and Origin, and of course there is no pricing for Container Platform, probably because it has the label "ENTERPRISE" next to it. Figures.
There is, though; "Origin is the upstream community that powers OpenShift". Meaning Origin is the open source project upon which commercial products are built. Much like Moby is the open source project upon which Docker is built, for example.
I fixated on something else, which was my impression that you were downplaying their engineers and the ability of their engineers to communicate with fellow engineers.
We have a similar mix of CxO-focused and engineer-focused literature. So I guess I felt you were giving us a hard time too.
Edit: and I guess it figures that, since I work for a competing company, I'm primed with some background details on OpenShift.
Red Hat's marketing is definitely a mess (I say this as an enterprise Red Hat customer in the midst of an OpenShift roll-out), but their engineering is top-notch. Many pieces of k8s originated in OpenShift, and Red Hat is heavily involved in k8s development.
A lot of people come to websites looking for specific products. When you don’t offer an obvious path to what they’re looking for, they get frustrated. I know I do.
Actually it is already possible to create persistent disks through Kubernetes by creating a persistent volume claim (pvc) which triggers an actual persistent volume to be created. Cloud provider controllers take care of this, just like they create load balancers for services.
That's not what I meant, though. A Kubernetes PV is an abstraction that causes a controller allocate a GCE disk (which of course is another abstraction), but if you create a GCE disk through the CLI/UI/API, then you don't get a PV.
I think the point is that I don't have to touch the cloud provider's control plane in 99% of cases. Which brings the portability that first attracted me to Kubernetes.
The portability is not as attractive as a fully integrated solution though, for types of people that are not already attracted to Kubernetes. Where I work, we use AWS and say that "we love vendor lock-in." The more integration with the rest of the vendor's cloud platform, the better.
I think it's a great example to say it is idiosyncratic that I can create a PD by creating a PV, but I can't reciprocally enumerate PDs by listing PVs. Another: Kubernetes secrets are hidden inside K8S and they are not generally accessible through any vendor-provided secrets managers outside of K8S.
I love Ingress, because it allows me to create new hostname-driven application endpoints inside of my cluster without any permissions on the AWS side. The cluster has a load balancer with a star certificate attached to it, the DNS has a star record similarly, and I can create new Ingress records without reaching outside of the cluster.
My InfoSec department hates stuff like that! Now if I use Kubernetes that means they have to learn Kubernetes too, and maybe even build out additional tooling around it, because their AWS-based automated service inventories are not reliable anymore.
I don't really have those star records (I'd really like to have them, it would be really convenient for me as a developer, so I don't have to ask for permission and go through layers of approval when I need to do stupid things, like my job...)
The unfortunate reality that I've encountered is that those teams are more likely to see the encapsulation features like this Ingress configuration as a pure security risk with no benefit to them, plain and simple.
K8s needs a role with the correct IAM permissions to create the resources on the AWS side, it can't do it magically. Which means that someone granted these permissions before.
If anything, your InfoSec department should love that. IAM is pretty fine-grained, so they can control what gets created. Types of resources, and whatnot. Just don't grant star permissions to all of EC2.
(Spoiler alert: you got me, InfoSec typically does love this stuff. If it was just up to InfoSec, these types of things would be approved because they are so well organized and easy to understand. The alternative being, to deal with another vendor that has another completely different security model, which InfoSec actually has to do all the time.)
Fact is InfoSec is not at liberty to set priorities for new technology development. I have to either drag the whole group with me at an institutional pace, or get someone at a higher pay grade "who will stake their career on it," in order to help me meet my goals, which really may or may not be achievable simply by means of adopting this advancement. On the other hand, lots of people who are not InfoSec like to "help" with the ostensible jobs of InfoSec, ...
which is often seen as simply delaying progress until the problem can be fully understood and the solution agreed to by a quorum? Man I can't wait that long
You can add a PV manually to tie to some resource that was already created, but you don't get the bi-directional link. I'm not sure I'd even want K8s adding PV's for all volumes (maybe filtering by labels/tags would be ok).
This is approaching Terraform levels of complexity though. Unless you mean that cloud providers should do that when you create a volume, in which case I understand where you are coming from and well, they could do that. Although I'd much have them exposing a native K8s interface to take care of this PV and PVC for me and not showing me the underlying volumes at all.
Why would you expect that? Or use it that way? The clouds are IaaS, and Kubernetes is like a virtual human that creates resources using the same APIs you would. It's on K8S to keep track of what it's creating.
Why not use Kubernetes to declaratively define nodes, disks, load balancers, firewall rules, DNS rules...? If you look at Terraform, for example, it's just an "object store" (the state file) that you modify in order to "converge" your infrastructure.
If you're already using Kubernetes via GKE or a similar IaaS, then of course Kubernetes has underlying primitives (GCE disks, GLBCs, etc.) that are exposed. But those primitives are exposed because GKE runs on top of GCP.
What I'm proposing is a "KaaS" where Kubernetes is the control plane, without the baggage of an additional API.
That's because Kubernetes isn't an infrastructure system, it's an application system. More PaaS than IaaS, so you're supposed to define what your app needs and let it do the work.
You don't specify a node but the CPU/RAM your application requires and let K8S scale up the cluster if necessary. Likewise, you specifiy the storage needed instead of managing disks, and set network policies to control load balancing and routing. If you want to live in Kubernetes only today, you can absolutely do that. Launch your cluster once and then use kubectl and yaml for the rest.
The good news is Docker EE 2.0 brings Kubernetes to you. However, I'm not sure if it's operating split-brain or if they made it such that you can pick your control surface.
Operationalizing K8s can be fraught with vendor specific parts. It doesn't help documentation wise when many things gloss over a concept that is offloaded into GCE or GKE.
Many of the k8s founders have left google. Google doesn't use it internally. I'd actually start looking away from gcp at this point for k8s, given google's history of dropping stuff that they're bored with.
Many of the founders (e.g. Brendan Burns) are still working on Kubernetes, so I don't see how that's relevant. Google is heavily invested in Kubernetes, and so is a bunch of other companies.
According to googlers here on HN, Google does use Kubernetes internally via GCP/GKE. But clearly those apps are in the minority, given Google's huge investment in Borg.
Coming from KubeCon Europe, I can confirm that Google is anything but bored from k8s.
Google's spring-cleaning has nothing to do with their open source projects. Kubernetes is one of their most widely used open source "products" and the industry is creating an incredible community around it with several big players that could easily jump in to take over project leadership and financing.
It is actually a good thing that some k8s founding members left Google because they already got what they wanted to achieve and went on to work on stuff that Google probably wouldn't have prioritized as much, see Ark and kssonnet.
Kubernetes is IMHO the platform for cloud native applications for the next 5+ years.
Not wanting to use it because Google dropped everyone's darling (Reader) is rather naive.
I mean, they just announced a new container runtime called gVisor to further increase security for containerized apps. It's pretty clear they want to be able to offer fully managed multi-tenancy Kubernetes clusters that execute untrusted code.
Google does not just drop enterprise products and services. They definitely have a problem with consumer (messaging) apps, but that's completely different.
I didn't realize all the pieces of the cloud stack that CNCF was working on and how each piece gives you more flexibility to move between cloud providers. Kubernetes and rkt replace servers and orchestration of course but you also have Open Policy Agent and Spire to replace IAM, Envoy and Jaeger replace X-ray, VPC, Security Groups, Rook replaces S3, etc.
It's kind of interesting because Google and Microsoft have opened a new front against Amazon to counter their strategy of locking you in with services. If software from CNCF is good enough to be used in place of these services on each cloud provider then you can nullify AWS's advantage there and Google can lure you in with ML and Microsoft with their enterprise experience.
But I'm not sure these projects can make progress faster than AWS can release and update their fantastic array of services. Kubernetes being the obvious exception (when is EKS GA?!?!)
Kubernetes is cool and all, but there needs to be a lot of simplification for it to be "nice to use". Essentially the problem is that it's the "opposite" of Golang - the number of ways to do the same thing is massive leading to huge numbers of headaches in trying to get things done.
Recently, I spun up a simple pod-to-pod communication example but I found it pretty difficult. If you look up cluster networking in Kubernetes (https://kubernetes.io/docs/concepts/cluster-administration/n...) you'll find a whole fire hose of different options from ingress to calico to fabric and on and on.
This was what it took for me to try and rubber ducky my way to getting networking to work on Kubernetes, and in the end I had to get help from a friend at work (https://stackoverflow.com/questions/50195896/how-do-i-get-on...). It may be better than what came before, but it's not great.
I don't see your point, but I should have clarified more:
Kops == AWS install
Rancher == Easy small cluster on-prem or private cloud. GUI push-button setup
kubeadm == CLI tool to set up a cluster manually with basic settings. Not very flexible, but can get a cluster working in minutes.
kubespray == Ansible playbooks for setting up k8s cluster. The most powerful of the bunch, especially if you're familiar with ansible. This is the preferred way to run it in production I believe, along with customizations on top.
Recently I have been playing with kubernetes through Rancher. I like how easy it is to install kubernetes through rancher, essentially I ran 2 commands: 1. on the machine that runs rancher and 2. on the first node in my cluster.
After a few days of playing I set up lets encrypt with load balancing, a running app (Rails). A remaining issue are persistent volumes and how truly persistent these are. I haven't found out yet which solution I should pick for this. Longhorn is a Rancher product, which is probably what I will read more about now, but I cannot be sure. There are so many concepts and terminology that you need to figure out. Having Rancher in between is not helping me get a hang of kubernetes itself more purely of course.
A while back I was playing with Docker Swarm and I must say that I like Docker Swarm better in the sense that it feels closer to the source and because it is built into Docker. I get a feeling however that Kubernetes is where the future is so learning more about Docker Swarm is probably less worthwhile.
There needs to be a digitalocean or heroku for containerization. Currently to start a k8s whatever you need to go through various manual rigamaroles and spend a lot of money, or go through the big lockin providers and spend a lot of money.
For those of us not backed by venture capital and not charging SV ex-googler rates to our clients, we need something to say "we'll host you on this git push containerization thing, it'll be cheap and easy, and we'll expand when the time comes, and that'll be cheap and easy too".
We are working on https://www.KubeBox.com to provide simple Kubernetes clusters for an affordable price.
For $36/month (8GB Ram, 2vCPU) you get a single node if you just want to have the Kubernetes APIs. Maybe introducing smaller node plans later in the year.
Unfortunately, not much info on our website but please sign up if you are interested.
https://hyper.sh aimed to be a CaaS (Container as a service) and Joyent’s Triton as well. It’s an interesting premise, but from the outside both don't have much traction.
I'm curious: how does containerization work for systems that utilize kernel-bypass? Is it even possible?
I did a (admittedly short) search for kubernetes and kernel bypass, and the only thing that seem remotely relevant was [0], however it didn't indicate whether they work together.
For background, I work for a Dark Pool Alternative Trading System, and we currently utilize kernel bypass for all of our networking using Solarflare NICs & openonload [1].
In the same vein, curious how containers work with CPU shielding and pinning threads to specific cores. Is it possible, and how do multiple containers on the same box interact in that regard. Do they need to be quasi-aware of each other so as to not pin a thread to the same core?
I'd greatly appreciate if anyone with experience with containers can answer these questions. I'm genuinely curious, but it's not worth researching further if there's no solution that can handle these strict requirements (e.g. it's a non-starter if containers increase latency).
At Datica (where I work), we started in 2014 with a bespoke container orchestration layer. This powered our HITRUST certified Platform. Think Heroku for regulated industries (like Healthcare). After years of hardship trying to keep up with the market we finally decided to go all in on K8s.
Kubernetes gives us the flexibility and community to focus on the compliance and security layer, while not having to build a Platform in house. Until now, no other open source solution had given us this type of flexibility. We're still working toward a GA release, but the speed at which we've been able to move has been incredible.
It’s always nice when there is some sort of consensus in the industry, however, there is one thing I worry about.
Now that there is a usable solution for complex orchestration, many newcomers will consider it the goto solution, regardless of scale. You need nginx and some python scripts? Kubernetes! You have 100 daily visitors? Kubernetes!
It’s not really their fault, it’s just a sad consequence of a convenient cloud solution.
Part of this is also people jumping to microservices when they really have no need for them. When you have less than 5 developers and very little traffic...it's just absurd. But people are doing it anyways. Because "monoliths" are "bad". And honestly, I wouldn't be surprised if a lot of these microservice architectures end up totally botched and creating a ton of headaches without delivering much benefit.
Is that really a problem though? The entire history of software development is just building on one layer of abstraction over another. Is it sad that people these days don't write their own web server for new projects?
A lot of people are only adopting Kubernetes because they think they will benefit from Dockerized microservices when they don't even have most of the problems that solves. There's a lot of extra development and operations effort going into things for no real reason other than mistaking this technology for a silver bullet.
The point is not to rewrite from scratch.. the point is that your judgement to use a piece of technology should come from your understanding of the problem and direction and not based on what is a legitimately a hot topic in a context that does not apply to you.. in other words do not let google search results be the bar you meet in solving your problems..
I found that only bitbucket deployments is a good one in terms of simplicity of managing deployments (just like Heroku). Most vendors forces me to use their crappy CI solution for doing CD. Why they want me to migrate to your (very limited) build system? I can pay you for a fancy dashboard, but please, allow me to use whatever i need to build my software. There are many options on the market for CI and you just can't build them for everybody. But good CD is a way to go and easier to manage. Just give me a hook for registering for a new build (say docker image versions) and help me manage this stuff.
I have tried Spinnakle, but it is too fragile for me - there are no simple way to install it (k8s even easier!) and UI is too bad for a small project.
We used Drone [1] from a while. It's almost exactly what I want in a CI/CD system, and it runs pretty well on Kubernetes.
On the minus side, the development has been slow, with major breaking changes happening along the way. And in the latest version, the open-source version has been a bit hobbled. In particular, it doesn't support global secrets, so every project to be built has to be created and managed separately. This is not just if it needs build secrets (such as to access private Go packages, NPM modules, Ruby gems, etc.), but it's obviously also required to authenticate with a container registry. We decided to skip that since it means duplicating the same secrets for every single application. We actually reached out to the Drone guys to ask about enterprise pricing, but they didn't respond (!).
(At the moment we're back to building with Semaphore, which is a hosted solution similar to Travis. It's an old-hat CI system that spins up VMs, so it's slow and awkward to work with when it comes to Docker. But it's working okay at the moment.)
I've heard good things about GoCD [2], though. It's next on my list to investigate.
I assume you mean the Spinnaker/K8s v1 provider abstracted away the K8s primatives. The new v2 provider allows you to deploy manifests and supports all K8s objects.
We tried spinnaker and are now using Weave (https://www.weave.works/). It works pretty well, actually. In particular we like that it stores all state in a git repo which is automatically updated.
We have our own monitoring stack, though, so don't use the additional prometheus integrations
I'm enthusiastic about Concourse (https://concourse-ci.org/). It's a pipeline-based CI system based on Yaml configuration files. It's highly flexible and has numerous plugins to interface with third party systems. There's a Helm chart to deploy Concourse to Kubernetes: https://github.com/linkyard/concourse-helm-resource.
I'm awaiting jenkins-x's maturation. I'm looking at some issues for fixing myself but there's still plenty to do. Support is there for GCS and AWS iirc, but it's still very much beta.
I think the real advantage of k8s is the ease to scale bare metal. This just isn't that important for the next 5 years or so. On the other hand, azure and gke make k8s easy enough if you are already using docker. Containers on the hand, the developer experience is still very mixed. OSX still doesn't have great container support and I honestly can only guess how well windows can handle my docker-compose workflow.
One great k8s tool I like is kompose. It allows our devs a very similar interface around secrets, networks, volumes, etc.
From snooping around at KubeCon NA last December, my impression is that Apple is doing something involving GPU computing on top of Kubernetes. My guess would be machine learning, but they were characteristically hush-hush about it.
What I'm waiting for, though, is for a big player to do a modern, clean "Kubernetes first" cloud offering. We're currently on Google Kubernetes Engine, and I'm disappointed in the lack of integration throughout Cloud Platform. GCP is great in many areas, but the Kubernetes offering is clearly an afterthought. As an example, if you create an ingress, this creates a bunch of load balancer objects (forwarding rules, URL maps, backends). But these are all given obscure names (k8s-um-production-zippediboop-f8da5e6c92f38300), and none of the Kubernetes labels transfer over. Same with disks; GCE disks don't even have labels, so they put a bunch of JSON in the "description", which of course cannot be filtered or queried. Similar things happen with VMs and other things; the "mapping" of Kubernetes to GCP concepts is basically either lossy or non-existent. Many other similar weaknesses exist: Cloud SQL (if you want to use Cloud SQL, you have to manually run the Cloud SQL Proxy as a sidecar), GCP service accounts (different from Kubernetes service accounts!), etc. GKE is solid, but everything seems like it's been put together without any over-arching design; GKE is just another thing buried in a somewhat ramshackle jungle of different services.
There's an opportunity for a vendor to come in and offer a platform where you start with Kubernetes. In particular, this means that the platform should be manageable from Kubernetes, through the use of CRDs and pluggable controllers. For example, I should be able to define a persistent disk using a Kubernetes manifest, instead of having to go through some web UI or CLI ("gcloud compute disk create" or whatever).
That said, it's hard to compete with GCP at this point. Whoever wants to compete in this space have to compete not just with GKE, but with the technical awesomeness of GCP.