That first needs to prove itself in practice, so far it seems more like a desperate try to tie it to Fedora and have it catch up with the popularity in cloud of the likes of Ubuntu and to eventually have people paying by locking into RHEL CoreOS.
Unlikely, I'd say. If you fail to upgrade when a new release is made, are they at fault?
You'd need to look towards providers that specifically take on more responsibility like https://compliantkubernetes.com/ (disclaimer: I have worked at Elastisys, the company behind Compliant Kubernetes).
No, although it might reduce your effort needed. AWS offers that responsibility shift under the Fargate ECS/EKS launch types, which might run this underneath.
It's buried a bit in the GitHub docs, but it looks like it runs two containerds. One is for Kubernetes. The other is for running administrative containers, including the API server for normal interaction with the host, a server that implements AWS's remote command protocol (enabled by default), and a container that runs sshd (disabled by default).
Packages are built with RPM but RPM isn't used at runtime. Instead, the system is image-based and reboots for updates.
Agree. It's cool so long as the number of names is small, and the names actually are a pun on the function and not just e.g. names of planets. If "updog" is what brings something "up" that's a good name.
WiX (windows installer creation) has a multi-phase command line interface where the compiler/linker/.. has different names indicating the order they are applied: candle, light, smoke... Also a working system I guess.
"bork" is dog-talk for "bark", and so something that randomly gets an updog going being named such makes sense too, it's just a slightly more obscure joke.
>It is safe to say that our industry has decided that containers are now the chosen way to package and scale applications.
Curious how the HN community feels about that statement. Not so much about the truth of the statement but about the fact that containers are becoming the de facto method of packaging applications.
As far as I'm concerned, this is an obvious truth. Linux containers are processes with better sandboxing -- who would not want this?
As kinks in the kernel support and tech get worked out, and OSs deepen support I can't imagine that it will ever make sense to say something like "I could have run the process with cgroup and namespace isolation but I chose not to, choosing to make a new user-level isolation or run everything as root instead".
Arguments against containers as the future based on the complexity may have weight but not for long.
The word "sandbox" is a bad choice -- "isolation and resource limiting" might have been a better term to use, but the idea that containerization does not sandbox at all is not a fair characterization.
It's not a good sandbox, but if we are pedantic about the definition of a sandbox, it fits, especially when we think of the benefits of namespacing (effectively removing access to resources like networks, filesystems, etc).
gVisor is a more focused on sandboxing processes specifically, so it's relevant but gVisor is not relevant to the wider discussion about a packaging format -- unless you're suggesting to run gvisor'd processes instead of containerized ones and containerization is still beneficial in that scenario.
It's about damn time? I, for one, do not like having to deal with shared library hell, conflicts, and compiling from source because some unpaid repo maintainer is responsible for integrating code into my system. Not to mention the security enhancements.
The only thing that would make them better is if we stopped over-complicating them and made them portable [0].
[0] As in, could be moved to different disks and run from there without a bunch of hoop-jumping.
Desktop and mobile is actually where you want containers most. Servers rarely run untrusted or semi-trusted code because everything comes from a trusted source, usually open source, or in house.
But users want to run lots of shady apps, either that they find on random websites or places like the Google Play store.
It's also the rare case where you can't accept a 5% performance hit because that's 5fps in a game or 5 seconds on a 100 second render time or 5ms instead of 95ms wait in an interactive app.
I find that the key to running desktop OS/apps is never use sensitive data and always be ready to wipe your machine and start over.
That sounds good but I’ll believe it when I see it. There aren’t many desktop container/sandbox implementations out there and most are “vm light” e.g the windows sandbox and sandboxie. I haven’t seen anything more lightweight that can run desktop apps (in Windows at least).
Not sure if that changed since its inception but originally windows store was exactly this: no SLI/Crossfire, v-sync always on, no overlays. Basically it was isolated from using the driver properly.
We rarely use containers for our deployments because you can get the same features that containerization provides by other means. The biggest issue with containers is the stability of Docker both the operational stability of containerd and the API stability with the tooling (like to rename command-line switches). As far as Kubernetes goes, my problem is visibility. I have very limited knowledge about what the containers are doing, the provided metrics that you can access are much less than I need to try to operate a k8s cluster. Once you need to look at the actual host-level metrics (CPU, IO, mem, ...) you need to have a map of what runs where. One of the reasons people are pushing for k8s is that you do not actually need to know what runs where.
As far as complexity goes, I would much rather have a nodes where a single application is running and using 100% of resources (instead of having containerd or k8s services running) and do simple autoscaling, having access to host-level metrics that I can map back to applications easier than use Docker, k8s & co. Maybe is it only me, but I care about efficiency. Why waste energy?
The counter-argument is that developer time is more valuable than setting up clusters or autoscaling groups. Well, this breaks down when you have SRE team(s) maintaining the k8s clusters (literally every company I worked for). If you already have SRE people either embedded into your dev teams or separately then you can just build out a CI/CD pipeline that produces that production setup based on blueprints. We usually use Terraform and Ansible with tempalte variables (stage = test|qa|prod, cluster size = x, version = y) that makes it easy for everybody to provision clusters on their own. Does this mean more work than k8s deployments? Yes. Does this mean we have less complexity we need to care about? Yes. In my experience containerization is a development tool to make it extremely easy to achieve fast development cycles but right now the accidental complexity to take that with you to production is not worth it. There are very nice projects like LXC/LXD that I would consider using for security separation and resource management but we usually have clusters where 100% of resources go to a single services. Example: Hadoop cluster, Elasticsearch cluster, Web application (mostly API) clusters. I need to care about the underlying hardware because of financial reasons (what is the cheapest node type I can use to run workload X). k8s would not help here.
To sum it up: I do not think that the industry has decided on this. I also think that we are in the era of wasteful computing which will be finished soon because of reliability and unnecessary CO2 production reasons. Running containers has to be much less fragile and efficient to be considered the way to scale applications. I personally think that Firecracker is a step in the right direction in this while Docker & k8s in the wrong direction.
> As far as complexity goes, I would much rather have a nodes where a single application is running and using 100% of resources (instead of having containerd or k8s services running) and do simple autoscaling, having access to host-level metrics that I can map back to applications easier than use Docker, k8s & co. Maybe is it only me, but I care about efficiency. Why waste energy?
Hallelujah! I'd imagine inefficient containered architectures do more for the cloud provider's bottom line rather than help the customer.
> This is a reflection of what we've learned building operating systems and services at Amazon.
I don't know how actually tied in it is, but it's not totally surprising to me that it's built for AWS infrastructure.
That being said:
> To start, we're focusing on use of Bottlerocket as a host OS in AWS EKS Kubernetes clusters. We’re excited to get early feedback and to continue working on more use cases!
>
> Bottlerocket is architected such that different cloud environments and container orchestrators can be supported in the future.
While our first variant is focused on Kubernetes and EKS, we have designed Bottlerocket in a way that new variants can be built that work with other orchestrators, or even without one (we have ECS support on our roadmap already). Also, we really enjoyed working in Rust for big chunks of this!
Correct, that toml file allows you to build everything from that level, and open editors on that folder with the language server functioning. RLS requires a cargo.toml file in the root of the workspace even if it just points at other directories
This is somewhat false advertising. This is not an operating system in the sense of being a new kernel. It looks like it's a set of build tools for building a Linux distribution.
An operating system is not defined only by its kernel. It's kernel + APIs + user land. If one of those components is changed radically, it's indeed a new operating system. That's why Ubuntu and Debian are distinct, even if both are based on Linux.
You're correct. Debian and Ubuntu are technically different operating-systems. Due to the similarity of 'operating-systems' built around the Linux kernel they're typically referred to as 'Linux distributions'.
In the context of software development, if you tell someone you're developing a new operating-system you're probably going to conjure up images of writing a new kernel. If you tell people you're developing a new Linux distro, this is closer to what they'll imagine.
Exactly. It is yet another Linux distribution added into the list of many, but specifically tied to AWS. But the magic word that changes everything is something called 'Rust' but not what you actually think it is used for in terms of implementation when first looking at the HN title.
> ...a new Linux-based open source operating system that we designed and optimized specifically for use as a container host.
Even the points made for creating this distro was really for stripping out the unnecessary software in a default Linux distro install and to also increase the startup time for the essential userland processes and optimizing the OS from any possible bottlenecks and security pot-holes.
It's a shame really that it is built on top of Linux rather than an actual new Rust operating system by Amazon. I'm not sure why I would use this particular one if it is based on Linux while it also promotes another lock-in opportunity for AWS.
This is true. Android is a good example of where the line between a Linux distribution and an outright OS blurs. I'm not convinced that this really affects my overall point all that much though.
Linuxkit allows you to build your own appliance like OS, while Bottlerocket is more of an end user project. A project that is more similar to Bottlerocket is https://www.talos.dev or https://www.projectatomic.io
Any insights on the design decision to go with Wicked instead of systemd-networkd which is already provided by systemd and better integrated with the remaining systemd tooling/components/conventions?
It looks like this supports automated updates within a specific time window, but it's not clear to me how the "waves" are defined. (Note that this is something that is currently lacking in Fedora CoreOS: https://github.com/coreos/zincati/issues/34.)
I do wonder if the dual partition approach was deemed more stable than using OSTree or why the latter wasn't used.
Question for any Amazon folks here that may know. Is this akin to something like Atomic or CoreOS that is used within Openshift as the Master or Worker node OS or is this more like the UBI (Universal Base Image) that can be used as the base image of a container via "FROM" within a dockerfile?
You don't need Amazon folks, just read the material.
This is not a container base image, it's a container host OS. It is somewhat similar to Atomic or CoreOS, but in some ways it seems to be a bit more of a radical redesign than those.
So, the main feature is that updates happen for all packages at once and not for each package individually. Sounds interesting, even for non cloud setups.
How does that work? The explaining image does not explain that. How is that different from rolling back on file system level?
The update system is image-based; when an update is downloaded, it's written out to an alternate set of partitions, and then it can flip over to those partitions with a reboot. That makes it easy to roll back with the same kind of single flip, too.
It's different than filesystem-level rollbacks because it's all-or-nothing, so you don't have to worry about update failures after a few packages, and because all of the components in a given image are guaranteed to be tested together, whereas with package-based systems, your combination of packages may have never been used together by anyone else. In addition, for builders, it's easier to sign, distribute, and verify a single image.
Nix has upside that you just need to flip a symlink to do the same . Downside is that you don't have thinks like dm-verity that can prove that your update wasn't tampered with.
In nix the nix store is remounted over itself read-only, but nothing stops someone from ripping out the disk and flipping bits. This is not possible with these kind of 2-partition schemes if you have dm-verity set up
I'm curious about why Bottlerocket is building base packages like glibc, bash, util-linux, etc. from source, rather than just pulling binary RPMs from CentOS or Amazon Linux.
Amazon Linux and CentOS are general-purpose distributions and need more features built into the base packages than we do for Bottlerocket. We’re able to simplify the spec files and produce smaller RPMs with only the content and dependencies necessary for our more narrow goals.
I wish somebody'd take some VC or R&D money and build distributed computing features into the kernel itself, so we could quit wasting our collective engineering talent, time, money and energy on distributed applications that run on non-distributed-operating-systems. It's like nobody wants to work on creating a round wheel, so instead we're spending all our time building custom roads for square wheels.
Erlang is probably the closest thing to this, but unfortunately everything it runs also needs to be written in erlang. If erlang could run containers as processes it would take over the world, all of the features from kubernetes has been baked in since the late 90's.
The closest things really are distributed operating systems from the 70's and 80's, and cluster operating systems from the 90's, and 00's.
Unfortunately, all of them were either research projects, proprietary products, or patches that never made it into the mainline kernel. Nobody has since tried to get the functionality into mainline, so people keep hacking together these non-standard pseudo-operating-systems and jumbles of disparate applications.
You could eliminate 80% of the need for K8s by adding OS primitives to connect and operate namespaces and control groups between nodes, as well as native i/o (block, file, "N-way pipes") between nodes. Once that was done, systemd (or something like it) could manage services across an entire cluster. Applications could communicate between arbitrary nodes without any added functionality. Virtually all of the complexity would be in the kernel and systemd, so apps could be simpler and we wouldn't need 100 layers of userspace junk just to keep an app running on 3 nodes.
To answer your first question, AFAIK, standalone services that can be used on their own are prefixed with Amazon (eg S3, EC2, DDB), whereas services that are deeply integrated into AWS ecosystem are prefixed with AWS.
Disclaimer: I work for AWS, but this is not an official answer. What I said is correct to my best knowledge, but I cannot guarantee its correctness/accuracy.
We know this is how you show you care. I don't think there was any reason to downvote or flag. But not everyone on the Orange Site knows one another, so I can see how it could be misinterpreted.
This looks pretty sweet. It seems to be a continuation of the same trend of "just enough Linux to run containerd" that CoreOS started, linuxkit continued, then Project EVE expanded to cover virtualization.
It is also interesting to note that every step on that journey seems to have picked the coolest runtime to implement it in (C/early Go, established Go, and now Rust)
Started to dig into the implementation. First impressions so far: loving all the Rust harness - really nicely done and way better than buildroot/yocto/etc for creating tight, single-purpose linux images. Speaking of tight, here comes the bad news: really NOT loving all the over-engineered upstream components like D-Bus and systemd that seem to be there by default. In that sense #linuxkit with its Alpine base and strong attention to how bit the image is still comes way on top.
One more thing on the good side: the TUF implementation in Rust seems really interesting. I'll be digging some more and may actually steal it for linuxkit (and by extension Project EVE)
Fun fact: a lot of the patches you will find in more system level packages like grub seems to trace their lineage to CoreOS (and potentially Project EVE) but I haven't seen acknowledgments anywhere. This is of course all fine from licensing perspective -- but I still would be curious to know whether it is indeed where it was taken from.
> really NOT loving all the over-engineered upstream components like D-Bus and systemd that seem to be there by default.
I'm happy with the usage of systemd if they take advantage of the hardening features in systemd units for core system services. I'm a bit less happy about the continued usage of Docker, but I get why that's happening for this (EKS and ECS both use it, so it helps support that infrastructure).
containerd is the supervisor for the docker/moby container runtime environment. It is not used for crio (for k8s), podman (for non-k8s), or any other OCI container management engine (obviously excluding docker/moby).
You obviously know this, but for everyone else playing at home, "Docker" is made up of three distinct projects: moby (CLI and API), containerd (supervisor daemon), and runc (container runtime core).
Of the three projects mentioned above, only runc is used by nearly all major "container engines" as people call them.
And as pointed out by another poster, you do have the rest in the Bottlerocket tree.
Sorry, I should have been more clear. The docker packages are there for the development build of Bottlerocket. The Kubernetes variant does not use the docker packages for its build. See more about variants here: https://github.com/bottlerocket-os/bottlerocket/tree/develop...
I'll take Buildroot, and Bitbake first (Yocto itself is slight different we'll get to that later). Both Buildroot, Bitbake share the ultimate goal with all these news systems like linuxkit and bottle rocket. They all aim at producing single-image, pruprose built linux distros based on the very same usual suspects of upstream components. So the question really two-fold: out-of-the box availability of said upstream components (this is where we should stop talking about Bitbake and talk about Yocto -- Buildroot kind of commingles the two) AND the usability of the build harness itself.
So on #1, if you look very casually at Buildroot and Yocto -- they will clearly come on top over this next generation of systems. It appears they have WAY more upstream components already available for you to chose from. Compared to them the list here looks almost laughable https://github.com/bottlerocket-os/bottlerocket/tree/develop... and https://github.com/linuxkit/linuxkit/tree/master/pkg The problem though is combinatoric explosion of how you can compose all these upstream components. The canonical example here is the choice of your init system. You pick one -- and your choice in everything else gets severely restricted. So to some extent that apparent embarrassment of riches that Buildroot and Yocto offer is misleading.
These next generation systems, on the other hand, don't pretend that you can build a host OS in any shape or form you want (hence very few base packages) but rather that you build "just enough of Linux to run containerd" -- the rest of what you would typically put into your baseOS goes into various containers. This is a very different approach to constructing the bootable system, but subtly so -- which I don't think a lot of people on either side of this debate appreciate.
I honestly think that what makes Yocto and Buildroot difficult is that they want to be all things to all people and they want it at the level of baseOS -- complexity-wise, this is a wrong approach these days.
That scores one point for these next generation systems in my book.
The question #2 is not even a comparison. In Buildroot most of the integration/package logic is implemented in Makefiles and usability-wise (if you're trying to actively change the system or add a new package) it falls apart pretty quickly (it is still great if you're just using what's already there btw). In Bitbake -- the codebases is REALLY complex Python which suffers from the same issue. Contrast it with Linuxkit/Project EVE where all that logic is golang and bottle rocket which uses Rust and ask yourself whether you would rather debug a complex issue with a dozen of Makefiles all full of non-trivial recipes or look over go/rust codebase (yes, I know all these things are turing complete and thus equivalent -- but life is to short to debug Makefile).
If you don't quite believe me, there's been a number of studies about using Buildroot and Yocto for building containers. Pretty much all of them came back with the same conclusion -- the usability aspect of extending them makes it a non-starter. Here's the one from the last Kubecon that VMWare guys did: https://blogs.vmware.com/opensource/2020/02/27/distribution-...
> These next generation systems, on the other hand, don't pretend that you can build a host OS in any shape or form you want (hence very few base packages) but rather that you build "just enough of Linux to run containerd" -- the rest of what you would typically put into your baseOS goes into various containers. This is a very different approach to constructing the bootable system, but subtly so -- which I don't think a lot of people on either side of this debate appreciate.
This is a valid approach if you want to build something that can only run containers, but IMO is somewhat orthogonal to the Yocto and Buildroot goal of building distros for embedded platforms.
It's awesome that people are making new tools to do similar things to Yocto and Buildroot in this post-container world, but I don't think it's really fair to say that bottlerocket is a direct competitor to Yocto/Buildroot. It's probably fairer to say that bottlerocket makes it easier to do things that Yocto/Buildroot aren't really designed to do. Hopefully both live on, serving their own niche! I'm all for specialised tools rather than generic 'do it all' tools.
And here is a post from AWS with more technical details: https://aws.amazon.com/blogs/aws/bottlerocket-open-source-os...