I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use.
The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environment setups.
I even use it with Longhorn https://longhorn.io/ for shared block storage on the mini cluster.
If anyone uses it with MicroOS, just make sure you switch to kured https://kured.dev/ for the transactional-updates reboot method.
How has your experience been with Longhorn? Performance, flexibility, issues, maintenance...? I'm interested in moving away from a traditional single-node NAS to a cluster of storage servers. Ceph/Rook seem daunting, and I'd prefer something easy to setup and maintain, that's performant, reliable and scales well. Discovering issues once you're fully invested in a storage solution is a nightmare I'd like to avoid. :)
Ceph is a nightmare if you don’t set it up exactly how the docs say - and in fairness, the docs are excellent.
My advice, having done Ceph/Rook, Longhorn, and now Ceph via Proxmox is the latter, assuming you have access to an actual host. Proxmox-managed Ceph is a dream, and exposing it to VMs and then K8s via RBD is easy.
Longhorn is fairly easy to set up, but its performance is terrible in comparison.
Thanks for the insight. I've tried Proxmox before, and as much as I appreciate what it does, it's mostly a black box. I prefer a more hands-on approach, as long as the docs are comprehensible, and the solution is simple enough to manage without a special degree. Ceph always seemed daunting, and I've mostly ruled it out, which is why these k8s-native solutions are appealing.
Good to know about the performance. This is not critical for my use case, but I guess I'll need to test it out for myself, and see whether it's acceptable.
Ceph is ... well, it's an amazing journey. Especially if you were there when it started, and watched as each release made it more capable & crazier.
From the earily days of trying to remember WTF the CRUSH and "rados" acronyms actually meant and focusing on network and storage concerns and [0] ... architectural optimizations [1] ... a RedHat acquisition ... adjusting to the NVMe boom [2] ... and then Rook, probably one of the first k8s operators trying to somehow operate this underwatere beast in a sane manner.
If you are interested in it ... set it up manually once (get the binaries, write config files, generate keyrings, start the monitors, set up OSDs, RGW and you can use FileZilla or any S3 client). For some more productionish usage there's a great all-in-one docker image https://quay.io/repository/ceph/ceph?tab=tags .
[0] oh don't forget to put the intra-cluster chatter on an internal network, or use a virtual interface and traffic shaping to have enough bandwidth for replication and restore, and what to do if PGs are not pairing and OSDs are not coming up]
[1] raw block device support, called BlueStore, which basically creates a GPT partitioned device, with ~4 partitions, stores the object map in LevelDB - and then later RocksDB
[2] SeaStore, using the Seastar framework, SPDK and DPDK, optimize everything for chunky block I/O in userspace
> I prefer a more hands-on approach ... which is why these k8s-native solutions are appealing
"K8s-native" here implies Rook, which is in no way hands-on for Ceph.
> as long as ... the solution is simple enough to manage without a special degree
Ceph is not simple, that's my point. I assume you've read the docs [0] already; if not, please do so _in their entirety_ before you use it. There are so many ways to go wrong, from hardware (clock skew due to BIOS-level CPU power management, proper PLP on your drives...), to configuration (incorrect PG sizing, inadequate replication and/or EC settings...) and more.
I'm not trying to dissuade you from tackling this, I'm just saying it is in no way easy or simple. Statements like "k8s-native solutions" always make me cringe, because it usually means you want to use an abstraction without understanding the fundamentals.
To be clear, I have read the docs, set it up on my own, and decided I didn't want to try to manage it. I've ran a ZFS pool for a few years on Debian and shifted over to TrueNAS Scale last week; not because I was unable to deal with ZFS' complexity (knock on wood, the only [temporary] data loss I ever had was an incorrect `rm -rf`, and snapshots fixed that), but because of continual NFS issues. I may yet switch back; I don't know - I just no longer had the time to troubleshoot the data layer. Ceph makes ZFS looks like child's play in comparison.
By "k8s-native" I meant solutions that were built from the ground up with k8s in mind. Rook is built on Ceph, and as such requires knowledge and maintenance of both, which seems like much more difficult than managing something like Longhorn.
I think you misunderstood me. I'm not inclined to give Ceph/Rook a try because of its complexity, regardless of the state of its documentation. I was refering to Proxmox in my previous comment, in the sense that I don't need a VM/container manager/orchestrator with a pretty UI. If I'm already running k3s, I can manage the infrastructure via the CLI and IaaC, and removing that one layer of abstraction that Proxmox provides is a positive to me. Which is why adding just a storage backend like Longhorn to a k3s cluster seems like the path of least resistance for my use case.
Ultimately, I don't _want_ to deal with the low-level storage details. If I did, I'd probably be managing RAID, ZFS, or even Ceph. For my current NAS I just use a single JBOD server with SnapRAID and MergerFS. This works great for my use case, but I want to have pseudo-HA and better fault tolerance, and experiment with k8s/k3s in the process.
So I'm looking for a k8s-native tool that I can throw a bunch of nodes and disks at, easily configure it to serve a few volumes, and that it gives me somewhat performant, reliable, and hopefully maintenance-free, block or object-level access. Persistent storage has always been a headache in k8s land, but I'm hoping that nowadays such user friendly and capable solutions exist that will allow me to level up my homelab.
Also, I feel obligated to ask “are you me?” at your stated homelab journey. I also went from mergerfs + SnapRAID, albeit shifting to ZFS. I also went with k3s, but opted for k3os which is now dead and thus leaving me needing to shift again (I’m moving to Talos). Finally, everything in my homelab is also in IaC, with VMs buiit by Packer + Ansible, and deployed with Terraform.
Happy to discuss this more at length if you have any questions; my email is in my profile.
I've run Rook/Ceph, and I run Longhorn right now. I wish I didn't, and I'm actively migrating to provider-managed PVs.
My advice for on-prem is to buy storage from a reliable provider with a decent history of hybrid flash/ssd, so that you can take advantage of storage tiering (unless you just want to go all flash, which is a thing if you have money).
If you must use some sort of in-cluster distributed storage solution, I would advise you to exclude members of your control plane from taking part, and I would also dedicate entirely separate drives and volumes for the storage distribution so that normal host workload doesn't impact latency and contention for the distributed storage.
Good points, thanks. What makes you wish you didn't use Rook/Ceph/Longhorn?
In a professional setting, and depending on scale, I'd probably rely on a storage provider to manage this for me. But since this is for my homelab, I am interested in a DIY solution. As a learning experience, to be sure, but it should also be something that ideally won't cause maintenance headaches.
Keeping separate volumes makes sense. I can picture three tiers: SSDs outside of the distributed storage dedicated to the hosts themselves, SSDs part of distributed storage dedicated to the services running on k3s, and HDDs for the largest volume dedicated to long-term storage, i.e. the NAS part. Eventually I might start moving to SSDs for the NAS as well, but I have a bunch of HDDs currently that I want to reuse, and performance is not critical in this case.
>Good points, thanks. What makes you wish you didn't use Rook/Ceph/Longhorn?
It seems like my volumes are constantly falling into degraded and then rebuilding. Resizing volumes requires taking the workload that's attached down, and then it seems to take forever (15m+) for my clusters to figure out that the pod is gone and a new pod is trying to attach.
Really, it's a PITA and all of the providers' storage classes seem better than Longhorn. Ceph I had less experience with but very similar problems - long-gone pods held a lock on PVCs that had to be manually expunged, or wait for a very long timeout.
I've had similar issues with Mayastor (another in-cluster storage solution). It's under heavy development, so I've assumed the more mature options were better.
I'm working on v2 of my homelab cluster, and I'm going with plain old NFS to a file server with a ZFS pool. Yes, I will have a single node as a point of failure, but with how much pain I've had so far I think I'll be coming out ahead in terms of uptime.
I can't speak to performance because the workloads aren't really intense, but I run a small 3 node cluster using k3s and Longhorn and Longhorn has been really great.
It was easy to setup and has been reliably running with very minimal maintanence since.
If you use longhorn, make sure to enable the network policies when installing the helm chart. For some odd reason, these are disabled by default, which means ANY pod running on your cluster has full access to the longhorn manager, API, and all your volumes
I wouldn't really treat it as a replacement for a NAS, mostly only for the container workloads running on kubernetes itself... Ideally, any apps you develop should use something more sane like object storage (Minio etc) for their data.
I push it pretty minimally right now, so no great performance testing myself, and I do run it in synchronous mode, so that means its write performance is likely going to be limited to the 1gbps network it syncs over.
Funny: I've been running a Talos cluster for the past six months, and just today decided to look into k3s. Talos has a lot of really nice things, but I have found that the lack of shell access can be frustrating at times when trying to troubleshoot.
I had a situation where the etcd cluster got hosed, making it basically impossible (at least with the ways I know) to interact with the k8s API at all. So I didn't have any way to get a privileged pod running.
Ah gotcha. Haven’t had to deal with that yet. Maybe possible to add a static pod via the machine config? But yeah it’s basically throwing out a bunch of linux admin muscle memory.
Kudos to you. I feel like setting things up on real hardware is somehow needed in order to make things concrete enough to full understand. At least for me (I fully admit this may be a personal flaw) working with a vm on in the cloud is a little too abstract - even though eventually this is where things will land.
Re: Talos persistent storage, why not run it as a VM and pass in block devices from the hypervisor? You also then gain the benefit of templated VMs that you can easily recreate or scale as needed.
"Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them.
Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code.
Although, it's hard to recommend Nomad for future uses as it sadly stopped being FOSS :/
Nomad seems to be a truly "lightweight Kubernetes"
k3s requirements: 1 node, 512MB RAM, 1 CPU core
Nomad requirements: "Nomad servers may need to be run on large machine instances. We suggest having between 4-8+ cores, 16-32 GB+ of memory, 40-80 GB+ of fast disk and significant network bandwidth."
That can be said of many systems, does hashicorp document it though? I’m guessing they must, otherwise that would be the first thing you hear on the other side of a support call.
That's a bad take and you know it. HashiCorp couldn't make money because their enterprise offering was slim on features and overpriced. Rather than compete, they're playing protect-the-castle with some aggressive plays to keep their company afloat. They're bad stewards of open source (always have been) and are now getting their just desserts.
This is for large deployments in prod.
Nomad and Consul servers at home are running on an old Rpi3 with 2G of ram managing several devices and about 10 vm's.
Larger deployment of ~30vm's was managed with a nomad/consul cluster of 3x2cpu4ram with no issues.
I've commented on this before in a different k8s thread (one about a k8s outage) but something that bears repeating is that the entire job market is Kubernetes.
My personal experience is that it is very, very hard to find a job right now if your professional experience is primarily non-k8s orchestration systems. Most job positions out there require deep Kubernetes knowledge as well as hands-on experience with different supporting products like ArgoCD and Flux.
I chose Nomad for a large initiative at my current employer and it is honestly pretty amazing given the use case but I regret choosing it because I feel like I'm unhirable now given that every devops/SRE/platform engineering position open on the market (each one with hundreds of applicants) is heavily Kubernetes focused.
A lot of smaller SaaS companies avoid k8s like the plague and only start using it once they scale to a certain size. K8s might be ubiquitous past a certain scale, but there are many jobs operating below the threshold for which the complexity of your system with k8s outweighs the complexity of your system without k8s.
If you find yourself on the Google cloud for some reason, GKE Autopilot is such a great hands-off solution that most alternative solutions will probably be more complex.
I only recently had to use AWS EKS, and am pretty sure that many people dislike k8s because that's the main incarnation of it they experience (how is there no pre-installed ingress, and even the setup for their own load balancer is a 10+ step manual process?).
It is really weird that the AWS Load Balancer Controller isn't an EKS Addon, and the permissions requirements being in the base requirements for EKS. Also, the only officially supported CNI is the VPC CNI.
And if you want to use EKS-A (EKS-Anywhere) it's even more effort to do the AWS IRSA setup.
Installing an ingress controller that will do SSL is a steep cliff preventing use with so many cloud Kubernetes offerings. I don't know why it needs to have dozens of steps, roles, and resource definitions.
Yeah, though I have to admit that even though you get SSL certificates "out of the box" with a GKE + GCP load balancer setup, their provisioning has always been really slow for me (usually 1-2h, sometimes half a day). In comparison self-installed nginx-ingress + Let's Encrypt provisions in minutes.
Cloud Run is just Knative which is also Kubernetes, so in the end it comes out to about the same. I used to use Cloud Run, but in the end there were usually always some things that ended up making GKE the more convenient choice (e.g. access to Helm charts).
If that is so then the industry has a lot to recon with soon because that would mean most at chasing a shiny new thing for absolutely all the wrong reasons.
K8s is complete overkill at best for the majority of companies/workloads and introduces lots of other dependencies on teams in terms of workflows and architectures to make things not be a dumpster fire for all but very mature teams.
I keep hearing this viewpoint, but I really don’t understand it. I use k3s for all sorts of personal projects, specifically because it makes things easier for me. I don’t have to worry about the underlying OS. I can focus on the application that’s actually providing value for me.
On the professional side, k8s handles orchestration issues far better than anything else I’ve worked with. Using autoscaling instances is a nightmare by comparison, and requires most of the same initial effort to do it right.
There is probably a middle ground where it requires a certain amount of complexity in the K8s configuration that isn’t worth it compared to other platforms (especially on bare metal), but I haven’t found it, yet.
I think it’s easy to forget how much goes into running even a simple application with high uptime requirements. Kelsey Hightower makes some great points about this here:
Nomad also doesn't have nearly half of the features that Kubernetes does. Need service discovery? Set up a Consul cluster. Need secret management? Install vault. Need vault enterprise? Install a separate Consul cluster! This was a few years ago, maybe it's changed? I dunno.
Anyway, lightweight here means that whole bunch of external dependencies have been ripped out. E.g. AWS/GCP/Azure and I believe other things too.
You can have sd and secrets with just nomad now, it is perfect for small projects in that way.
K8s also can do secrets! base64 encoded secrets! But in case you need a more secure solution. Please pull in a Vault helm chart or mess around with sops or whatever. Want to scale out? See how long it will take you until you will have to move your etcd to dedicated hw/vm's. And let me tell you - setting up external etcd in prod sucks. Also maintaining mission control, because there is a bunch of ways to do things and devs love using things that they are not supposed to touch.
Nomad approach is modular this is why it works. Need proper multi dc SD? please run a consul cluster (single binary + single config file)x3. Need mesh networking? same. Need secrets? Vault (single binary + single config file)x3. Pki? same. Or maybe you don't need it? Then you have an option to just not use it.
One thing that i really don't like about it that there is not a lot of docs describing all the ways these tools can interact with each other.
K8s supports encryption at rest, ACLs and audit logs.
What it actually lacks is versioning, and a dynamic secrets engine, though you could build that with an operator, and things like the postgres-operator do.
You also don't get the whole sealing thing, but I'd argue that's more annoying than useful.
Unsealing is the process of obtaining the plaintext root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.
Prior to unsealing, almost no operations are possible with Vault. For example authentication, managing the mount tables, etc. are all not possible. The only possible operations are to unseal the Vault and check the status of the seal.
You can run k3s with a single node. In that case it uses sqlite instead of etcd which is great for a smaller resource footprint. If you're comparing k8s distros, you'll be hard pressed to find a setup that uses fewer system resources.
Is it actually smaller though? For those non etcd data stores, Kine is often the component doing an etcd to database translation. Im my experience using it generated a load that was greater than a one node etcd instance.
> Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code.
Does Nomad expose an API that you can extend with controllers running inside the cluster? Because Kubernetes without operators is not Kubernetes.
> Because Kubernetes without operators is not Kubernetes.
Alright, so, what is it, then? I've been unfortunate to work at firms that generally have a minimal level of competency with Kubernetes, but across several billion dollars worth of firms, not a one has used operators in any capacity, but they leverage Kubernetes substantially. Help me understand the gap, would you?
> While Nomad doesn’t support the idea of Custom Resource Definitions, we can achieve an operator by utilising a regular Nomad job and the nomad HTTP API.
Kubernetes operator is a combination of a CRD reconciled by a controller. Without the CRD it's simply a controller.
I do like the linked post, it's well written and a valiant effort, but not close to comparable.
> "Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them.
Among its many strengths, core services for running Kubernetes running inside Kubernetes itself is one of its greatest.
> it's easy to understand all the moving pieces
It's legit to leverage your pre-existing knowledge. Nomad + Linux Pets + Other Hashichorp Pets works well.
There are many ways to run an application. Kubernetes + Machine Cattle, in my experience having started from zero, is superior to any other application deployment paradigm I've ever used.
Treat your machines like cattle, not pets. When I used to run kubernetes in the pre-'kube as a service' days. We had two autoscaling groups, one which had 3 or 5 nodes in it to run the control plane nodes and an open eneded one for workers (which were managed by the cluster autoscaler). Each ASG was set up in such a way that the nodes would come up and join the cluster fully automated and they never needed to be shelled into for anything but troubleshooting the weirdest failures (it was early days, the kubelet and docker had some extra fun bugs).
Ok sure, I'm very familiar w the cattle vs pets analogy / meme -- but the (capitalized!) "Machine Cattle" made it seem like a reference to a particular software tool or product.
Nomad is an alternative to k8s, but it is not really a "lightweight k8s". Nomad and k8s are way to different to call them versions of eachother.
And the whole license thing. Nothing changed for end users.
I think the tagline can be a bit misleading, you can run k3s on devices traditionally considered IoT (like the raspberry pi) but it will run on big heavy x86 servers too.
The amount of services running is not an indication of how lightweight the underlying tech is. K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it.
Are you trying to say that 10 services and 2 nodes would be less on Nomad?
> K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it.
That has not been my experience with k3s. Unlike normal distributions, k3s servers do allow workload to run on them unless you manually disable. From https://docs.k3s.io/datastore/ha-embedded :
"
An HA K3s cluster with embedded etcd is composed of:
- Three or more server nodes that will serve the Kubernetes API and run other control plane services, as well as host the embedded etcd datastore.
- Optional: Zero or more agent nodes that are designated to run your apps and services
"
So you can have a full HA k3s cluster with only 3 server nodes and 0 agent(worker) nodes.
This is not true of k8s in general, maybe certain on prem 'distributions' enforce this, i'm not sure. Sometimes kubernetes admins/operators who are running their own control planes and etcd will run etcd on non k8s controlled nodes, or they will run it on the same nodes that run their control plane services (scheduler, apiserver, controller manager, etc). Often those nodes will be tainted in such a way that they don't get overloaded by a noisy neighbor 'customer pod', but that's in no way 'by design'.. perhaps by convention or best practice.
Nomad is a lightweight container orchestrator, not a lightweight Kubernetes. The whole point is to keep syntax, terminology and config options etc as similar as possible between k8s and k3s.
If Nomad provides workload scheduling, a runtime, workload networking, service networking, and some sort of overall orchestrator/controller plus an API, then it has these same services. Running them in one process instead of ten doesn't make it more lightweight.
You're free to run either on one node, but other than for toy purposes like demos and learning, why use a cluster compute system when you're not actually running a cluster?
Interestingly, SUSE now owns Rancher [0], so k3s has been backed by a large company for some time now. I've never tried k3s, but I have always thought it's probably the most loved-by-its-users version of Kubernetes.
If you want to easily try it out, we baked it into Rancher Desktop [1]. RD let's you run Kubernetes (via k3s) and containers on your desktop. You pick the k8s version.
We use it as a free replacement for Docker Desktop and it works very well.
Unfortunately, we cannot use the Kubernetes part of it because it comes with "steve" (used for dashboards), which uses and blocks (!) the same ports as our application, and those ports are not configurable.
So, please please solve this request here: https://github.com/rancher-sandbox/rancher-desktop/issues/18...
I just made an account here to say thank you. Rancher desktop is amazing on my M2 mac, it's easy to use, solves a bunch of challenges from k8s to docker and has been surprisingly reliable for a young product.
I've honestly been scared to switch from Docker Desktop to Rancher Desktop on Windows. Like do I have to switch or do they play nice together? Do I need to install Docker in WSL to use it, or kubectl, or is it like Docker Desktop where it "just works."
You can have Docker Desktop and Rancher Desktop installed on the same Windows machine. Last I checked, you can't run them both at the same time due to docker socket locations (assuming you run Rancher Desktop using Moby).
Rancher Desktop will provide the CLI tools to work with it.
The idea is "it just works". It's a desktop app so it installs what you need.
K3s is fantastic especially for local development with Kubernetes when orchestrated using k3d. This is what we use for most of our internal K8s testing at OpenC3.
Yes, I recently moved my Dev team to K3D as well. Previously we were using AWS clusters that were spun up on demand every time someone needed to test something. the time needed to do this grew and grew overtime eventually coming close to 30 to 40 minutes. With k3d testing locally can be done in a minute. Plus I learned a ton about k8s while doing this.
What you've listed for k3s is mostly included in k0s. I wouldnt go far to say k0s isnt convenient.
* A helm controller is included in k0s
* Etcd is bundled and bootstrapped automatically which I perfer because I dont want the overhead of the translation that Kine does. Although Kine is available for a non-etcd datastore if that is preferred.
* Upgrade controller is included (autopilot).
* They have a local storage provider based on OpenEBS.
* Ingress is missing, but due to the built in helm controller that can be boot strapped upon cluster initialisation.
Overall, together with k0sctl and its declarative configuration it is easier to deploy k0s than it was k3s.
Kine (https://github.com/k3s-io/kine) is a shim or an external process (when not k3s) that translates the etcd api to enable compatibility with a database or alternative data store.
Kubernetes natively talks etcd, so this translation is what enables its usage with sqlite or another database, but it incurs an overhead.
I don't have specific numbers unfortunately since it was years ago I benchmarked Kine against etcd. But I had a better results with etcd both in cluster and single node.
There are lighter Kubernetes "distributions" (kind, minikube), but what makes k3s special is that it's (a) packaged as two binaries that provide all of the Kubernetes components in one, and (b) it's 100% suitable for production use.
Lots of teams are using K3s to run Kubernetes at the edge and in IoT applications, and with good reason. It's a fantastic Kubernetes distribution that's well-maintained, easy to get going with and well-documented.
(Ironically, if you look at the first commits to kubernetes/kubernetes, the Kubernetes components were originally shipped as a single binary. They decided to break them up later to simplify releasing, but the k3s monolith lives on.)
My only worry is the 10+ domains (involving 4+ major companies) being used for import paths. I know there are solutions but that would be a mess to figure out 10 years from now when half those links are broken or changed subtly.
I've been running k3s on my home server and it's been painless to set up compared to other options (e.g. kubeadm) while also being very lightweight. For single-node setups, it defaults to Kine instead of etcd, using SQLite as the database. This removes a significant chunk of overhead for dev clusters and running in tiny devices.
It also has Traefik set up with sane defaults, and the local path provisioner is also pretty good, too. But recently I've moved to Longhorn since I plan to eventually scale past 1 node. My only complaint about Longhorn is that applications that are write-heavy and delete old data (e.g. Prometheus with short retention) will require aggressive trimming (e.g. trim once a day) to keep the actual size of volumes down. Besides that, Longhorn makes backups to S3-compatible storage very effortless and you get RWX volumes, too!
Regarding k3s itself, you can persist modifications to the way some components (e.g. Traefik) are installed through a HelmChartConfig CRD. This is what I personally use so I can use Traefik to route SSH traffic for Forgejo. Another nice thing is that although components like kube-proxy are baked in to the single k3s binary, you can still scrape metrics with Prometheus provided that you expose their endpoints somewhere on your cluster network.
(I tweak this every now and then since I both used it as a training sample for my customers/peers and as a way to run my own batch processes as cheap as possible)
This was my introduction to K3s as well. RPis running K3s with enough resources left over to actually do some small tasks. I hosted a small data pipeline that analyzed trading data from an MMO. It was as fun as it was impractical, and I learned quite a bit.
Can this tool help to simplify self hosting implems? K3s was recommended to me to replace my personal pile of systemd units starting docker compose configs and manual reverse proxy configs.
Check out https://kamal-deploy.org, it just hit 1.0 and 37signals moved their whole Kubernetes stack to it. I was playing around with it recently for side projects and I think it's a nice fit for simpler products like that.
Lol no. Docker Swarm, Nomad, even ancient Pacemaker are all simpler and easier to use than k8s. I don't think there's a single feature k8s adds over those options that is actually useful in a non-Enterprise use case (especially homelab stuff, assuming the point of your homelab is not learning k8s).
I use it for self hosting. It really simplified my reverse proxy config like you said. I use the internal networking between services quite a bit. I have it auto-provisioning volumes on my NAS and using them via NFS. I love it to death and would say it has simplified my setup overall. But the upfront cost of learning k8s is indeed high.
Seconding this as well. Similar setup (though I opted for longhorn) and journey. I'm very grateful for learning k8s but man, it was definitely rough for a long while until I really understood all the pieces and cemented my understanding. I can very much appreciate that's a tall order for many people and not wanting to embark on that journey unless there's a good reason.
Start with hashicorp nomad. Less of a learning curve and easy to set up. Traefik can do automatic discovery of services in nomad. Definitely better than a bunch of systemd units.
I had a great k3s experience with a silly, over engineered weekend project to automate my fog machine with motion sensors.
I connected motion sensors to battery & wifi enabled RPis, built a remote circuit to control the fog machine, and ran a k3s cluster with NATS to bring it all together. 10/10 would do again.
I've been closely following the discussion on k3s and Kubernetes in general. I recently acquired an M1 Mac Ultra, and I'm curious about the best options available for running Kubernetes locally on it.
Does anyone have experience or recommendations in this area? I've heard about a few tools but wanted to gather insights from this knowledgeable community.
I switched to it completely, it’s very convenient to have both fast (-est on Mac) Docker support and a really smooth VM setup for running occasional Linux tools (such as Yocto in my case).
Edit: added some background info to my recommendation.
If you go the kind route, be sure to not use Docker for mac and instead opt for podman which is much more resource efficient. Now that I've switched over to podman, my computer doesn't sound like it's about to blast off when I'm running clusters.
The tools are a bit rough around the edges if you try and do something outside of the happy path with them. Nothing bad as such, just the user experience isn't as seamless when say running the VMs on custom, addressable host networks or managing vms with launchd.
hetzner-k3s : This is a CLI tool to quickly create and manage Kubernetes clusters in Hetzner Cloud using the lightweight Kubernetes distribution k3s from Rancher.
https://github.com/vitobotta/hetzner-k3s
Agreed:( Someday I'd like to see how hard it would be to use microos to build a replacement cheaply (cheap in terms of maintenance; ideally it would just be a combustion config or so that deployed k3s on vanilla microos)
> And while k3s sounds easy, it's not after even a slightly larger scale.
Agent nodes failing to join after reboots or long network partitioning. Manual re-joining works. Additionally, hard to debug issues with the supplied Traefic.
Lightweight is a misnomer. Running even the most basic of tasks requires an insanely complex setup with vxlan, unintelligible iptables and bridge rules, all taking up a great deal of resources on something like an embedded x86/arm devices
If I remember correctly, the new F5 BIG IP instances also internally use k3s under the hood. That's probably it's efficient resource usage. But this is also an evidence, at least for me, on the stability of k3s.
If you want something lighter than k3s, k0s could be worth a try. I recently switched to k0s and noticed the controller, and the various kube processes consume less cpu resources compared to when I was running k3s.
k3s users: I tried using k3s a few years back and on a fresh single node setup I was seeing ridiculous system usage with 0 deployments. Something like 20-30% cpu load. The same system with a k8s deployment using kubeadm was 3-5% without a deployment.
There's this issue that goes back to 2019 that got closed without any real resolution. I moved on but I'm curious if anyone else has measured this?
k3s is my preferred way to run k8s locally, for the purpose of developing helm charts locally. It's way more lightweight than minikube, and with k3d it's way easier to use as well.
Is the overhead of Kubernetes worth it enough to make it smaller, presumably to run in less resource intensive environments? Docker compose fits this bill nicely and is much more simple.
Kubernetes is offering much more features than Docker compose. And then you have all the kubernetes operators and other extensions that makes it very complex and powerful.
It’s fine to use docker compose if it does what you need. If you start hacking weird things using Docker compose, it’s perhaps time to suffer and learn kubernetes.
vCluster is an interesting related technology. It spins up k3s inside Kubernetes. Handy if you want a lightweight way to manage multiple staging environments.
There's no real "over kubernetes". K3s is a kubernetes implementation.
You have to think of k8s as more of a specification of what can be done with multiple implementations. k3s, kind, microk8s, minikube are all implementations of kubernetes that you might want to try out.
k8s is much like the original IBM pc or a linux distribution. There are a bunch of interchangeable parts that you may land on depending on your needs.
Why use this over something else? I've found k3s to be really simple to setup and upgrade, which is why I use it for my home lab.
It makes sense to use full Kubernetes in that case. The lightweight distributions are good for single instances or limited multiple nodes. Like homelabs, edge nodes, or local development. But hosting application in data center makes sense to use full Kubernetes.
If you are in a datacenter, it's best to use the K8s distributed by that datacenter (EKS for AWS, for example).
The EKS provided by a datacenter will have integrations with external to Kubernetes resources you will almost certainly want (like storage, DNS records, or load balancers).
k3s and microk8s definitely support multiple node setup. I'm not sure about kind or minikube (Maybe? You'd have to look into the docs). How ready they are and how well they'll scale out will depend a lot on how the controllers are setup.
k3s, at least, supports multiple controller nodes on setup.
Where things really get squirrelly, at least with k3s, is storage and ingresses. k3s supports both but in a super robust fashion.
Kubernetes is sort of like Linux. There are distributions that excel in different areas. For example, RKE2 is for those that prioritize security. It deals with compliance and security things. There are numerous distros right now. The CNCF even has a certification program and listing for the certified ones.
K3s is the lightweight distro. You can install and run it from a single binary on pretty much any Linux distro.
> We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10 letter word stylized as k8s. So something half as big as Kubernetes would be a 5 letter word stylized as K3s. There is neither a long-form of K3s nor official pronunciation.
> We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10-letter word stylized as K8s. So something half as big as Kubernetes would be a 5-letter word stylized as K3s. There is no long form of K3s and no official pronunciation.
Either you use it a lot (likely at work) and are therefore comfortable using it outside of work as your one-size-fits-all solution to everything, or else you don't use it a lot and see it as a complexity behemoth that hides all of your system behind esoteric CLIs and config files that just make everything a nightmare to manage.
I'm tired of seeing comments with an apparent layer of objectivity which very quickly goes away as you reason about it for a second. You could've just said "I don't like K8s, it's too complex for my taste" but then your comment would be gray for being short and concise, so you decide to say this.
Opinions are fine and you may as well hate K8s or any tech being discussed. I just hope to see less of "people who like this just because they don't know any better vs the smart ones like me who know The Truth™" comments.
That wasn't the intent of my comment at all. I don't hate k8s - I've never used it beyond experimental fiddling. I'm sure if I used it at work all day I would love it.
I'm trying to grapple with the fact that k8s is apparently ultra popular technology, yet every time I try to read up on it or start using it, it quickly turns into a suck fest in terms of learning curve and complexity. So my conclusion is that the people using it "for fun" must have acquired a taste for it first by using it all day at work and I wanted to see if anyone who acquired the taste might chime in as to what made them see the light.
I find it very acceptable on my local servers that run my mail, web, mastodon, etc servers. I prefer the configuration in kubenetes templates instead of spread all over my filesystem in /etc (config in one place and fucking unit files in another), /var/lib, /opt, etc.
Right, you've acquired the taste. But why should people who haven't acquired the taste do this? To me seems like a mountain of unneeded complexity compared to just running a local webserver directly in a docker container with "docker run --restart always <image>".
Because when your docker server gets smoked, what happens? If you're using Swarm or Nomad or k8s, there's an argument there. `docker run` isn't.
I've very recently moved over to a home k3s cluster--a couple old desktops and some relatively new ARM SBCs with NVMe slots. I don't use it at work, at least not directly, but it's been the least-painful solution I've found to run stuff; once you understand the model (not trivial but not crazy) and find the headspace to explain why things in k8s are the way they are, reasoning through it is pretty easy (which is really just the story of any moderately complex system, exacerbated in a few ways because of some Google-attitude carryover), and I can also yank half the machines out of the cluster before an application fails.
(Both Swarm and Nomad are fine tools, but the lack of easy resources to get up to speed led to me failing them out of consideration.)
> Because when your docker server gets smoked, what happens?
In the context of local or home servers: my raspberry pi currently has like a 3 year uptime now, but if it randomly croaked I'd just buy a new raspberry pi, docker pull the image, and get it up and running again.
So you are saying the main benefit of k8s is that if some of your hardware in the cluster dies then your services are automatically still up and running? Isn't a home k8s setup vulnerable to the same failure as well if the master node gets smoked?
Seems like a lot of complexity cost for something that happens extremely rarely in a home setup at least (I could see how it would be useful for high reliability cloud systems).
- I have three control planes, not one, for about 20 x86-64 and 24 arm64 cores and ~160GB of RAM across 7 nodes. For much the same reason as my NAS uses raidz2, I don't have to drop everything and fix the universe if something fails because Home Assistant just keeps going.
- With Longhorn, I have three instances of every data volume in the cluster. Longhorn also provides incremental data backups to an S3 server--Minio on the NAS, in my case, which replicates to an offsite.
- I can run things like CrunchyData's Postgres operator, which provides a solid baseline setup and additional features like point-in-time restore--and as I run personal but public-facing apps on the cluster this is beneficial.
- Logging and monitoring are centralized and were easy to do. I don't have to go attach to a container, I just go look at Grafana.
- I have a consistent interface for working with all my applications, and I like it more than Swarm's by a lot.
By having this cluster, made out of spare computers, some cheap NVMe drives, and a couple SBCs I wanted to experiment with anyway,I've removed everything I care personally about from cloud providers except for offsite backup and my email sending. I know where it is, I can hack on projects trivially, and I know it'll be there tomorrow so long as the house doesn't burn down.
The benefit to me is if one of my nodes physically dies i have enough room on my cluster to still run all my pods, they just automagically reschedule to another node.
If my master dies then I have to do a little extra work, but not much.
Turns out to run the stuff I run I need more than 1 machine :).. luckily a couple of years back when my job was dissolving a dept and had a stack of uneeded NUCs headed for recycling (pretty nice ones, i've added a little ram but that's it) I tossed them in a box and took them home instead. I've also added a couple of PI/Arm nodes just for kicks.
-can run any workload
-on any hardware
-at any scale
-with load balancers
-and persistent disks
-and auto scaling
-and resource limits
-and certificate management
-and no single point of failure
-and etc.
requires a lot of esoteric complexity. Kubernetes did not create this complexity, but it is solving it in a standardized and universal way.
The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environment setups.
I even use it with Longhorn https://longhorn.io/ for shared block storage on the mini cluster.
If anyone uses it with MicroOS, just make sure you switch to kured https://kured.dev/ for the transactional-updates reboot method.
I'd love to compare it against Talos https://www.talos.dev/ but Talos's lack of support for a persistent storage partition (only separate storage device https://github.com/siderolabs/talos/issues/4041 ) really hurts most small home / office usage I'd want to try.