I'm currently bumping the versions manually (via a commit), but I plan to automate that with system upgrade controller [1] and Dependabot [2] (or similar)
To everyone saying that Kubernetes is unnecessary, try implementing autoscaling, service discovery, secrets management, and autohealing in a vendor independent way without it.
Of course none of that is necessary for a self hosted home lab, but neither is gitops.
This is a very nice example of how to set stuff up properly.
OP, I would love to see Robusta (https://robusta.dev) as part of this too. It's definitely in line with your vision of automating everything, as it let's you automate the response to alerts and other events in your cluster. (Disclaimer: I'm one of the maintainers)
Oh. I’ve actually done that. It’s not as hard or complicated as you claim.
This was before kubernetes, arguably learning kubernetes is harder than implementing auto discovery and auto healing because you have to learn a lot about cluster semantics, the various network providers, service accounts, role based access control, integrations with secrets manager or the very disparate implementations of custom operators which may operate in your environment.
Kubernetes is a platform, I don’t doubt it solves a lot of problems. But it’s a complicated wee beastie, and I say that as a person who did it the easy way and used GKE on Google Cloud
I don't know how you implemented it, but the big innovation of kubernetes was doing this with declarative inputs and using control loops for reconciliation instead of edge triggered changes.
It might not matter on a small scale, but on a large scale that's what makes it robust. It's also not trivial to implement a distributed version of this in house at scale. At the very least you need something like etcd at the core
The way we did it is that we registered to a centralised “name” service which was an in memory database.
Failure to register to the name service leads to registration towards the next name service in the list, we ran three- so there were two spares, the library would then keep trying to register to the first name service, this would be our reconciliation loop.
When you request a service you request “traits” through the name service.
Not “trivial” but even though it was implemented in C++ on windows it was much less complicated than kubernetes discovery systems, which there are many ways to integrate (env-var, dns, api) and some should or must be disabled for security reasons (eg. The environment variables that display all services) or can change depending on RBAC and service account.
We deployed about 300,000 or so services with this infrastructure and it’s been running in production since 2015.
(Original product release was supposed to be 2013 but due to the client changes it was delayed, not due to the backend. This is Video Games unfortunately).
I’m not like, “hard” on these opinions, but people talk about kubernetes as if it’s the only way of solving these problems, when it can be the case that you have to learn a lot about kubernetes before you can actually do what you need to do; if you already understand distributed systems then it’s a bigger effort to understand kubernetes than would be to implement a distributed system that solves your needs from scratch
Have you considered something like Tailscale so you can securely access it from outside your home? I've been thinking about spinning up my own home server that way, seeing as Tailscale makes it easy to securely access it from my phone when I'm out and about.
Exposing Home Assistant over it allows me to do things like turn my lights off when I'm away from home, without exposing stuff to the public internet.
https://www.blog.akhil.cc/shelly-dimmer
Before I clicked I thought “man, a single config.nix for his whole system? this is going to be big…” But I’ve continually been surprised how little configuration you actually need to do in nixos. I’ve been planning on configuring Home assistant on an RPi myself, so this will help. Thanks for sharing!
I was just looking into doing something similar after realizing how little my RPi3B+ running only HomeAssistant was actually doing. I'd love to have PiHole and Shelly among others so maybe this route on a RPi4B is the way to go. Thanks for this!
It's not quite the same as a VPN, but in addition to Tailscale and some of the alternatives suggested in other comments, it may be worth looking at using HTTPS client certificates.
Done right, you get a pretty simple second line of defence - you can validate the client has an appropriate certificate (running your own CA is pretty straightforward for small-scale home use or for a small group of users). Without such a certificate, users can't access the web service.
If your goal around a VPN is a second line of defence against the application's own authentication logic failing, client certificate authentication might be worth a look. If your threat model needs to cover a major issue in your web server, you might still want to stick with a VPN-based setup.
(You can of course do both, and bind services to an internal-only IP that you can only reach via a VPN, then have certificate auth on that too if you so desire)
The headache I find with client side certs is that it's annoyingly common for client side apps to have their own CA store which you have to track down and install your CA.crt in.
I just stumbled on a self hosted project similar to tailscale: https://github.com/gravitl/netmaker. There is also https://github.com/juanfont/headscale which is an open sourced implementation of the Tailscale coordination server. Tailscale "just works" and makes things incredibly easy for personal use, but it's awesome that there are also similar self-hosted options out there
(Repo owner here) I'm planning to build my own router with OpenWRT and install Wireguard on it, however due to the chip shortage I can't get the hardware for a reasonable price yet. Netmaker looks amazing, I need to play with it.
ZeroTier is certainly another viable alternative, but their self hosted option still relies on the ZeroTier root servers (you can self-host roots, but they don't support removing theirs [1]), and their licensing for the self hosted product is more restrictive than something like netmaker.
I'm using Tailscale on my home network. Right now my two most interesting use cases are 1) I have it installed on my Home Assistant VM so I can access it wherever I am without having to futz with internal vs external URLs, and 2) I have a few Raspberry Pi's scattered around the house running zwavejs2mqtt which doesn't have any authentication built in (i.e. anyone on my wifi could theoretically mess with my zwave network). I told it to listen on each Pi's Tailscale address and use ACLs to lock it down so just Home Assistant can talk to the websocket API.
Isn't headscale a clearly better option, since it removes the need to trust and depend on external sources? It the same software essentially, but if we're talking about self hosting, headscale is just inherently better, since it actually is self hosted.
Normally I would prefer a go implementation given the option (especially since this is network focused), but I was surprised to see that the go implementation of wiregaurd (not headscale, just the vanilla wiregaurd-go) is considerably slower than the rust or C version.
The rust version (not that I'm that rust guy) may be the optimum for maintainability/readability and speed.
I found the Tailscale Android app to take quite a big hit on my battery when used as an always-on VPN, so I use cloudflared on a VPS to create a Google authenticated tunnel to home assistant (over Tailscale).
They solve different problems. Tailscale basically allows you to ignore any NAT topologies separating you from your machines. You can have a pseudo local network of your machines behind a VPN, allowing SSH to any machine with zero routing issues. And it’s incredibly seamless. I’m a big fan.
But in this model wouldn't you control the NAT config? You're already running servers, so outside access isn't an issue. I dunno, OP is running that "Cloudflare tunnel" thing for exposed ports so maybe there's something I don't know, or that the internet connection is still some consumer thing that prevents or forbids running servers.
Actually, now that I look closer they aren't running their own email, so maybe this really is primarily a mostly-internal-access project. I do like those SFFs tho!
The difference vs SSH for me (using plain Wireguard but achieving the same thing) is that I can set up my phone with always-on VPN to home and then wherever I am I'm effectively on my home network. I can access all my home-hosted services in my browser, securely access IOT things that can't talk to the open internet, use my home-hosted DNS rather than my mobile provider's, etc. I do use ssh over that connection (with Termux) to talk to servers, ofc.
NAT busting is a plus. Avoid needing DDNS and opening a port. Also easier setup for accessing other stuff on the network via that computer vs up/down rules with a standard wg config.
For a small setup thats the big thing, but for anything a little more it does key rotation, handles ips for you, and offers "magic dns" which makes all the devices get a nice DNS address when tailscale is on.
NAT busting is a minus. If apps can create holes in your network and “bust your NAT,” your network is not well secured; you should disable UPnP and filter egress.
Also, you gain simplicity but give up to coordination servers, that can be a privacy and security problem.
I enjoy reading about homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like. This is just another "how I installed kubernetes" thing only without any real scale behind it. There are other ways to deploy and run things, and it takes people toying around with alternatives for something unique to spring up.
My "homelab" is just debian stable, with a moratorium against containerization. Systemd keeps the services going. Updates are manual. It's a bit incredible how clean and easy such a set-up can be. Like there's almost nothing to say. It's extremely reliable and just keeps ticking.
I actually did use to have a bunch of virtualization and kubernetes and crap, but got rid of it because it ate literally half the systems' resources, in a death by a thousand containers-type of way. There was also a lot of just jank, stuff was constantly breaking and I was always trying to piece together why from half a dozen out-of-date tutorials. Felt like herding cats.
I've transitioned from a single server running services on baremetal Debian, to running services with Docker on Debian, to running services with Docker on Debian VMs under Proxmox (aka Debian with a repo), to running services on a three-node Kubernetes cluster using k3os (I was using Talos, but it doesn't have Longhorn support yet, and Rook/Ceph was a nightmare).
The baremetal server was the easiest to set up, obviously, but the hardest to maintain. Where did I put that random config? Wait, why did this break when I upgraded X? Switching to Docker (and later Compose) wasn't that difficult, and made maintaining services much easier. Going to k8s has been challenging, mostly because of how I'm doing it - a controlplane and worker node on each physical node in Proxmox. Writing the YAML is, well YAML.
I'm mostly looking forward to not having to keep my VMs updated. I automated a lot of that (VMs are Packer templates) with Ansible, but it's still annoying. Upgrading k8s nodes is a lot easier, IMO.
Yea I think I missed the whole web-complexity bus and am probably out of touch, but I still don’t get the use case for docker and containers and kubernets and and orchestration and all that stuff, just for a simple home setup. I serve a tiny web site, email, backups, a NAS and a few other internet services for my family, and my “stack” is vanilla Debian Stable.
Maybe I don’t know what I don’t know, but my setup works for me and I don’t really have any problems maintaining it so I figure why add all the complexity?
It always feels weird to see threads and threads of people talking about dozens of software programs I’ve never even heard of, let alone used. Maybe I’m living in the past but to me a “stack” is: OS, server, database, application. Like LAMP. Wonder when this changed!
It makes me curious about what kinds of stuff people do in their home networks that I never even considered doing.
"This is just another "how I installed kubernetes" thing only without any real scale behind it."
It is just a learning project, for me K8S didn't 'click' untill i tried to configure it myself on a few machines.
There is something about doing it om physical real things that aids learning, like you could read all chemistry textbooks in the world but untill you actually try it yourself it's not quite the same
I think it's important to keep you 'precious files/services' and you experiments separate.
> but I still don’t get the use case for docker and containers and kubernets and and orchestration
For docker there's a simple motivating case: some services are difficult to configure securely with minimal permissions, and having a standard docker image provided by people who know what they're doing would be a big net win for security on the internet. There are a lot of poorly configured and insecure http server's out there. Think about how many vulnerable http servers are running on routers.
I would argue that poorly configured docker images as far bigger issue than insecure http servers: https://www.infoq.com/news/2020/12/dockerhub-image-vulnerabi...
Security wasn’t the motivation imho, but being able to package your app with all it’s dependencies, including OS libs and deliver it extremely easy, sure was.
> I would argue that poorly configured docker images as far bigger issue than insecure http servers
I think the number of router vulnerabilities alone refutes this argument. Yes, security wasn't the original motivation, but it could be one good motivation.
It doesn't make sense, but often people who set up K8s at home do it for educational reasons or just having a playground so they are more capable at administering systems like that at work.
Sure, using K8S in your learning envornment because you're managing 800 different services in your dayjob is fine. Personally I have my learning environment as part of my day job though.
At home I just want things to work, hence a nice simple LAMP
I can see benefits of something like ansible if you have a command-and-control environment where you deploy 300 machines from a central database. Still allows you destroy 300 machines in one single misconfiguration or mishandled error though, ouch. If you have a federated environment where you have many actors all managing their own machines though I don't see the point.
The number of times I have to do something to the 160 machines I'm responsible for is about 3 times a year. Clusterssh does the job in 10 minutes, and inevitably shines a light on some unexpected error somewhere. If I was doing it on a weekly basis though, using ansible would be beneficial.
Personally I don't like ansible's way of throwing everything in a file in /tmp, then running it as root. authlog gets sent to my central syslogs servers so I've got a record of what happened. With ansible I'm never quite sure, and I'd have to look in two different places for logs.
Instead we just have a lightweight phone home agent which reports processes/diskspace/mounts/interfaces/hostname/ownership etc so you can identify who owns a device on your network and what it's responsible for without consulting out of date documentation that relies on humans to do things.
Kubernetes I guess is useful if you're building a cluster of 100 machines scaling and doing the same service because you're handling a million users a day. Most companies don't do that, most services don't need that.
Use the right tool for the right job, and don't assume the stuff google use is the right tool (unless you're building a lab because you want experience to get a job working for google. If you tried that shit on my network you'd be starting your job search far earlier)
I want multiple conflicting versions of Elasticsearch on the same host. My OS packages are built to support a single deployment. I could manually setup users, config files, systemd services, etc. for the second deployment, or I could just run them in containers, which is the lower effort option
This is what I'm getting at. That is an enormous overkill for a single server, and exactly the problem with kubernetes: It drives so much complexity that you kind of need this type of heavy-duty enterprise grade solutions even in an area where you really shouldn't need it.
If it's just a single server, you're fine with logrotate and grep.
So first of all: I do have to clarify that I just use bare podman containers deployed by ansible and not a while k8s setup, so I'm defending containers and not k8s. But anyway:
Who said graylog is only logging on that single server? That server is by far the biggest and hence why I run multiple services with big dependencies on it, but I have that server, 2 cloud servers (one running my personal web page and other publically accessible services, one running a couple of game servers) + a couple of Raspberry Pis.
Even if I didn't have all that, a web interface to configure it to email on ssh login or send me a mobile notification that my offsite backups are still happening and/or not happening for devices or similar items would still be a nice feature, even if you personally feel it's too complex or you could write a cron job for each of them.
I've used everything under the sun and for homelab use IMO nothing beats using docker-compose and salt.
1. Unless you have crazy dynamic scaling needs, there's literally no point in using k8s or nomad
2. Unless you want RBAC and fine tuned ACL, theres literally no point in using k8s or Nomad
I hate this idea that you need a Ceph cluster and a 3 node HA cluster to have a home lab.
Not saying don't do it because you want to play/test/learn any of those platforms, just that if you are going to run a server service maybe consider if you are doing it for learning a task orchestration service or just want to run XYZ reliably.
I was settled on Docker-Compose and Ansible for a long time, and while it was indeed quite easy to use once everything was tweaked correctly, what finally got me to move to k8s was HA. Having a single node with nginx, DNS resolution and Unifi controller running on it meant if I ever took the node down down for something, everything broke. Plex being down because I wanted to play with something unrelated wasn't ideal.
Agree that Ceph is ridiculously overkill though, I tried. Longhorn is much easier and perfectly fine for distributed storage.
Unifi controller isn't a dependency for your network to continue working. Config is distributed to the devices themselves, the controller is just a UI and collector of data.
I also run everything with docker stack(s) currently. Not using salt and haven’t heard of it. Why would I want salt and not ansible or something similar?
Salt (https://saltproject.io/) was an almost-winner around the time that Ansible blew up. It's pretty irrelevant nowadays since terraform and kubernetes - but still it's a very very good tool.
I'd suggest if you already know Ansible then don't change. If you have no idea which is which then try them both and let us know :)
The reasons that I used it were because the agent (called “minion”) initiates a connection towards the master, which can be handy if you’re behind NAT- but it was interesting to us because incoming connections are easier to manage at scale than outgoing ones.
Another reason was that the windows support was much more mature (though not perfect) and our environment was mostly Windows servers.
That said, installing the minion agent was easy, much easier than enabling winrm.
If you have sunk significant time into ansible I wouldn’t recommend switching, but it’s definitely not dead as per the sibling comment, I personally found it more enjoyable and easier to work with once understood the DSL pattern and added a few custom modules, it’s very simple underneath.
Personally I don’t use Kubernetes for the scalability, but because I haven’t seen anything else remotely maintainable (and yes, Kubernetes still has lots of room for improvement).
I don’t want to build a good chunk of Kubernetes myself just to host my software. It’s nice to be able to Google for solutions to my problems or to ask on various forums/chat rooms (or in many cases, not have the problems in the first place because someone else committed the fixes to Kubernetes long ago).
I've come from bare metal, to virtualisation, to Cloud. I then went from AutoScaling Groups and AMIs to K8s and Docker Images. Recently I went from K8s to Nomad+Consul and Docker Images.
Now I'm building a small platform to host a business I've started and I'm going back to ASGs with simple EC2 Instances.
There's really no need for the features Nomad/K8s offer at almost any scale outside of a few big companies.
It depends. Orchestrators like Nomad can give you a lot of leverage as a small team. It's not very difficult to manage and can give you lots of redundancy, control, and a path to scalability while utilizing best practices so you don't have to think about it. There are lots of ways to do this and not every app needs this but as you begin to get traction, you either have a choice of outsourcing your devops or bringing it in-house which happens way before you are even a big company. Nomad solves the problem of the latter.
Also, I'm curious why you didn't just end up sticking with Nomad now that you know it.
Orchestration engines can give you a lot of leverage if you run a managed solution because all you're doing there is deciding on the size you need and providing a credit card number. This is absolutely a great way of your DevOps to the next level with little effort.
If you're going DIY, however, then you're not doing yourself any favours by starting out with an orchestration engine.
In the case of Nomad you'll also need Consul for it to be truly useful. That means six EC2 Instances (for example) for a truly production trade system - 3x for Nomad, 3x for Consul. Of course you also need mTLS as well - may as well throw Vault into the mix too, then... that's another three EC2 Instances to support that cluster... 9x EC2 Instances into the DIY solution and all you have is an orchestration engine and you've not even started on the workers yet. That cost so far is USD$30,416.04 for the reference architecture for all nine nodes: https://learn.hashicorp.com/tutorials/consul/reference-archi...
On the other hand if you keep it simple and just work with AMIs you bake your self when you need to release something new, you need two EC2 Instances. The cost of the compute resources for this for 12 months: UDSD$330.
There's no raft consensus which needs three nodes to be stable (but a single AZ failure can result in split brain, so really you need five AZs), no new technologies to understand, no new vendor to work with or pay capital to, nothing more to support.
These things - these orchestration engines - have their place, and that place is outside of 99.9% of businesses (large or small.)
As for moving away from Nomad: too expensive and too complex. I ran Consul, Nomad and Vault on the same master instances (which is a risk and against best practice) without mTLS (just TLS) and it took a week to get it all running. None of that includes monitoring, backups, auto-scaling, etc.
I believe the work involved with going DIY on the orchestration front is not worth the benefits in the short to medium term. In the long term you'll likely outgrow your own simpler solution and at which point you'll likely have the capital to switch to a managed service anyway /shrug
There's no need to go truly production grade initially. I run some projects in production on single node deployments (with Nomad, Vault, and Consul all on one node, i.e. bootstrap_expect = 1) and it's solid. The hardware specs required are no different than if you were to just use systemd to run everything.
The beauty with HashiStack is you don't need to go fully monty. There's also no need to use ACLs which I agree adds a lot of unnecessary complexity if you're only at the MVP phase or doing things solo/in a small team. And you can run everything on a Digital Ocean $10/mo droplet -- it's very lightweight. I even have one HashiStack deployed on a Raspberry Pi running Home Assistant and other ancillary wares and it works great. But there's always the path to building a proper cluster if you need the redundancy.
I'm very bullish on HashiCorp. They can do a much better job though of communicating that not every deployment needs to be enterprise-grade and offer better ways of bootstrapping the type of cluster that fits your needs. I think a lot of that messaging gets lost due to their ongoing war with Kubernetes and what large companies are looking for.
I actually totally agree with your approach, but not the technology stack choice.
I think if you're at the MVP/solo-team stage then a single server that you push the app too with Ansible/Puppet Bolt, run via systemd, and then monitor, is even simpler, easier and "cheaper" (in time, which is money) than having to learn and use the HashiStack.
My app compiles into a single binary thanks to Go's binary packing feature. Does deploying a single binary reeeeaaallly need the entire HashiStack? Seriously? Do I need TWO operating systems (the host's and the container's) to run a binary?
Honestly I just think people over think this stuff and are easily swayed by cool features.
The path to redundancy is there if you need it but it's not something you have to go with from the start. You can also run a database with replicas to handle any drive failures, etc. I have been running a production-grade system with thousands of customers like this for a few years.
> homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like.
I've had great success with Ansible+Proxmox. The proxmox CLI interface is good enough that you can basically do DIY terraform like setup straight against CLI without any agents / plugins etc. (I know declarative vs imperative but hopefully the comparison makes sense)
Lately I've been toying with local gitlab-CI building and deploying images directly into the environment so that any changes to the code redeploy it (to prod...this is home after all haha)
Considered terraform too but not a big fan of their mystery 3rd party plugin provider system
Yeah I’m with you. I run a rack at home with servers running a few applications orchestrated with a bit of Puppet, it’s 100x more reliable than K8s, I’m not having to do weird hacks to make persistent storage work, edit dozens of YAML files and run a whole host of dodgy control plane services.
Sadly, I leaned this the hard way by setting up a 4-node k8s cluster and realizing bare-metal support is abysmal and the maintainers simply don’t care about you if you’re not running a cluster on AWS or GCP. Unfortunately my day job still involves k8s and people give me strange looks when I suggest metal so substantially less complex, cheaper, more performant, and easier to orchestrate and maintain.
I'm building my own education platform around getting people into IT and towards a career in Cloud and DevOps.
A few months back I built a Nomad based cluster with Consul and was like: wow! This is super cool... but it's also expensive and complex.
I'm going to be building out the platform with an ALB, two EC2 Instances in an AutoScaling Group based on an AMI I bake with Ansible and Packer. I'll use Terraform for the entire infrastructure.
The release pipeline for content updates (it's an MkDocs/Material based book with videos blended in) will be me updating the AMI and firing off a `terraform apply`.
I won't be using Docker. I won't require or need an orchestration engine.
It's a static website delivered via a custom web server written in Go (as I need custom "Have you paid to view this content?" logic.) That's about as complex as it gets for now.
I'm actually looking forward to implementing it and watching as something so simple hopefully goes on to allow me to build and scale a business.
Kubernetes is just young. The story for bare metal is improving all the time. You just have to pick the right components for things like storage and load balancing, which is tedious but no more so than picking the right combination of tools to use to build your own Kubernetes replacement. In time even this pain will go away once someone builds a decent bare metal distro with all of this stuff available out of the box (k3s is trending in that direction).
I'm currently building out my single-node "homelab" with coreos + podman + systemd. Most of the config is pulled from 1 ignition file during coreos installation. Its been a learning experience and but also very lightweight.
My setup I just have ansible deploy to a couple of hosts running Arch. Generally I just use OS packages and have ansible deploy appropriate configs then for some software where that's more hassle I use podman rootless containers. Such software includes e.g. Graylog, which wants particular versions of Elasticsearch and mongo, or postgres which requires a manual data migration for annual version upgrades
k8s and nomad are on my "maybe eventually later" list though.
This is a very good example of how you can save yourself the mental hurdle of remembering how to configure something etc.
I can only suspect how much time, trial and error this must've taken. This is my main issue with IaC. The concept really lends itself to any kind of modern infra, however I'm really put off by the sheer amount of time it takes me to whip out a bunch of Ansible playbooks and helper scripts, and on top of that make sure that all of them are idempotent.
Maybe I'm doing something wrong and this should be easy?
Pretty sure you're not doing anything wrong - well, if you are then I am too :-)
What's often overlooked (I believe) is that when you're doing this work in your day job you've for existing infra to support your infra along with other folks you can rely on to for help.
With home infra, you first must invent the universe (apologies to Carl). Having built 3 variations of home infra (Swarm on x64/Beelink, K3s on Pi, K3s on x64/Odroid) I've gained a strong admiration for anyone who take this on irregardless of there relative success.
What I've learnt over time is to add as little accidental complexity as is possible which I think is what you're getting at. One incarnation of the Pi K3s was provisioned by Ansible (on it's own stand alone Pi that the cluster would netboot from). Was Ansible better that imaging the usb drives manually and running a small script on each of the nodes? - probably a wash. I did however learn a bit of Ansible.
Today's stack is way too complex, and fragile. The entire stack relies on the goodwill of other projects to continue to be maintained/not have breaking changes. Even with the initial setup cost s(h)aved there will be a continuous maintenance burden. What you get in abstraction and automation you pay back in babysitting the additional connecting interfaces.
> The entire stack relies on the goodwill of other projects to continue to be maintained/not have breaking changes
I agree with the sentiment of your comment, but when was this ever not the case, other than the days when you built your own OS and tooling from scratch?
No. It never gets a habit. It is torture, and you have to look up the commands every time you do it, because you do it infrequently enough not to learn by heart.
Commands you use every day in the CLI? In your editor?
I can trivially enter Emacs and modify Apache config when it breaks backwards compat between distribution upgrades, but when it's infrastructure as code...
I can’t think of a single time I’ve had an Apache config break with upgrades - other than having to recompile the webservice proxy module to change the case of the Upgrade header for a device that doesn’t work with the standards and is case sensitive
(Major upgrades every couple of years break the binary compatibility)
Projects are making this easier. For example moving from large mono config files to using conf.d/ directories where you can drop in extra files and manage each one independently is great for IaC.
While it's challenging up front I do enjoy being able to freshly install the latest version of Fedora, run my playbook and be more or less up and running.
It feels cleaner and more reliable (at least until this week when a broken selinux policy made it to stable) rather then trying to upgrade packages across major release versions in place.
<hat material="tinfoil"> If I've somehow acquired secret hidden malware or someone has opened up something to come back in later that's also going to get flushed out at least once every six months.
Awesome! hajimari is neat, haven't seen that before. I'm building a startup that's trying to put a good chunk of this in a box for folks to buy and plug-n-play self-hosting (as much as that is possible). Since you're running k3s already, you might checkout our templates at https://kubesail.com/templates for more home-hosting apps :)
Ack, will look at that bug! No, the templates are currently plain YAML with a handle-bars style variable replacement (and a couple other neat tricks). We do support Helm charts tho, I just haven't had time to add them to the "Templates" view. There are loads of good Helm charts here: https://kubesail.com/helm/k8s-at-home
This is downvoted but I think it is true, self hosted services don't get nearly as much love as they need. It takes a huge amount of effort, monitoring, and knowledge on the specific tools to be able to host them securely and detect intrusions. I attempted self hosting for years but decided it ultimately wasn't worth it and that I have very little ability to prevent against attacks or detect that they had happened.
Maybe they want to practice. I installed proxmox and I'm pretty much set up, but my needs are pretty huble, just a couple of VMs and maybe containers in the near future.
Pretty sweet. I have been called a devops person by others around me (and I am hesitant to proudly identify as one) because this homelab is more impressive and modular than a lot of stuff I have seen colleagues and I put together for professional stuff.
Well done. I was not aware of the Cloudflare solution. Is this something someone can use, _with_ their Cloudflare Access offering, for personal dev/lab envs without breaking the bank?
Why not use something like ZeroTier? I’ve seen a lot of people talking about Tailscale; I’ve been using ZeroTier to connect to my local NextCloud and it works great.
I've been thinking about it but I wanted to set it up on a BSD VPN appliance, and I was not sure that was easy for pfSense in an easy installable way, then expose some NUC systems.
Are you just running it on a system behind a firewall/router/NAT-ed network device or on a terminating device itself?
But, as usual, far behind on my personal projects ...
Nope. It runs on every client that I want to remotely access.
This includes a Proxmox host, a few LxC environments, home PC, work laptop and on my phone.
I did have some issues getting it running on WSL2 but as long as I can get to the Windows host running it (and I can), that's fine for me.
I didn't need to change anything on my network equipment which is why I was so amazed at the ease of it's use. I do have a static IPv4 at home but from what I understood that had absolutely no bearing on my ease of installation.
No port forwarding, no changes to my Meraki MX gateway, no nothing. It was essentially plug and play.
Edit: looks like BSD is supported too. Though I don't have a BSD environment to test it on.
Fwiw you probably don't need to pay for tailscale for that. You can just have them set up their own single user tailscale accounts and then share your Emby node with their account.
Cloudflare Tunnels (formerly Argo Tunnels) are a free complement to Cloudflare Access. It's an easy way to expose internal servers to the public internet, which you can then lock down with Cloudflare Access.
Very cool, this is like a perfect little kubernetes development environment. The combo of tekton + gitea + k8s is really nice and just a shame it takes bodging all the lego pieces together manually right now. I wish there were a one click exe install like Docker for Desktop that gave people an entire environment like this on their desktop. Like a little private github but so much more powerful.
If you're looking for some nice stuff to develop on an environment like this, check out VS Code and Google's cloud code addon: https://marketplace.visualstudio.com/items?itemName=GoogleCl... It's not totally tied to their GCE cloud offerings and is actually just a VS Code integration of their Skaffold tool (a fantastic inner dev loop tool for k8s). It works great with minikube or any other k8s cluster, presumably this one too. You can very quickly get up and running with code in your IDE now running and debugging on your k8s environment.
Although it's still incomplete and not one click yet, that's the direction I'm heading in: anyone can try my homelab on their PC, and if they want to use it on their real hardware, they can continue with the full install.
For people who self host with like setups or even simpler - how you guarantee network access? where I live internet might be shaky at times, and for that I can't access my home setup while away. How do you deal with that?
Regarding my server connection: I can't help you there, my ISP has only had one incident of downtime I've noticed in the seven years of being with them.
Regarding my client connection while away from home: If my client doesn't have a connection, cloud services don't work any better. But a lot of apps (e.g. bitwarden, some jellyfin clients, etc) are smart enough to work offline to some extent via operating on cashed resources.
I recently wanted to give my current RPi4 home server a GitOps makeover, migrating services (Home Assistant, Nextcloud and others) to k3s in the process.
What has been an obstacle is the availability of officially maintained Docker images for some of the components I've been wanting to use - afaict neither Argo CD nor Rook have official armv7/aarch64 images (though it seems Argo will release one soon).
Until then, I'll hold off on that pet project until I get my hands on a reasonably priced x86 SFF PC (the ThinkCentre M700 Tiny from TFA looks interesting!).
FWIW, FluxCD has full armv7 and aarch64 support (it definitely works on a Raspberry Pi 4, the earlier models with 1GB of RAM too, although depending on the size of your git repos that may not be enough memory, 4GB-8GB certainly is for Flux.)
How many Rpi4 servers are we talking? I have never run Rook before (though I ran Ceph, when it was called deis-store way back before it was cool) and I always remember that you needed at least 3-5 servers with at least one dedicated disk for OSD per each to make it work well, if at all!
I'm looking at synology-csi right now for my homelab, and wondering if there is hope that I can stop using local-path-storage this way, it looks a little bit outdated, so I'm afraid that may keep me from running the latest version of Kubernetes on it... unless I hack the support into it myself :gasp: surely that's gonna void the warranty
I always found the hurdle with self-hosting to be maintaining, not the initial setup. Things like upgrading, in order to keep getting security fixes, and verifying everything works after the upgrade, are what has taken the most effort and time in the past for me.
This looks like a great setup by the author, but difficult to maintain in the long run without significant time investment.
Docker and docker-compose make maintaining stuff dead simple to me. In like 95% of the cases it's just a matter of changing the version in a YAML file and running one command, and in the remaining 5% of the time I'm not concerned at all with some downtime because I'm (mostly) the only user.
Docker-compose doesn't need "served" per-se - it lets you run some containers, and handles some lightweight orchestration around them, so that you have groups of containers for a service (i.e. a Wordpress service compromises a mariadb container and a Wordpress container). The compose file handles port bindings/mappings.
If you then want to put a reverse proxy in front of the docker containers (you almost invariably will), then you can look at different options like caddy, Traefik, nginx etc. I, for one, like to be old-fashioned and have my docker containers' ports bound to localhost, then manually maintain my own "outside of docker" instance of nginx as a reverse proxy that uses these as upstreams. That's not the most "container-first" way of working, but it worked for me. Caddy can do similar. Traefik is more integrated with docker and the docker ecosystem, but that might do what you need better.
True, it's not easy. In the past I used to run everything on Debian stable for that reason. And after every two years or so, I'd book two days for the upgrade, with some time offline.
The problem was, at some point Debian stable got so distant from modern infrastructure you had to patch it from the beginning with newer versions, so reluctantly I switched to testing (that was years ago). I was surprised to find out things were working just fine.
The problem is today everybody distributes software in their own way. Especially web software - native packages are more and more rare. So automatic updates that work are indeed a challenge.
The question I have whenever I see something like this is what the upgrade process looks like. How much time do you spend per month upgrading dependencies, how many mailing lists you're subscribed to, etc. When the log4j thing blew up, how long did it take you to gain confidence that you were no longer impacted?
Doesn't really matter, as long as it's fun. Some people play games, so people enjoy re-configuring their setup (I do enjoy it too). It's not a business / prod env setting.
I was thinking more about the user-facing apps you use. Scanning https://github.com/khuedoan/homelab, Gitea is Go, Grafana is Javascript (approximately), Element is Javascript, Dendrite is Go, Vault is Go. K8s itself is Go. Are you planning to run your own email server?
Now I find myself wondering what app choices exist if one were to try to constrain oneself entirely to operating in the Go eco-system, with Javascript only for web clients. That would be quite parsimonious.
Probably no worse than the alternatives. Just about everything is going to be easy to upgrade except possibly the Kubernetes masters (and even that isn’t so bad if you can spare a bit of downtime).
For sure no worse than the alternatives. I also don't care about downtime. But I'm looking for something that minimizes the overheads over not self-hosting.
I'd love to prove Moxie Marlinspike wrong that "people don't want to run their own servers, and never will." (https://moxie.org/2022/01/07/web3-first-impressions.html) This is the key bottleneck in getting people to run their own servers.
For sure. There’s not much of an economical argument for self-hosting—if you’re doing it, it’s almost implied that you’re doing it for fun. Although having run a small (but really scaleable c/o k8s) homelab for <$2/month, I’m not sure I’d save much effort versus using some PaaS or cloud provider now that I know what I’m doing. Like I’d need a real load balancer and a few other things so we’re talking < ~$100/month to productionize.
I still believe :) I'm looking not for an economic argument but for a strategic one. I think[1] a self-hosted setup with minimal dependencies can be more resilient than a conventional one, whether with a vendor or self-hosted.
https://sandstorm.io got a lot right. I wish they'd paid more attention to upgrade burdens.
This needs a companion guide about how to set up the host machine. Which Linux distro to choose, how to set it up / harden it, nftables / firewall, public key login, etc.
Several people in this thread stating they are working on solutions to assist others learning modern cloud/devops stacks. I’ll throw my name in the hat too. I have a single node home lab running enterprise cloud/devops software stack. I have been working on a project to deploy on-prem cloud (OpenStack) in disconnected environment. I’ve had to learn how to build and host all of the infrastructure required to do this without the luxury of internet access. I realized the same thing can be simulated very cheap in a modern computer with enough cores and memory to run the stack as a single bare metal host. My build cost $1300 2 years ago but the knowledge gained has paid me back 5 figures worth of raises in my career in 16 months.
Having my own personal CloudBox has allowed me to experiment and fail fast. I am ahead in experience and knowledge than the rest of my team as a result. I have a tool they do not. I realize it would be better if we all had a tool like this.
So that’s the pitch, a single node “cloud in a box’, the CloudBox for IT students or professionals to learn any aspect of IT.
Now I just need lots more time to actually turn my prototype into a product.
You are unlikely to find what you're looking for. I'm going to dig into this article a bit, but what you're talking about is inherently hard, and delivers tremendous value to businesses. There are a lot of people doing it for money, and someone builds something that starts to actually work, they get acquired, or they take their project and make it enterprise-y (because that's where the funding is).
There's flatcar and k3os and fedora coreos and talos and lokomotive. There are maybe a dozen others as well, but those are the ones I know something about.
The real problem is that the orchestration of PXE boot, DHCP, name services, server registration, cluster bootstrapping, while simultaneously building a light distribution that makes the right calls on persistence, security, etc. is just *really hard*.
I took at a stab at it myself (failed startup) and have a custom distribution based on alpine, but the amount of work to go from there to everything is so large that it's tough to take on if you're small (and there is the constant desire to go enterprise because of the money)
Thanks, some of this went over my head and sounds way too complicated.
I'd be satisfied with a home-user-oriented manual tutorial, like install Debian with these packages, a nftables setup that firewalls everything but these 3 ports, how to setup auto-update, turn off root & password-only logins, and general things to be reasonably secure; as well as tips for on-going maintenance and so on.
(Repo owner here) I've already automated that with Ansible, including Linux installation (Rocky Linux). There's no hardening yet but that's on the roadmap.
I wonder how the fan noise is on those M700s. These little Lenovo machines are very pleasant. My only wish with my M93 is that the temperature would stay below 50’C and keep it 100% silent.
It's sad to see so many people that are dismissive of this setup, because they have some preconceived notions about Kubernetes and other tooling that's used. What has happened to curiosity? Of course nothing about this is "necessary", but the fun is in trial and error. In my experience running Kubernetes in my homelab and colo, every issue has been a learning opportunity, and was usually because of my own mistakes.
What do you use to provision Kubernetes persistent volumes on bare metal? I’m looking at open-ebs (https://openebs.io/).
Also, when you bump the image tag in a git commit for a given helm chart, how does that get deployed? Is it automatic, or do you manually run helm upgrade commands?
Now all we need is robots hooked into a Ci/CD pipeline that turn on computers and initiate the process.
And a few more years to automate computer production, let the computers find bugs & optimize computers, and we have a full circle of life.
This is cool, but are you doing this as learning experience? For a homelab it seems severely overkill. I understand it all being self-hosted, but it could be drastically simpler if you adopt a few SaaS products (using free-tier)
(Repo owner here) Yes, I'm mostly doing this as a learning exercise; there's still a lot of work to be done before I can rely on it to host my services.
> it could be drastically simpler if you adopt a few SaaS products (using free-tier)
For some people (including me), the risk involved in any SaaS product suddenly either dropping or imposing unworkable restrictions on free tier is high enough to make the extra work involved in self-hosting worth it. (Granted, my current self-hosting setup is a lot simpler than the one described in this article, but even if mine were more complex I would still say the same thing.)
I've been burned too mamy times. Once a product realizes that x% of its free tier uses only feature X they suddenly put that feature behind a paywall. Now I"ve got to research and setup an alternative or pay 20$ for some minor convenice I set it up for.
Most free tiers requires a card on file. I shudder to think at having to check dozens or hundreds of such services regularly to make sure they have not sent out an email about how we are now charging you. Then I don't notice for 6 months and find out I spend $150 on something I could have setup myself in less than an hour that will never charge me.
I have had hundreds of such products subscribed at work. At least every year or so an entire day gets burnt updating credit card info from logging in and looking up the bespoke way every single site requires the updates to be done for payments.
The services that suddenly shut their doors and again require me to research and setup an alternative.
Services get acquired all the time then switch to pay only and they have your CC on file already for easy billing.
Bravo, nice work! I am certainly going to use this as a model for my personal setup. Right now I have a few different hosts running different hypervisors, but would like to consolidate on kube and an iaas tool.
I've been eyeing off OpenStack for sometime for a similar use case, but I must admit it is an uninformed opinion. What am I missing between the two solutions (OpenStack vs this specific custom solution)?
Overkill probably, but useful practise for when working on something of a scale where it will make a significant difference. And they might just like playing with these things. A home lab does not have to be DayJob practical!
Talking about overkill, he also use 4 physical machines each with a 4 core CPU. But its not about what you need, its about what you want to learn. So if you want to learn about cluster/orchestrating i think its warranted.
Some people just like to tinker and try other things even if it's overkill. I have a k8s cluster on DO to host shitty side projects, it's pretty nice to just `kubectl apply -f manifest.yml` and be done with it, especially since I'll probably wind up tearing everything I just set up anyway.
I fully agree that setting up Kubernetes is a lot of work, but I think people forget how much work it is to set up anything maintainable for more than a few services running on a handful of services.
The nice thing about Kubernetes is that it’s probably not much harder (if at all) than the alternative (assuming you already know which tools to use to compose your bespoke platform) and you can easily find online resources and support (never mind all of the problems you don’t run into because they’re already solved in Kubernetes).
I would rather choose something like xgp-ng as a hypervisor for a home lab environment such a this, instead of a custom one-off kubernetes environment.
Or just a total DIY xen or KVM hypervisor on debian or centos approach. But with considerably more RAM in the host machine than 16GB (like 64-128GB).
the kubernetes sort of makes sense if the purpose is also for the person who owns/operates it to test kubernetes things, learn kubernetes, or use it as a home lab of some larger similar environment that they have elsewhere.
This is definitely cool, but it also highlights a huge problem we have with software nowadays.
There are 19 stacks in this repository. 19 pieces of software that require their own maintenance, JUST TO RUN YOUR APPLICATIONS! The amount of extra work required just to host the software that views your pictures, plays your videos, and allows chat with people is absolutely insane.
I'm not talking about this particular repo; it's just indicative of how complicated things have become that you must do the equivalent of building and maintaining your own operating system just to get even simple things done. And I belive that it's unnecessarily complicated (once again, not this repo, but rather the state of affairs in general). We're at an inflection point in the industry, and haven't come out the other side yet.
I could post my fully automated self hosting infrastructure, but it would be boring. Few bash ssh scripts and LEPP stack. None of this complexity and yet 10+ years uptime. Interesting nonetheless but think i'll pass on this. I have done these setups at cloudproviders for clients; that is definitely the risk averse version; if something goes wrong every month with AWS, it's just 'ok'; if something goes wrong every 5 years with your non-cloud, it's panic & heads need to roll (even though it costs a fraction).
Yup ! This I've used bash+ssh for the last 10 years for my solo founder and commercial products. Is sexy or even nice ? Absolutely not, but damn its simple and most Linux semi professionals can understand it with a glance.
Bash as infrastructure+maintenance is almost like SQL for data.
Everyone agrees is not fantastic, many have tried to improve and yet SQL + BASH is still around chunking along doing their unsexy things just good enough :)
Never underestimate software that is 'just good enough' :)
It's not just running an app. He's building a setup that supports FANG-level infrastructure with autoscaling, self healing, etc.
Maybe it's unnecessary for tiny companies, but if you're dealing with infrastructure at scale then the complexity is unavoidable. The only question is whether you want to invent it in house or use a robust system that someone else built properly.
> There are 19 stacks in this repository. 19 pieces of software that require their own maintenance, JUST TO RUN YOUR APPLICATIONS!
There's tools like Yunohost [0] that can save a lot of work and are focused on making the subsequent maintenance burden easier too. There's an application catalogue [1] with a lot of popular self-hosted apps to choose from and new ones being packaged all the time. I haven't used it myself yet, but hear very good experiences from people that have. Especially those that lack the more sysadmin tech skills and just wanna click-install & go.
There's also Freedombox [2]. I'd be interested to know about other such tools and your experience with them.
Thank you so much for name-dropping YunoHost! The demo alone sold me on finally going the self-host route for things such as ebooks, music, and photos.
Yes, but after the initial monoliths of mainframes, we had a golden era of operating systems, where all of the plumbing and infrastructure to run your applications was maintained by someone else (Microsoft, Apple, Commodore, Atari, SGI, Sun, Palm, Redhat, etc).
Now we've come full circle back to the bad old days where you need an entire team of dedicated people and arcane knowledge just to run your application software again.
I suspect that this will continue until we reach a point where the big players out of necessity come to an agreement for a kind of "distributed POSIX" (I mean in spirit, not actual POSIX). These are exciting times living on the edge of a paradigm shift, but also frustrating!
That’s actually very insightful. The OS era you describe was mostly local with not so many things happening in online interactions and way less devices collaborating to create for example your images folder.
Now that we mostly have the server-client model ingrained in every online activity there is an obvious complexity centralisation on the server side.
Might be interesting to see how web3 and other decentralised web movements might factor in to this!
From a standard system upgrade? Never happened to me in 20 years. Between OS versions sure, once every few years spend 20 minutes building from scratch
If it did happen, same as if your machine caught fire. restore from your backup
I want to visibly second this (with a comment, not only an upvote). I dubbed the LAMP stack the CHAMP stack for some time now (CH standing for CSS and HTML).
Ummm I'm sorry, but I don't see how this is practically simple. Theoretically, maybe - [and I agree with the idea that there is potentially (hopefully) a paradigm shift in initial play now, with the thrust being multifaceted including companies' lack of upholding user privacy/selling you as a product, slowly increasing service charges, and lack of customer customization.]
I think tis thrust towards self hosting should make the construction of these systems closer towards what you describe, but it isnt really that simple yet.
For instance, if you want to self host even a decent all-in-one calendar/notes/etc system like nextcloud, it isn't just `sudo pacman -S nextcloud` and youre done.
There is an enormous consideration of how to construct the 1) network architecture, 2) encryption of disks, 3) secure/encrypted, incremental, and full filesystem offsite backup, 4) security and encryption of network, 5) secure interplay between self hosted and non self hosted data, etc.
An example of (1) may be that, much if the discussion around self hosting states that one should not open their LAN to the internet with port forwarding (as it's inviting you to be hacked), and you should rather point your registered domain to a VPN such that you must login with any remote devices to access your server (just to point out - likely here you would need to use a dynamic dns service to get access to your system, which is also another hassle for newcomers).
Further on (1), this involves ensuring that the choice of VPN is adequate (many older technologies have known security flaws, like oracle padding attacks, etc.) - and finding a combination of routers with correct firmware to allow for VPN servers, (also VPN clients on top of that if you care about not having your isp sell you to everyone) the correct hardware and network architecture that can properly mitigate the encryption of traffic on your network (e.g. can a raspberry pi really handle all of your traffic and maintain 1Gbps file download/uploads with vpn encryption?).
I recognize that much of this is not really possible to `sudo apt-get` on one system with recommended hardware, but the more FOSS out there that can get closer to mitigating all of this, and to have well packaged tutorials on how to safely and properly self host (e.g. complete google service replacement) while maintaining security and 10Gbps speeds, the better off we will be.
This really is what we should be driving towards. Companies can certainly offer their servers for customer use, but I think it should be expected that companies have the code that is used to store that customer data fully as FOSS, for auditing and to allow for self hosting.
There's still money to be made for companies in helping people self host everything for themselves - we just have to push for it.
One decent enabler of this is companies that make software for data backups. This is another one of the enumerations above, because it is quite difficult to find a solution that has quantum secure encryption, excellent compression, incremental (only pushing diff like git) capabilities, and is reasonably simple to use.
AFAIK, Duplicati is the only option that seems to be able to hit all of these well (hopefully someone can correct me).
Anyway, this was a meandering long way to say - self hosting is complicated, but i do really hope we can change that.
I’ve been self hosting for over 20 years on apt based systems, aside from changing from telnet/ftp to ssh, adding in letsencrypt, very little has changed. You can still collocate your hardware, but it’s cheaper to use a generic VPS. You can host from home, but even a pi will probably cost more in electricity use than a $3/month vps.
Backups are far easier now as you use a VM provider like linode, digital ocean, even lightsail, which will schedule snapshots. If you run your own hardware then mysqldump and restore to your backup server is a small shellscript and cronjob. Disk encryption is handled by your OS.
LAMP was trivial in 2002, it still is. You can use nginx or postgres instead of apache and mysql, but it’s broadly the same.
If you want to make your site complex and dependent on thousands of JavaScript libraries and frameworks which change every year or two, that’s fine, but you don’t need to, it’s a choice, one which adds complexity. If you want five nines or absolute guarentee of not failing, you need to think more about replication than just a nightly snapshot, but that’s not a problem solved with thinks like kubernetes.
If you want to scale to millions of concurrent users pulling terabytes, sure, don’t self host from your DSL on a pi. If you want to serve a personal site for hosting bits of stuff, it’s not hard.
I'm interested in this, as it somewhat fits my experience on local stuff. (I haven't spent on a Vps yet, as I haven't figured out how to do so in a completely secure, private, and perhaps maybe even somewhat anonymous fashion).
I have been able to set up a simple system hosting a nas with NFS locally very easily, but ensuring secure connection lan remotely has been a very big hassle, and migrating to things like zfs have also been frustrating. The use of containerization for everything is just absolutely absurd these days, but it seems like the solution everyone is using. I wish there were more of a community for people self hosting who make all encompassing solutions without containerization and complicated JavaScript apps.
Most of that is kubernetes, which is completely unnecessary for self hosting purposes. It's mostly for learning, or for the convenience of hosting multiple different services, trying many different containerized applications.
If your only goal is to serve Nextcloud, Plex or whatever to your family, you can get away with much less than that.
I agree that the number of different technologies here is shocking, but you get a lot more value as a result than "just to run your applications". Most notably that includes CI/CD, scaling, and redundancy, all with IaC. These things used to require a lot more administration and development effort.
Plugging together all these different tools has become so much work though that in many organizations the platform team(s) who are mainly occupied with doing just that take up a lot of engineering resources.
In my opinion the next evolutionary step would be for all of this to be bundled and abstracted away. Funnily enough, we pretty much have that product already with autoscaled "serverless" cloud services, GitHub Actions/Azure DevOps pipelines etc.
The biggest problem is probably that things like AWS Lambda, SQS, and DDB lack versatility and user friendliness. If we get some improvements on that front, many organizations might opt for that instead of dealing with their own K8s deployments.
Even better would be if we had something like a stripped-down version of OpenStack, just focussed on doing "serverless" right, and rebuilt from the ground up.
my main objection to this is choosing host machines that don't meet the criteria of anything like serious server-grade hardware.
if this is for a home lab where any one of the services run on it are not actually going to affect you if it goes belly up? or the whole host machine? sure, okay, but that's self hosting a home lab, not self-hosting actual infrastructure...
clearly the hardware shown in the image is meant to be small, not noisy, and not take up a lot of space, and operate in somebody's house.
but the people I know who seriously self host all their stuff have it on something like a few old Dell R620 1U servers with 1+1 dual hot swap power supplies, RAID-1 for the operating system boot, RAID-1 or RAID-6 (or something like mdadm raid6) for storage drives (again all hot swap capable), etc.
note that I am not talking about a lot of money here, you can buy the hardware I described in the preceding paragraph for $300 on eBay and then add your own choice of SSDs.
and not in a house, but in something like a small to medium sized ISP's colocation environment, with UPS, generator backed power, etc. also attached to network equipment and a DIA circuit that's considerably more serious than a tp-link unmanaged switch attached to a residential internet service.
All overkill. If you want something more reliable, get some late model HP or Dell thin clients and put Debian on them. Usually you can pick them up for <$75.
I setup something like this to control some farm equipment when I was in college 20 years ago. The farmers’ son called me a few months ago because they were having an issue with the equipment. Turned out oily dust coated the thing and got inside - I had him pop the device open (old Wyse box running Slackware iirc) clean up the goop and hit it with rubbing alcohol. He dried it and everything is working again.
I was surprised to find this on Hacker News, I wanted to wait until the stable release before posting on HN, but thank you for posting :)
This project is still in alpha stage, but please feel free to critique; I'd appreciate it.
Edit 1: After reading some of the comments, I want to clarify a few things:
- Because it is currently in the alpha stage, I do not host anything important on it.
- This is also my learning environment, I use Kubernetes in my day job.
- Yes it is overkill ;)
Edit 2: Wording