I tried doing a homelab with six Nvidia Jetson Nanos using K8S, maintained it for about a year, and I have no desire to ever do that again. I ended up just buying a single rack mount server and using that for two years, and now I bought a mini gaming computer which I use as a single server. Maintaining the k8s cluster was becoming a second job that actually costs me money, that I enjoyed less and less every day, and it made me dread actually using any aspect of my server, meaning that when something broke it would take me a long time to actually muster up the strength to fix anything. My home server runs a Transmission server, Jellyfin, Apache Kafka, Apache Spark, Cassandra, and RabbitMQ, with 32 gigs of RAM, and it works fine.
Distributed systems are cool and they're fun to play with, but the combinatorial explosion of maintenance shouldn't be underestimated. If you're making something that needs to serve 10,000+ users, then it's probably worth it, but homelabs generally aren't that. Generally a homelab situation has like a Plex/Emby/Jellyfin server, a torrent server, a reverse proxy, maybe some kind of message queuing solution, and it generally only has like four concurrent users.
Obviously if your goal is to learn K8s, then doing it with a bunch of Raspberry Pis isn't a bad idea at all, and try to have fun doing it. However, I would warn anyone thinking that they're going to make their NAS a k8s cluster, you're likely going to regret it. I recommend buying a slightly beefier computer and just installing NixOS or something.
I do dread that exact scenario you're mentioning. I know it is a possibility but I'm hoping that should this day arrive I'll be able to put it to rest and hopefully have learned stuff along the way.
You should absolutely give it a try, and maybe you'll have better luck than me, and it's a fun little experiment if nothing else.
If you get frustrated and don't want to do it anymore, you might also look at Docker Swarm. For homelab stuff, I found it considerably easier to work with. I'm not sure how well it would work with hundreds of services, but for a homelab I found it considerably less frustrating to deal with than K8s. It can still be useful to play with distributed systems and it's a bit more intro-friendly in my opinion.
It's something that I think most engineers should try at least once, if nothing else to learn firsthand the annoyances of distributed computing. It's easy to read about these things in textbooks and blog posts and university courses (and you should!), but at least for me these things didn't really sink in until had to fix broken stuff, or deal with weird heartbeat issues, or see things that should go N times faster actually go slower because I didn't take into account latency etc.
I have pretty much landed on a similar simple solution for my homelab. A minipc with dietpi + docker + dockge + bunch of homelab apps using docker compose. Important config backups on a S3 compatible bucket in cloud + photos backup on connected USB HDD.
I tried doing a homelab with six Nvidia Jetson Nanos using K8S, maintained it for about a year, and I have no desire to ever do that again. I ended up just buying a single rack mount server and using that for two years, and now I bought a mini gaming computer which I use as a single server. Maintaining the k8s cluster was becoming a second job that actually costs me money, that I enjoyed less and less every day, and it made me dread actually using any aspect of my server, meaning that when something broke it would take me a long time to actually muster up the strength to fix anything. My home server runs a Transmission server, Jellyfin, Apache Kafka, Apache Spark, Cassandra, and RabbitMQ, with 32 gigs of RAM, and it works fine.
Distributed systems are cool and they're fun to play with, but the combinatorial explosion of maintenance shouldn't be underestimated. If you're making something that needs to serve 10,000+ users, then it's probably worth it, but homelabs generally aren't that. Generally a homelab situation has like a Plex/Emby/Jellyfin server, a torrent server, a reverse proxy, maybe some kind of message queuing solution, and it generally only has like four concurrent users.
Obviously if your goal is to learn K8s, then doing it with a bunch of Raspberry Pis isn't a bad idea at all, and try to have fun doing it. However, I would warn anyone thinking that they're going to make their NAS a k8s cluster, you're likely going to regret it. I recommend buying a slightly beefier computer and just installing NixOS or something.