I used to have an overkill home network, complete with self hosting everything and 10G backbone.
These days i'm more pragmatic. I have exactly 2 wired machines, and everything else runs Wifi.
The network "rack" is simply a firewall and a 16 port POE switch, everything else network related has been retired to the closet.
My NAS is gone. With electricity frequently going above €1/kWh last winter, it seemed like a bad idea to have a NAS sitting there, consuming ~35 kWh/month.
Instead i spent about €20/month on cloud storage, and simply upload everything to the cloud (encrypted with Cryptomator), and i have redundant cloud storage (different provider) included in the cost which is where my backup goes.
All that's left at home is a small ARM based machine that handles making daily backups of my cloud data locally, as well as run a small Plex library on a single USB3 external drive.
Besides saving about half my previous electricity cost (as well as all the hardware cost!), i have gained SO MUCH spare time. My firewall is completely closed (save a wireguard VPN), and i no longer have to maintain anything except the normal patch routines.
I moved house recently and the only things I did were running CAT6 from where the internet enters the house, up to a PoE switch in the loft that then has a few Wifi6 APs connected to that.
Everything is wifi (all cameras, smart switches etc, office laptop, a single RPi4 for "projects" and so on) and all seems fine. The only device that is directly connected via ethernet is a headless gaming desktop that lives in the loft that I occasionally use for streaming games to a laptop.
But kudos for this guy if that is what he wants to do. Personally I did not enjoy doing the cable runs (especially for external-grade CAT6).
My cameras are still POE (UniFi Protect), but i had already run the cables for it, so not much point i replacing them just for the sake of moving to WiFi.
My "server" is also wired, as is my Hue Bridge, Tado Bridge and other "IoT" devices that doesn't support wifi, but when it comes to user accessible machines there is exactly one, which is my "go to" machine in case the router/switch dies and i can't get on the WiFi.
> The only device that is directly connected via ethernet is a headless gaming desktop that lives in the loft that I occasionally use for streaming games to a laptop.
Really curious how you set this up, sounds like it could be the perfect solution for me.
I went the same way as... honestly, my IT burnout kicked in. I went from an overly complex setup down to... well, a Ruckus AP running as an Unleashed gateway directly hooked to a FiOS ONT, its second port feeding a QNAP.
That's it, the entire network. Just enough storage to enjoy some media, the QNAP runs HomeAssistant (literally just to bridge some stuff to HomeKit) and Tailscale, and that's all I really need.
I recommend you checkout Rclone (https://github.com/rclone/rclone) instead of Cryptomater, much more robust and powerful software. I used to use Cryptomator as well, but after my data exceeded certain size, it became slow and difficult to operate. Plus, once I went to pull my data out of it, I realised that it had reset file metadata on all my files (I didn't know what picture was taken when anymore, for example)
Round Sync is a currently active fork. Updates are infrequent, but there was one last week that fixed the (experimental) SAF support and updated rclone, asking other things.
I still have my local NAS and will in fact be upgrading it sometime this or next year. I don't trust cloud computing with my data, even tough I'd rather forgo the complexity and electricity cost.
I intentionally wrote the consumption in kWh instead of just Watts.
Watts is the energy consumed "right now" (which is the product of Volt x Amps), but needs a time dimension to give a measure of historical consumption, which is where the Watt Hour (Wh) comes into play.
While most people are aware that X consumes Y Watts, few consider how much that addes up to over the course of days or weeks.
Few people are aware that a simple ethernet switch will roughly consume 1W per plugged in port, so an 8 port switch might consume 8-16W of power just sitting there, so 4-9 kWh/month.
My old homelab used to consume around 300W when idle, meaning it was consuming 300 * 24 * 30.5 = 219 kWh/month, and with electricity prices reaching €1/kWh, that suddenly becomes A LOT of money to have a Plex server available 24/7.
My Synology NAS was failing. Saw that as a great opportunity to set up a proxmox box on a reasonably powerful used HP SFF, run TrueNAS on it. Plenty of time to get proxmox, TrueNAS, Gitea, a docker host, … just right.
It’s sat there with 4/16TB used, no real data transfer apart from TimeMachine from my Mac and my git server, sucking power at European rates.
Wish I hadn’t bothered, and had just gone with another Synology.
> It’s sat there with 4/16TB used, no real data transfer apart from TimeMachine from my Mac and my git server, sucking power at European rates
I have a similar setup with SFF running backups & ProxMox. However, I have smart plugs for my SFF and have a script to turn it on/off using cron (or on demand) from an always-on Pi. Wake on LAN would have worked too.
I'm considering making my Raspberry Pi the gateway proxy/"Load Balancer" to the SFF; turning it on if needed. It's trivial to proxy HTTP (Pi works as a Caddy-based proxy with long timeouts, powers on SFF in background if off. Things got trickier for proxying NFS & iSCSI)
I think a NAS is fine when you lock it down to sleep when it's not being used. Sure, means you have to wait for it to wake up every time you want to use it, but I mostly use mine (a 4 bay Synology) to back up photos (raws) from a camera so I only need access sparingly.
My cloud backup is rather simple, i use Arq backup with Family365. Fits nicely that each user has 1TB space in their personal OneDrive, and nobody here uses OneDrive.
My local backup is also done with Arq backup to a local USB drive, though i have a long time test running with Kopia backing up to Minio on a Raspberry Pi.
Is there a post that covers the "why?" of all this? As someone researching maybe running some Ethernet for more stable WiFi and faster Internet access for general stuff, maybe some fiber for infotainment other purposes since it can lengthen various cables, and hooking up some security and nature cameras, I can't fathom why all of this is needed outside of it just being a hobby or running some sort of business (trading?) at home. For example, isn't $600 for redundant Internet for like the minutes the Internet is out every year worth it? Why is that level of connectivity needed?
I built a pretty serious "homelab" over a few years, in the sense of building out 24/7 services as robustly as possible. There is an element of practicality for very long-term maintenance if you run i.e. VMs off a different storage layer.
But mostly? It felt exactly like exploring computers in my childhood for the first time (I'm 43 now). I was bumping into things I had never bumped into it before, and it was really satisfying to figure all that stuff out.
I'm a game developer, too, and I realized awhile ago a big part of that is I like technical puzzles, and game development is very fertile soil. But there is a whole new world of enterprise networking/storage/virtualization/etc things that you wouldn't normally bump into in the course of software development.
And as a bonus, I have set up a lot of build systems for friends; being infrastructure-savvy is like the digital equivalent of owning a truck and helping people move all the time.
> There is an element of practicality for very long-term maintenance if you run i.e. VMs off a different storage layer.
That is very good advice. I know about a datacenter outage/degradation in a very large company, which all of you know.
A couple of network switches got overloaded. The switches connected the SAN to the VMs, so it caused all kinds of weird problems in different applications.
Speaking personally, it's just another form of tinkering. Nobody HAS to do it, much like nobody has to buy and maintain their own computer, car, home, etc... We do it because we can and it makes us happy.
Extra note- I would NEVER recommend a business use their home resources for work unless you know what you're doing (and why you're doing it). Even my homelab has vulnerabilities and I'm a security professional- more moving parts = much larger attack surface. There's a reason why corporations pay big buck$ for managed security services. Most homelabs I've seen are mostly for fun and personal comfort.
"What good is knowledge that is never applied" is what drives me to stuff like this. I can't speak for others but I'm sure I'm not the minority here.
Gotcha. Thanks for the reply! To be clear, it wasn't a judgment but just a question to understand there weren't some unapparent reasons why. I think it makes total sense as a hobby and learning, but perhaps still overkill as the post title mentions. Haha.
I apologize if it seemed like I was irate- I'm not and your question is 100% the question we all ask ourselves before doing things like this (or for some of us, it's the first question on the wife acceptance factor audit). If anyone is running a data center in their home for a serious reason they either have small loads to justify the power consumption (rpi k8s cluster says hi), stacks to blow, regulatory pressure, or isn't factoring the costs in and is in for a rude awakening. I don't think (but also don't know) these labbers are the majority, and us homelabbers are already a rarity.
If it makes you feel better, megacorps are getting out of the "self-managed data center" industry and embracing the cloud, to exemplify your very point.
Lol. I was under zero impression that you were irate or anywhere close. My question was a little judgemental perhaps but not necessarily meant in any way. I was also curious if there was some interesting need that had come up. My primary thing that I want is a bunch of PoE powered nature cameras, buy I'm still figuring that out. It will affect whatever comes up though. Oh, and stable WiFi coverage.
I certainly have hobbies that go above any need or reasonable collection, namely synths and books. Haha.
It's interesting that you mention PoE nature cams- I designed a PoE home surveillance system for a friend that involved setting up a solar panel on a 30' pole that fed a box with a shitty camera system in it at ground level. From there he set the cams around his property- particularly where the foxes and coyotes would travel to get to his hens. The whole project was apparently about $800 aside from the solar panel (I just gave him the idea- I didn't help him build it).
He eventually got rid of the cameras because, well... They were shitty and only told him the critters were near AFTER he popped 'em. I think he's got a for-purpose system (in his own words "the new cameras didn't fall off a truck") now but it was a fun project!
I still think your question is reasonable. Even when someone is doing something for joy, there's usually a spark - an essence to it. There's usually an initial motivation that sent them down the rabbit hole, and it's interesting to hear what it was.
There's a reason other than "because it's fun to tinker." I do it because I love the capabilities it provides. I don't love managing Ceph & Proxmox & whatnot, but I love being able to deploy whatever I want into a beefy cluster with 10gb without having to worry about cloudspend. If I wanted to replicate what I run in my home infra (I don't consider it a lab), it'd cost 200-300 in compute and easily another 300 in storage a month. Instead I spent ~2k on hardware and ~35/month in electricity.
It's the same thing with 3d printers, some like them because they want to install clipper and tune the best/fastest benchy. I do it because they love being able to CAD something and have it in my hands as soon as possible.
That said, nobody goes as far as this guy without really enjoying the tinkering
No, but that'd be neat! I don't have GPUs in my cluster though.
My big services are a Ceph cluster, a VPN, Borg (backups) and a K8s cluster (using kubespray).
In K8s I have the main stuff: Plex, Gitlab, Gitlab runner (for CI), vaultwarden (passwords), miniflux (RSS, might be moving to freshrss though), rust desk (remote-desktop), home assistant (smarthome).
My next project is to stream metrics into grafana. I have soil moisture sensors in my garden connected to stm32 boards, I just need to setup the receiving side and I can control my drip lines (they're using opensprinkler) with soil moisture information along with weather info.
> I have soil moisture sensors in my garden connected to stm32 boards, I just need to setup the receiving side and I can control my drip lines (they're using opensprinkler) with soil moisture information along with weather info.
This is a use case close to my interests. I want a network of environmental things monitoring and doing some detection stuff.
I have been working remotely since about 2005-2006. I have always love a good Internet connection with backups (wherever possible). I remember befriending the local cable guy so I can get 1Mbps in 2000.
Now, I have three Internet connections bonded and balanced. It is not about the minutes of disconnection but more about the disconnection when I needed it most. I have ample non-internet time and personal/family downtime. However, when the time comes, I’m happy that the Internet is never a bottleneck in my work and play. I have had this setup since around the early days of the Pandemic and our home, per se, “never had an Internet Outage” since.
I'm not into servers, devops, networking or anything of that sorts, but I love tinkering with them. I would love to have an "overkill home network" one day.
Just wondering:
1. How often do you actually need the third? As in, both primary and secondary are down.
2. Is it that important that you always have an internet connection?
After the 2nd, it was more of a fun and why-not! It is cheap, and comes built-in with a free Netflix subscription and quite a few others. I think it even has an unlimited voice call if I want to plug in a phone and use it.
Why not? I had a pretty elaborate setup in the house I sold during covid (fortunately the purchaser was enthusiastic about it) with a wiring closet in each wing and a fibre spine running between them. Not at all as elaborate an external connection as the author but I had a lot of machines and wanted to keep wireless bandwidth for things that moved around.
One handy trick: I had at least one drop in each room, often one for each wall of the room. Behind the wallplate was a NEMA box with a conduit running straight down to the crawlspace (6' high in my house so hardly "crawl"). That made it easy to pull cable, not only initially but if I found I needed an extra run, which hardly ever happened. Instead of trying to run through a rat's nest of conduit the conduit was a straight shot and then the cables could easily be managed.
Wifi is rock solid, networking is fast, backups good and outages are almost nonexistent and are always due to me messing with things.
I have Netflix, Apple TV+ (is that what it’s called?) and a few others.
Particularly with Netflix, the quality is junk so I watch off Plex.
Other benefits are pretty neat too. POE just works. You can power cycle things remotely if you want (have never needed to) and the abundant local storage makes everything easy.
Enterprise stuff that is getting old is also dirt cheap. Converting sections of the network to 10gbe was very inexpensive.
I mean, “need” is a pretty strong word. No one needs a MacBook Pro, you can do everything on a Raspberry Pi just slower (exaggerating but you get the point). Personally, having reliable internet has been a significant increase in quality of life but admittedly, mesh routers are getting closer and closer. The biggest draw are VLANs. When even the FBI recommends separate networks, it shows how prevalent these issues are. I know a bunch of people that got affected by cryptolocker. While having good practice is probably what helps me a lot, a hardened network helps tremendously.
If, say, you use AT&T and you need Internet for your job (WFH?), redundant Internet is crucial. It's not exactly "minutes" they are out. (Their outages states are often "meh, we'll get around to it in a day or so", figuratively)
If you live in a rural area, even more so. You're not high priority for fixes in the first place. (And AIUI, the author lives somewhere in rural Texas)
Do you need to run it as high tech as that? Probably not. My alternate Internet is my cell, tethered. (And if that breaks e.g. during travel, yes, I have a second cell with a separate provider, because heaven forbid the US had consistent cell coverage)
And most of us probably don't give a damn about e.g. 1,000 year photo retention. Or many of the other things he's doing. But it sure is fun if it appeals to your personality, and you can afford to keep it running.
Most of the time it's the IT Infrastructure, Operations, Security guys that love to do this for (WHY) learning, testing and as a hobby. Check out r/homelab and r/selfhosted for more on this.
Although not as elaborate as this, I have what one would call a "homelab" and it's for, well, testing and experimenting -- I write a lot of high-performance server/network-centric software (e.g, saturating a 100G link with 64B frames is a common test) and virtual machines just aren't suitable/capable most of the time.
It’s a good question, as long as the why is not a burden to be met.
First it’s a useful skill set that helps troubleshoot problems in code and apps that run on networks by knowing how such things work.
Second, try to think of it as a private cloud instead of a network. Because it uses a lot of the same types of software cloud provider do. Proxmox is one example that is a self hosted vps provider that is tremendous. So, it’s a private hybrid cloud that can push from your cloud to the other voids (or back). Build using a private hybrid cloud and you can push to,or between manu clouds
Wifi is for convenience, wired is for reliability. When wifi gets jammed, spotty, interfered with which can happen more often than imagined, especially during break ins, wired wins. Transferring files? Wired wins. 4K streaming ther doesn’t cut out when multiple devices are doing it at once? Wired wins.
Self-hosting is much easier than it was 5,10,15 years ago. Tools like proxmox loaded with yunohost running on a 1L usff/mini/tiny pc with 64 gb of ram and mirrored ssds ther maybe mirror to another i de oval box can sip power but power production grade apps for you personally. If you buy big beefy servers it will cost a bit more electricity.
Still, hosting locally can quickly pile up the savings on saas not spent.
Imagine being able to keep up all your test projects you might spin up a paid VPS for and run for way too long. They might not be production grade, but there is something valuable about having them around.
Data backups - the cloud is just someone else’s computer sold as convenient but not secure. Having your own local copy when the internet goes down means not as much of your life or work goes down. Remember it’s not a backup if there is only one copy of it.
Backing up your computers in the cloud are only so helpful when they are down, a lot of time to download them. Local backups win again.
Multiple connections can matter for people who wfh and can’t be down, or have spotty internet that is up.
If you like smart home tech, it’s a ticking clock until the cloud based supports for it that are usually free dissapear leaving perfectly good gear unusable. Instead you can run a local instance of home assistant, etc.
I used to host morethan I wanted ina. Data centre much like this rack. I was hesitant to come back to self hosting or homelabs but j have realized a home server that runs like an appliance (in between that wiring) capturing the sum of all my data and worn as services come and was pretty much unavoidable. Luckily it’s getting easier and easier to do.
Come on, let's admit it fellow HNers... Most of us are jealous/envious of this network setup and would love to have the money and time needed in order to own something similar. Extra points for powering all these from solar panels and also for all the "little" gizmos (ADB, NTP, LoRa, TinyPilot, MQTT and temp/humidity/power monitoring).
I'd guess a good portion of us have been at that point in our lives but has moved on and downsized or shudders at the thought of keeping that monstrosity of a rack in service or updated :P
I'm not, even though I also DIY. A multi-2.5GBe port fanless N6005 mini-PC (plus a cheap barebones WiFi AP) does it for router, switch, storage, game server (Valheim & Satisfactory), timeserver (chrony), DNS (unbound with blocklist), InfluxDB, MQTT, and it doesn't even sweat, and consumes less than 20W combined.
Was it a 1U server? 2U and 4U servers can be typically be configured or modified to run at a noise level comparable to a gaming desktop. Performance 1U servers you'll hear through the floor though for sure.
I can tell you my 1U R620 servers lasted less than 6 months before I sold them for 2U R720s. I still wouldn't want to sit in a room with the R720s but you couldn't hear them through a door. Sold those when I moved and downsized to a tower server R430, equivalent to a 4U. It's in my office now and no louder than my desktop. I'll get rid of the raid array and downsize to a single hdd at some point to silence the heads clicking though.
I'm more interested in my colocation setup than my home setup. I have 2x1u in two different data centres.
Just built one of my new servers (Cisco UCS 220 M5) with 2x 4TB Enterprise NVMe drives, 4x 2TB enterprise SAS SSD drives and 4x 2TB SSD drives, 256GB of ram and 2x Intel Gold 6230 (20cores). I don't even need this, I just want it.
Just deciding on if to keep with FreeBSD and bHyve which I love, or go with VMware.
If owning an IP block wasn't €LOL, I'd complete my monopoly set.
It's very cool, I just don't know what I would ever do with it. And I'm not saying that means therefore I don't see why anybody else would need or want it, I would never use it though. I was thinking about running an ethernet cable or two around the house to hook up a 1gb link but I couldn't be assed in the end because wifi is good enough. It's not like I can max out my 11mb/1mb home internet. \My new laptop's wifi is a bit flaky so maybe I'll do it this year if the drivers and firmware don't come good.
I'm definitely not jealous, it just looks like an annoying pain in the ass. I have a faster network than the OP and it's cheaper and wireless and I don't have any servers or cable everywhere nor maintenance to do...
I find this really disturbing. On the one hand this looks really nerdy and cool but then I see the immense amount of hardware that also relies on power... so much power. This is completely unnecessary but then I also see the pictures of an F150.. so yeah it makes sense. I have to categorize this as a (un)typical American person that does not care about the environment.
Do like the fact that he runs a RIPE probe though.
Sigh. This is a guy who is clearly passionate about his hobby and was excited to share it with us. Leave it to hn to shit all over it.
Some people fly to Africa and shoot elephants for fun - as far as hobbies go, this is fairly benign. If you care about the environment, I mean actually care, then you should recognize that policing how many computers someone have in their closet is counterproductive. I don't buy the "every little bit helps" argument because there is always an opportunity cost. Attention spans are limited and focusing on this non-issue does nothing but draw attention away from things that actually matter like meat consumption and fossil fuels.
In many U.S. states, their primary energy generation is nuclear. This is extremely environmentally friendly, meaning no issues with OP using a bunch of power.
We really need to stop demonising the use of energy. Energy is amazing. It is the reason we have seen to much prosperity in so little time. It's the backbone of our economies. Of medicine and the internet and EVs and well insulated homes. We should be doing everything we can to make energy cheaper, enabling more people to use more energy. Few things have been as damaging for climate activism than the war on energy. It's unnecessary and misses the point of the entire movement: saving the environment. Focus on cleaner sources of energy rather than demonising people using energy.
I looked it up, 5 US states get the majority of their electricity from nuclear power, 4 from wind, 3 from hydro. The other states use coal, natural gas and petroleum.
It's not about the energy for me. A lot of these posts make networking and homelabs seem like a dark art that only folks with tons of space, capital, and time can get into. Networking is anything but, it's about sending a packet of data over a wire! But when we glorify these elaborate, expensive setups, we not only lose focus on what the goal of a network should be (to satisfy needs for its users), we also discourage newcomers from tinkering with networks which to me is the bigger crime.
I also have mixed feelings. I stopped home labing because of the energy costs. I now have everything I need (HomeAssistant / NAS / Router) running on an old Dell Precision laptop. Laptops make great servers because they are optimised for low power consumption + you get a free console and UPS built-in!
I'm tempted by the new Alder Lake N100 mini PCs that are available on Chinese websites. But I need to wait until the laptop dies before I can justify it. It's not just about power consumption but the (hidden) cost of manufacturing. Make use of the stuff you already have before buying new stuff.
Maybe you could consider an old framework motherboards? I personally don't own one, but is tempting if you own a framework laptop or are planning on buy one, it seems to be one of the most sustainable options out there.
My main problem with labbing is the noise (I live in an apartment). His servers are mostly small PCs, so I think the most power draw is from the disk drives. If you need the storage, say for the cameras and whatnot, I'm not sure that running a laptop instead would save that much power.
I am a big fan of thin clients as servers. It is true you lose the "built-in console and UPS", but they usually come in very nice and compact form factor, many are fanless and consume 10~15W tops while being acceptably capable (in general much more than a RPi but with much better memory and storage specs)
Well it depends what you're doing with it. In my case my CPU is averaging about 1% with the odd burst here and there. As a precaution I propped it up to allow increased airflow underneath.
You do have a point in that certain devices can suffer from overcharging and bulging batteries. This issue is largely addressed in newer devices. I can see that the BMS in my laptop server is holding at 90% charge. Definitely worth checking if you plan to use an old phone/laptop/tablet as a 24/7 device.
I find your comment to be at the absolute peak of sanctimony. Who are you to decide if it is necessary or not? Furthermore, why are you justified in negatively stereotyping Americans – a diverse group of individuals?
A diverse group of individuals who consume 12 MWh of power per capita, while it's around 6 or 7 MWh for modern European countries. It's not even a stereotype, just statistics. (got my data in two minutes from Wikipedia, so maybe not up-to-date)
"who consume 12 MWh of power per capita, while it's around 6 or 7 MWh for modern European countries."
Lol, you write that as if the Europeans were consuming half as much because they are morally enlightened.
Europeans use 6Mwh/person because thats as much as their emasculated societies can steal from Africa. May I remind you they didn't leave Africa very willingly in the 50s-70s. But they kept on stealing anyway!
Thats right. Steal. Libya, Algeria, Niger. The French convertible Franc. The French senate bemoaning just this week that Macron "lost" Africa. Lost Africa? Africa is not France's to lose!
Unlike, say, Japan or Korea that have more than just a token few LNG terminals and pay for their gas with monitors and great cars.
I don't either but certainly it's a big part of it since I rarely have seen a US household without some sort of cooling and of course heating. Add to that we have larger houses than Europe, like larger.
A part time job of mine is helping a buddy's solar company and right now I'm working on a project, for an older couple who are building a 4200sqft house with 4 mini-splits, 2 electric boilers that switch to gas when it's cold, 3 refrigerators, 2 deep freezes and a potential future electric vehicle. There's a pool too but that changes the dynamics of the system greatly but I don't design and quote that unless they really want to spend $$$.
I'm glad they're deciding to go solar but I see this so often it hurts. An older couple without kids in a giant house in the desert that requires immense heating and cooling wants some solar to help offset their gross over consumption. Rather than scale down to a, still large, 2000sqft high efficiency house they go with cheap construction so they have more than enough space for the one time a year all the family shows up. Oh yah then they fill it with crap that's rarely if ever used.
There's no judgement thrown to the customer as this is the reality of the world we live in, but I see this as seriously problematic. Especially going forward as these older individuals die. Can a family of 4 or 6 afford that place? Do they even want a place so large which requires such an inordinate amount of maintenance? Three 1400sqft high efficiency homes could be built on approximately that same space bringing down the price of the home, reducing maintenance and providing more than one couple space to live.
Sorry, that was more of a rant than anything. Europe is not the promised land but the US really has some screwy priorities when it comes to housing.
I agree with pretty much everything you said. I'm trying to convince my own aging parents to downsize. They're already running into physical issues getting around their house, let alone trying to keep it clean, organized, and maintained.
Please don't make assumptions on something you don't know anything about.
Even many northern US states (around the same Latitude as Madrid, north), on average, are 30-35+ every day during the summer and 0C to -10C in the winter. Many Canadians have air con.
So yes, heating and air con is a requirement in many areas. I live in a northern state, and we regularly have full weeks that are 38-40C.
Hehe, you also described Madrid and basically the center of Spain. From -5 to 45 extremes. More normal is around 0 in colder winter days and 35+ in hotter summer days. 36 today.
>Why? IMO we should all avoid waste, whether that's plastic or energy.
Allow me to respond with a slight tangent:
Suppose you care about reducing plastic in the ocean. So you try to reduce your usage of plastic straws. You spend a whole lot of time convincing everyone you know to buy bamboo straws or such.
Plastic straws make up less than 0.1% of the plastic in the ocean. Meanwhile, fishing nets make up ~40%. Plastic straws are a hard problem to solve, and solving them accomplishes less than 1/400th of the fishing net problem.
And yet, we hear a lot about plastic straws but nothing about fishing nets. Why?
Because it's the more consumer-facing solution. It's vital that people focus on the solution that solves the problem, rather than wthe one that is most easily pointed to.
Should we do both? In theory, sure. But in practice people have finite time, and you have 30 seconds max before they start tuning you out, so you're better off giving your elevator pitch on fishing nets.
Using energy doesn't cause climate change. Producing energy from fossil fuels causes climate change. Humans don't need to minimise energy usage to fight climate change, instead they need to produce energy from non fossil sources. I wish we produced 10x energy that we do now but from renewable sources.
You're right, in the most meaningless sense of the word. I too wish we had 10x the energy from renewable sources, but we don't and won't for the foreseeable future. What we actually can do is get maybe 1/10 of our energy from renewable sources and then, to be sustainable, we need to use 1/10 of the energy we currently use.
How do you know OP cares about the environment? Because they criticize others?
I have no doubt we could examine OP’s life and find plenty of “wasteful” use of energy but no doubt OP would say “that’s different”. It’s always “the other guy” in these cases.
Add on top it’s a lazy comment that adds nothing to the conversation.
Good question, not sure how I would know the answer though. You probably should ask the guy. Given how he seems to track everything, I wouldn't be surprised if he came up with a way of measuring that as well.
Would any answer to your question mean mine is not one worth asking, or that it's not worth estimating those quantities to include in calculations of CO2e-opportunity cost?
Yes. This is a prime example of what people refer to when they say that someone is being an "asshole". The reason being that the OP is being incredibly condescending while simultaneously lacking any trace of empathy and knowledge.
I've seen stuff that or more vile etc. of course, but as far as sanctimoniousness goes, this is the worst I've seen in my several decades on the internet.
You do realize you're posting this self-righteous comment from a computer that was in all likelihood manufactured in China, right? I wonder how many tons of coal were burned to manufacture it. I wonder how many western developed nation comforts you're unwilling to give up while you "categorize" people on the internet based on the constructive activities they choose to engage in with their own time and money.
I dunno, a lot of people on here use more power without noticing it, because it's on whatever cloud provider they use. Every time I push some commits, an array of servers spins up in an AWS datacenter somewhere to run all of my verifications and do a build.
Second, this person seems to have a very large house anyway. My house manages with a modem, a wifi router and two IP cameras, but that can just scale up linearly.
When I grow up I want a switch rack like this in a basement or closet, wired network everywhere and PoE security cameras. Bigger house also means I can get an array of solar panels and become net positive.
> I dunno, a lot of people on here use more power without noticing it, because it's on whatever cloud provider they use. Every time I push some commits, an array of servers spins up in an AWS datacenter somewhere to run all of my verifications and do a build.
Most AWS (and generally cloud) instances are already virtualized from bigger machines. Power, heating, and cooling in these large racks is running much more efficiently than running these things in your home on dedicated compute that is only being used for your purposes. Even more efficient is using something like serverless compute or something like Cloudflare workers. While cloud compute probably is not running at low idle power, it's probably maximizing it's power utilization as much as possible.
> Bigger house also means I can get an array of solar panels and become net positive.
There's a lot of hidden costs associated with becoming "net positive", everything from adding more peak load during inclement events to issues associated with grid fragility.
Look I don't think it's good form to roast OP on what's really standard behavior on the internet: showing off your cool digs. But I also think we should be intentional about what it is: showing off your cool digs.
I do a lot of similar stuff in my home network with a lot less equipment. I have a trunk switch that sits behind my router, Ethernet running into each of my rooms (came with the house), and small point switches that sit in front of each patch. I have 3 APs in my house. I have a UPS which connects to our router, trunk switch, and a single AP that uses PoE which is good enough if we need Wifi when the power goes down. To much shock and horror, I just bought discount Cat6 cabling and didn't bother buying a roll and crimping and making my cabling look nice (but I did test every point in my network exhaustively to make sure I don't have high amounts of latency or jitter anywhere.)
The rest of the services that OP runs I run on a combination of a 12 year old SBC and a Raspberry Pi. Modern computers (read < 15 years old) when used as server appliances are very capable and can packet switch very quickly, even while running a kernel and a usermode. Doubt me? Run your own iperf tests and check where things are bottlenecking.
My IP cameras are all Raspberry Pis that use the camera module and stream using RTSP back to the SBC. Instead of managing VLANs (I do offer a guest VLAN but my partner and friends found it a pain so it doesn't get much use shrug), I run Zerotier through the house. My IP cameras sit on a single Zerotier network. All other services I run internally use Zerotier and if I do accept external traffic, I expose a port on a VPS (like OP) and have some nftables rules which NAT the traffic through Zerotier. I've thought about getting a failover ISP (our area has 2), but our primary ISP offers us fantastic peering (I've exhaustively tested the bandwidth and jitter to the VPS I proxy traffic through) and good enough uptime for me to not bother.
Like a lot of other hobbies, people can get very consumeristic. This is what a consumeristic version of homelabbing looks like. This shouldn't stop you or anyone else from building a humble homelab that can do almost all of what OPs home network does. The key is to test where your network has bottlenecks. I've iperf tested my network exhaustively and frequently look at kernel counters and other things to make sure that my network is performing up to spec, but that's a lot less fun than buying lots of cool networking gear. Much like you don't need an F-150 to get around day-to-day, maybe even just a bike :)
> I do a lot of similar stuff in my home network with a lot less equipment.
This is also my case. For Internet access, I use a router which has 4 2.5 GbE ports for the internal LAN, which was made several years ago with an Intel NUC and 4 USB Ethernet dongles, but which could be done today with a small and cheap fanless router using an "Alder Lake N" Intel N100 or N200 CPU. There are also such small routers with more 2.5 GbE ports for the internal LAN, e.g. 6 or 8. That router implements a lot of network services, including DNS servers for my domain and my own e-mail server.
I also have 5 servers that are interconnected by a 10 GbE subnetwork, but that is done without external switches. Each server has a dual-port 10 Gb/s NIC and they are interconnected directly, in a ring. The servers are normally powered off. Whenever needed, I power on the first of them through Wake-on-Ethernet. Through it, I can power on, also using Wake-on-Ethernet, as many of the others as needed.
Huh the ring topology is a really cool idea. With the right routing rules it could open up some ports on my trunk switch, though I'd need to upgrade my NICs. Great idea for whenever I decide to retire the SBC though!
When all the servers of the ring are powered on, the packets are distributed on both links, so the throughput of the network doubles.
When a part of the servers are powered off, the topology of the subnetwork becomes linear and the throughput becomes limited to 10 Gb/s, but with less active servers the traffic is also less.
Greater performance can be obtained by using two Ethernet cards with dual 10 Gb/s ports in each computer. With 4 ports per computer total interconnection can be achieved up to 5 nodes and above that each node can be connected to two intersecting rings, which is a topology very frequently used in HPC.
An alternative is to use a single card with dual 25 Gb/s ports, per computer. Such cards are not much more expensive than the 10 Gb/s NICs, but the 25 Gb/s switches are very expensive, so a ring topology without switches is even more attractive.
Its really disturbing how whiney Europeans (I assume you are) are with their superiority complexes.
I don't even know why I'm replying to you, but here we go
So I should ditch all this and rely on cloud services I guess. Cloud services are hosted in magic land where they don't use any power, right? If you use any cloud services, you are just as bad as me.
I have 17kw of solar on the roof. As we type this, I'm exporting 10kw of power back to the grid. I generate enough solar to completely cover all of my electricity usage, and during the night when I'm pulling from the grid, my power plan is sourced 100% from wind or solar.
All of this is my hobby, but also helps me learn skills for my job. That job is 100% work from home, which means I don't drive pretty much at all. I'm sorry my F150 offends you, which has less than 10,000 miles on the clock despite being 5 years old. That same F150 also gets better miles per gallon than a lot of regular cars on the road. My wifes Ford Escape (A compact "SUV" if you can call it that" gets worse milage than my truck. But I'm sure if you saw a Ford Escape, you'd be fine with it right?
Where in the post did I say how much power all this uses? You said "power... so much power" yet you don't even know how much power it uses...
I won't even go into detail how I've removed over 1000sqft of concrete on my property, how I've planted an 800sqft native wildflower meadow, have a bat house, use rainwater for plants, etc etc. But no I'm just such a terrible American. Grow up.
Its not quite as green as that right now, since we have a pretty bad drought here in Houston, but it will come back.
I had to make the small fence as it just kept trying to spread out. The fence is made from scrap 2x4's ripped down the middle to make 2x2's, and the rails are old scrap cedar pickets pressure washed and ripped down
Snide categorizations aside, there is a point somewhere in your comment. It is a running joke in the /r/homelab community that their servers use an unreasonable amount of power that they have to hide from their wives.
It really IS unnecessary, as you can tell by the author of the article referring to their home network as "overkill".
It's never about how green is the source of energy. It's about people doing things with their energy in which they don't approve. I mean... just read their xenophobic remarks about a giant country of diverse people.
There’s nothing hysterical about the OP’s argument that the setup is overkill and power hungry. It’s important to note here that OP is merely criticising the setup. It’s their right to speak freely about it just as you have the right to call them a climate hysterical. Stop shaming people for speaking their minds.
And my opinion is that the setup is indeed overkill. The author(s) literally admit so themselves.
My entire region runs on hydro, so even if I had a home lab x10 the size of this it wouldn't have the same environmental impact as someone who ran a laptop with power from a coal plant.
You can't tell from a blog post if what he's doing is good or bad for the environment. And further more, its unfair to tell people that they shouldn't improve their niche skills when those same skills are what they use for employment.
I find it very interesting how awareness of climate change (and let’s not forget: the price of energy) has had so much impact on my and others’ judgement in a relatively short time.
5-10 years ago I would think this is perfectly fine. I believe I was not alone in this, but maybe I was. The energy would have cost pennies too and why whine about it?
Now I think it’s exorbitant as well, for me. The F150 etc, it’s telling a different story now.
I do think it’s kind of below the belt to snipe any single culture for this. It’s too easy to target Americans. I don’t know about this guy’s life. Maybe he has a ranch with 20 square miles of renewables and I don’t care actually. It’s just personal reflection.
> 5-10 years ago I would think this is perfectly fine. I believe I was not alone in this, but maybe I was. The energy would have cost pennies too and why whine about it?
When I moved into my own home (8 years ago) I brought my 'homeserver' with me. Which was just a simple i5-2600 build with some shucked drives in it. I never thought about electricity prices when I lived with my parents. But that changed rather fast. With the server gobbling up a constant 90W, I quickly realized that, even back than, it would cost me 15 euro's a month on electricity alone.
I than proceeded to put a Pi next to it, that would listen to incoming Plex requests and would start up and shut down the server with WoL. That only reduced costs by about a third. The next couple of years I would move on to a NUC with a NAS that would only consume about 29W/h on average. Which was decent, but not great considering the poor performance of both machines (J4105 and J1800).
Last month I have gone back to the DIY route. Now with a i5-13500. I'm still in the process of optimizing it for power efficiency. Although much more stressful than the prebuilts, I love the hunt for the last watt.
Anyway, what I wanted to say is that I notice that family and friends don't really care about saving power in general. They mostly just pay for it and there's that. While my house runs 100% on electricity and I'm really proud if I can get 9kWh/day on average. Even when I see that (for example) the 8-bit guy uses 100kWh/day on average [1]. Which seems out of this world for me.
>I than proceeded to put a Pi next to it, that would listen to incoming Plex requests and would start up and shut down the server with WoL.
How hard is this to configure? I have a server at home I use to run a database and computational heavy code, however I am the only user so realistically it is only in use 8 hours a day and some weekends etc. However in the fear of forgetting to turn it on before I go to work (or if I suddenly find time to work while away) I find that I default to leave it on. Being able to control it would be fantastic.
Not at all. Just ensure that you have WoL enabled on the host machine and than proceed to send a magic packet. You could even do this with Home Assistant [1] if you are into that. I did this with a script that used tcpdump to monitor for incoming traffic [2] for Plex with an additional (dummy) Plex server on the Pi. I also remember faintly that I had to add 1 library and 1 video file to make this work though.
Powering down - or sleep - is a bit harder. I built a 'Sleep on LAN' app [3] for myself years ago that could power down (or sleep) a system on demand using a REST API. I used this and Tautulli [3] with Home Assistant that would check if there were any active streams and if there wasn't any activity for a specified amount of time I would send a SoL request to my service.
As you can see it isn't super hard or complicated, but a bit cumbersome to find all the moving bits and make it work. But when it does, it's IMHO fantastic.
If it's just that, you can have a Pi next to it and just ssh in to send a WoL command. Basically nothing to configure.
You can make it simpler to use by making an alias in your shell, or a button on your phone (with one of the countless "ssh button" apps). Or even make a web page for it (some php or python that just calls the WoL function).
OP describes a more transparent (and complex) setup where the Pi presumably acts as a reverse proxy. I'd be curious to know the exact setup too, one of the simplest ways would be to use wake on unicast: https://news.ycombinator.com/item?id=35627107
Other ways include wrapping some scripts around socat, writing your own proxy, systemd socket activation, etc.
I used tcpdump with a dummy Plex server that listened to incoming requests [1]. Because those request are automatically generated when a user opens up the Plex app. And I than proceeded to send a WoL request.
A reverse proxy would of probably worked too, but I didn't want to be limited by the 100Mbps network interface of my Pi 3B.
Hold on a second - how large of a household are we talking? House or apartment? What consumes all that power? Electric vehicle? Work from home?
I'm environmentally conscious (0 cars, 0 pets) but I haven't spent any time measuring and optimising electricity usage.
I've had a look at my own electricity usage every once in a while, now averaging 48 kWh/month (between 38 and 68) for a larger than average apartment for two people.
Isn't 48 kWh/month = ~1.6 kWh/day? That seems impossible. Even 4kWh / day seems impossible for me. I exclusively use electricity, except for heating. My water heater alone consumes about 2kWh/day. Not even talking about cooking, using my computer or watching tv.
I am not running serious network gear right now due to power, which is ruinously expensive in the UK to the extent that I find myself longing for PG&E's rates in California.
Currently having a house remodeled, which I'm putting as much solar (and battery) on as possible, and the gas is being removed entirely. Going to try not to use more than the system can generate, though I am not sure I'll be successful.
Yes, we should care about the environment. No, we should not be using less energy, but more. The way we use energy is the reason why it hurts the environment, not the amount of energy. I think this mindset is hurting our ability to innovate.
Americans tend to drive larger cars than other nationalities. These cars use more fuel. Most of these people do not need large cars and would be fine with smaller ones that use less fuel. Therefore, by choosing larger cars, they're using more fuel than necessary. An explanation is that only care about their own comfort, not the environment.
Unfortunately that trend is going international as I see other countries also starting to buy up huuuuge SUVs etc.
My childhood in NZ was like, medium sized perfectly reasonable Japanese cars on the road, some small-ish (by American standards) utes where people needed to haul a month's worth of groceries back to the farm. Now everyone & their Mum is driving an SUV in the UK (where I'm living now) and this is made even more obscene by the narrow, historic roads here.
And you look at the inside of these plush, rounded range rovers designed not for crawling up a mountain but the urban commute and if you're lucky enough to spot one not being driven by a single occupant (also common), expecting ridiculous cargo and passenger space and...no, they've just somehow managed to fill the space with more nothing. 2-2.5 tonnes of nothing. A >3L engine to move your nothing around.
Even American friends of mine here agree & laugh at "oh v6/v8 is common and they're 3-7L and yet aren't really that efficient at making power".
Not really, American is pretty urbanized. Less than 20% of the population lives in rural areas. Yet the most popular vehicle for the past few years has been the ford F150. It's not being bought by rural or working class people, it's a luxury status symbol now.
> American is pretty urbanized. Less than 20% of the population lives in rural areas
i'd like that number put into perspective, but it does not feel like on the low end.
> Yet the most popular vehicle for the past few years has been the ford F150. It's not being bought by rural or working class people, it's a luxury status symbol now.
thanks for the insight! but i struggle with the combination of "most popular" and "luxury status symbol"
On that last statement, see iPhone for an example of a ubiquitous luxury status symbol with it occupying the top three slots for smartphone sales by model. However, it is not accurate to say that rural people are not buying F150s, they like status symbols as much as the next person and it does fit the requirements if someone wants a truck.
Sometimes people who live in urban areas like to pretend that no one lives outside of cities except cows and guns. 20% is still quite a few people at about 60 million. Not a exactly a rounding error.
As a Kiwi now living in the UK, "unpaved roads" are not an excuse for the American excess found in vehicles like the F150, F350 etc. Unless you're literally crawling over raw grassy hills, a standard sized Japanese car (like a civic etc) will still do gravel roads just fine.
I hope you're trolling.
Power can be generated without an environmental impact. There are companies (and F150s) dumping kgbjillions more CO2 than this person's homelab is. Let people learn something and have some fun.
Power cannot be generated without an environmental impact. Even solar panel and wind turbine generate a non negligible amount of pollution when produced, and a non negligible amount of waste when they have to be decommissioned.
Also, it is highly unlikely that he is using his own grid, he probably pulling most of his power from the U.S grid, which (like most grid to be fair) contains a fair amount of CO2 heavy sources.
And that is not even talking about the amount of pollution and waste used to produced all that hardware he is running.
I get it, it looks cool and damn fun. I dream of this kinds of setup. but brushing off the environmental impact is also wrong.
Now admittedly, he is but a drop in vast ocean when it come to pollution and waste, and there is definitely far worse offender. But keeping our environmental impact in mind should be a necessity in this era.
There, it's in mind; what can we do now to stop the big companies from making things worse?
Individual accountability is important, but self-flagellation or (vicarious) shame because forces bigger than us are causing problems is a waste of energy (ha!).
Big companies don't produce pollution in a vacuum. Almost all of their pollution is a indirect consequence of our own consumption. Buying a ford F-150 and having a oversized server rack full of expensive hardware that generate a lot of pollution to be produces is a sure way not to stop big companies.
Responsible consumption is the first important step to stop big companies from making thing worse.
This is the argument that holds the most weight with me. It's why, as much as I approve of Extinction Rebellion and their kin, I think it's counterproductive to deflate SUV tires, when the environment harm caused by SUVs is a fraction of that caused by the companies profiting off oil and gas extraction and refining. If you're gonna do a crime anyway...
It's similar to when there's hypocrisy charges levied against "champagne socialists." Nice virtue signalling for worker's rights for someone with a 6 figure income... meanwhile that 6 figure salary accounts for 0.01% the net worth of a billionaire.
Go for the high stakes first, then we can worry about homelabs pulling more power than they should.
My homelab is a mess of cables, and half-baked deployments running on old Supermicro and custom servers, that is way too noisy, and probably pulls a lot of power. I wouldn't know, as I've yet to track power usage... (One of those projects I never get around to ';-D)
Everything's stacked on a Lack rack, even though I have a 24U rack in a closet doing nothing. This is partly because of space and cabling constraints, but _one day_ it will all be neat and tidy, clean and efficient. One of these days...
>My homelab is a mess of cables, and half-baked deployments running on old Supermicro and custom servers, that is way too noisy, and probably pulls a lot of power.
Then you could have bought 4-6" Cat 5e/6a pre-made patch cables to connect in the majority of your patch panel ports. The only cables that would traverse through the cable mgmt panel would be longer cables to the servers in your rack.
I have a similar Cisco switch as your 24 port. I am not fan of how they arranged all the ports at the right side of the enclosure. Why not make the 24p one row along the top (or bottom), like the 48p with the 2nd row removed? It would be a lot easier for cable mgmt.
The reason its arranged like that is because it started as 1 switch and 1 patch panel, and slowly evolved, so its not ideal, but I am fairly happy with the outcome
If I could re-rack it all, I might make some changes. But that means turning everything off, and I'm not sure when, if ever, I want to do that
I had the patch panels above a collection of switches in both the MDF and IDF closet. About a year in I rearranged it to the more common interleaved switches and panels, mostly just to make it look a bit cleaner.
I moved from 10g to 40g for switch interconnects in a few places, as well as used the fiber I installed to do 10g to most of the desktops, etc. Fun stuff for sure!
I noticed this as well. The patch panels seem only to bring some wire to the front of the rack, only to immediately send the patched wire toward the back of the rack again.
Biggest problems I have faced for home networks : ISPs are poorly configured and do very shady things sometimes like DNS hijacking. More often than not poor ONU firmware or hardware means you will see weird things like latency spikes or straight up packet drops. Sometimes IPv6 won't work if the ONU is in bridge mode etc. etc.
This post is overkill but what mostly works for home networks:
SQM - like cake sqm in openwrt - will work wonders if don't have a great ISP. I have seen as much as 5% packet loss on one ISP who said thats acceptable for a home network. Just reduce the load until a point where packet loss is really low. Then use a DNS resolver. DNS resolver will be much more reliable like unbound for your local network as packet loss has cause really problems with DNS for me. Cheap home network equipment dns is not reliable. You could really just buy a cheap Pi and run a local resolver. It would work a lot better than your home router. You may also want to consider replacing the ONU. This is rather easier than you think. The ISP ONU are really bad. A cheap NOKIA SFP GPON is like 20$ on eBay and if you can figure out the PLOAM password and Serial and Mac and VPN ID of your ISP, you will be able to simply swap it for a more stable link.
All in all I would suggest for a simple home setup: get a cheap SFP GPON and replace your ONU. Get a cheap router (either router on stick with Rpi4 for <500mbit or x86 box with SFP for hitting 1gbps with the SFP GPON) and run your own unbound resolver. Get some APs and either wire them or get a cheap mesh router 3 pack on amazon (you can get a wifi 6 3 pack for under 150$ these days on sale) and run it in AP mode.
Why would you use a local DNS resolver instead of using a remote well-known resolver like 1.1.1.1 or 8.8.8.8 in a home networking situation? Your systems will cache DNS entries to paper over unavailability issues, and as long as you use one as your primary and the other as a secondary, there's a very low likelihood (probably < 0.0001%) that you'll ever have issues resolving DNS.
If you look at systems like Deco (and a bunch of others I've noticed recently like linksys) the devices have stopped being dns forwarders themselves to clients and instead they assign your ISP (or the ones you've set for your ISP) DNS directly to clients. This means your device now resolves dns over wifi directly. This means each device is now relying directly on google and cloudflare instead of relying on the gateway (which being wired will experience the lowest latency especially when it comes to redirects and should ideally have a robust DNS server).
A local network resolver instead of forwarder on a client seems to work a lot better in my experience especially when I use root DNS on unbound. Yes a root DNS is a lot slower than something like 8.8.8.8 but you could use an upstream dns with unbound if you really want something faster. Either way, a local DNS will be lot less problematic compared to if you devices has to handle all dns over wifi including the redirects (which happen a lot).
Almost every time I see a network problem, it's always a DNS problem. In a typical home network setup these days:
client DNS cache -> upstream DNS -> root DNS.
client DNS cache -> local DNS - > upstream DNS -> root DNS.
My setup:
client DNS cache -> local DNS -> root DNS.
The first one works better if the network is wired. Second and third one works better on wireless.
One alternative that works 'okay' is if the client directly uses DoT or DoH for DNS. TCP works little more reliably than UDP (wink) but neither Google nor Cloudflare offer signed profiles for iOS and macOS (third party is there). Stubby works good on linux and android and windows have a built in solution.
For a homelab yes, but I figured GP was referring to a robust home network, not necessarily a homelab. "it's always dns" goes both ways as I've had local resolvers add huge amounts of latency to connection establishment.
Almost all ISPs use Serial and (optional) PLOAM to authenticate your ONU. Nokia GPON SFPs allow you to change almost everything that is exposed to the ISP for authentication. ITU standards mean that ISPs have to work according to the spec.
ISP ONUs are locked for easier management for them. They will give you 1/10th of the speed promised if it helps them deal with support calls remotely rather than visiting your place to fix it. Support calls where people complain for things like 'I forgot my wifi password' cost ISPs a lot of money. So they can basically just login to your ONU at any time remotely and change settings for you like your wifi password. They do remote firmware updates and what not remotely. This is the biggest reason why ISPs love such modem router combos. Support can be guaranteed with a phone call to fix your wifi for the average Joe.
You will not get ISP support if you use your own ONU but if you are using your own ONU then you are already at that point where you know what you are doing. As far as signal issues are concerned, like I said ITU specs mean they can see the signal strength remotely. Everything else they don't have access to but you don't need them to have that access. You can always swap out the ISP provided box to troubleshoot.
If you actually swap out the ONU to a better one, chances are you'll never need to call your ISP unless there is a fiber cut or some serious signal loss somewhere.
I used to have a home network, until lightning hit nearby.
It came up the power connection and the copper phone connection. The modem/router/switch fried, then it spread out though the twisted pair Ethernet and fried every interface connected to the network.
These days I use WiFi unless I really need low latency / high reliability in a particular location. Optical would be nice, but WiFi is the poor man's lightning safe network.
Yeah this is why I like fiber optic cables. Lightning can't surge the modem if there's no electrical connection.
And then standard power isolation and surge protection will handle all the power connections.
I've yet to loose equipment to lightning from the power cables but I literally can't count the amount of equipment that has been killed by lightning striking a copper data line and then surging through my network.
+1 for the TP-link, I've got one running fiber to my AV equipment :)
It was a lot easier to run a pair of fiber-optic cables there than it would've been to do the same with cat6. Also I got to play around with fiber for the first time.
Never thought of isolating my modem from the rest of my gear though, I need to look into that.
You can never get perfect isolation from lightning (a direct hit will almost certainly take something out) but you can do worse than making sure you have an excellent grounding rod setup and a whole-home surge protector, which you monitor.
And then you can put surge equipment on any other incoming wires.
When lightning strikes, the phone network and power network are often driven to different electric potentials, meaning a current will flow between them. When that happens equipment that is connected to both the phone line and the power will act as an (expensive) fuse.
With only one connection (power or phone) the whole device tends swing at the same potential. Damage can occur if it arcs to ground, but that's less likely as in a device the isolation to ground is typically higher than between the power and data parts of a circuit. (Extra bad news if the arc to ground happens via a person.)
Incidentally, if a lightning strike is nearby a typical consumer surge protector will improve the odds but probably won't prevent damage.
When I was younger lightning struck our apartment building and it killed my consumer router. I replaced it and put it on the surge protector.
A few months later we got struck again. The router survived but the building caught fire.
Now that I'm older I just accept the risk that "sometimes nature hates you". Instead of jumping through hoops to protect my gear from nature I just set aside money to replace anything I can't afford to lose.
Life's too short to play tug of war with the planet and I'd rather set aside $50/month to pay for replacements when I need them.
I had an ISA dialup modem that got hit by lightning. Loud boom then my DUN connection hung up.
When it came back, that modem still worked, but it was stuff "off the hook" forever. I had to unplug/replug it every time I wanted to dial out, and the phone line was permanently busy.
I love this so much! For anyone asking why, the simple answer is: because they can! Hats off to you, Sir.
I wonder if this person works in IT? I’m a dev lead, I do silly amounts of hours and when I actually have a decent amount of free free time up my sleeve, the last thing I want to do is tinker with my homelab.
I have a similar network setup (actually a bit bigger) and a much larger computation setup. I also have a 20kw 53 panel solar setup, and a complete 42kwh battery setup.
Overall power usage throughout the year for me is pretty high, on the order of 8MWh/month, and I get about 25 MWhs of total solar production. My servers and network gear take about 4kw sustained, which is 35MHws, so my solar covers about 70% of my computation and network power usage. I live the NW, so I produce most of that power in the central 5 months of the year.
If you don't mind me asking, does your energy bill take a huge blow because of this? I had a modest homelab set up and had to start shutting things off due to how much it costs to keep it running
Having a rack alone doesn't consume anything, it's what you put on the rack.
I have a rack setup and most of the time it consumes around 200 watts during the daytime and 100 watts at night, but can spike upto 600 watts if I put a heavy CPU+GPU load on it.
I also put my desktop into suspend at night, something which I think a lot more people with desktops could do. Don't run 24/7 services (e.g. Home Assistant) on your massive desktop with an i7/i9 and a GPU. Run that stuff on a NUC or Pi4 or anything that has low power consumption. Then turn your desktop on only when you're actually using it.
I get that but the blog discussed a pretty beefy setup. My whole rack still used less power than my gaming PC at load but after doing the math I ultimately saved more money by going serverless for my apps and dumping (non-critical) data into a B2 bucket on paper. In reality I just started shutting things off and only turn them on when I need them.
I'm curious as to what others are doing to save costs if anything. I love the hobby but we're in a recession, lol!
If you don’t require ECC RAM a common setup these days is a Proxmox cluster set up on refurbished USFF PCs like the Lenovo m720. They sip power at idle.
My personal approach is separate archived storage from working storage and keep archive storage offline until I need it. Keeping hard drives spinning is costly.
MiniITX boards are low idle power, 17 watts for idle with a Ryzen 5600 compared to a full size boards 50 watts. I would no longer buy anything larger, even for a gaming machine.
Amen on that. The past few rigs I built for myself were all ITX machines. I went through and undervolted my CPU, RAM, and GPU. Once my current gaming rig dies I'm just going to stick with the consoles for gaming until they goes to shit as well.
In an era of abundance we seem to be trying to use more power to support what we already do, rather than rethinking what we do to use less power. It's a shame!
Honestly I could I could easily solar power all of my compute needs (and probably a lot more) from my apartment's roof.
But at last, I'm a renter and my property management isn't going to allow that.
We're literally wasting clean energy because of a bunch of rich turds hogging real estate with shitty rules about what you can't do. When my property changed management the new management even banned EV charging on the property and locked up all the outlets in the parking lot. These kind of management are not welcome in Silicon Valley but somehow they are here.
It jumbles my jimmies hearing about outright rejection of modern tech by industries that would best benefit from said tech.
I'd almost be willing to bet money that your landlord would charge a "renewable energy" fee even if they're just dumping power back to the grid for $$$ even if they had the panels
I really wish more Mini-ITX boards came out that allowed GPUs to be horizontal so it could fit in a 2U case. Unfortunately making it vertical makes it 4U, negating the size advantages of Mini-ITX.
So I actually "repurposed" a riser cable from an old ITX build to slap it into my primary proxmox server to play around with home-rolled VDI. Surprisingly, it worked. If you're feeling bold you could try that route
I had bought an used enterprise server (Dell something), but it just used too much power and made some much noise. It's not as professional or reliable as op's homelab, but I'm now hosting most of my stuff on an Intel NUC. It's a much better fit for my use case and budget
That would definitely be a big help, lol. I also imagine part of my problem is my house wiring- the previous owner had the place reno'd in '96 and clearly got the landlord special
Much nicer than my setup. I have three small form factor PCs running OpenBSD tied together with a small Netgear gigabit switch. Each has an external drive for backups using rsync and dump. Copies of the backups are made to another drive for offsite storage. One computer is a workstation, the other two are servers. One server is for internal use, the other is for external services. WireGuard is used to tie the outward-facing server to an Internet hosted server so services can be passed between them. There's no UPS though. I didn't want to deal with battery maintenance... Power outages are taken when they occur. One thing about that, the computers bios are set to leave the computers off if the power blips. The whole thing sits on a single shelf of the bookcase.
Wow, and my wife says my network is complicated! :)
In all seriousness, thanks for sharing, this is really incredible. I see a few similarities (fellow Harbor Freight shopper, ADS-B receiver)... but I took the mostly lazy way out and just use the TP Link Omada router, controller and access points. Works great for well over 50 wireless clients that we had at a recent BBQ. I particularly love your note about encrypted LoRa networks at the bottom there, I'll be interested in a follow-up on that topic. Thanks again!
I've been very interested in the new TP-Link stuff, they have really come a long way. I don't know if I've really ever heard of much wrong with the TP-Link stuff
Exactly. Those 50 wireless clients must be fed Internet while their 50 subjugated humans eat BBQ and prioritize device interactions over human ones. (I’m exaggerating of course! At least you are having gatherings — better than many of us!)
but is astonishing how expensive is the connectivity in other countries.
>>> Verizon Gateway sitting on top of the rack. I get pretty good signal here and get the rated speed, of around 300Mb/s down and 20Mb/s up. This costs $50/mo.
There are offers [0] in the EU of 10Gbps / 25 Euros/mo.
But why? I looked at the page but could find no understanding of what he gained from doing all this? I have a single wifi router and power everything through that and still have much faster bandwidth than the author does. It seems like a lot of energy, time, and effort (especially for maintenance) for something with minimal gains. I get 1 Gbit down and 500 mbit up and even in the worst signal rooms it only drops to 600 mbit down and 93 mbit up). Printers etc all work through wifi and backups are done to several redundant cloud services. No wires required, no expensive and energy guzzling servers needed, no patching or maintenance needed.
As someone that has build huge nation wide networks and data centers I can tell you this is far from what I want at home (now)! When I was a young network engineer I had a Sun SPARC5 running OpenBSD as a firewall. Ultra10 for my desktop. I ran my own DNS and a rack of servers. Wifi was new and of course I ran cutting edge OSS firmware on whatever was the latest HW. Too much work!
Today, Ubiquity HW.
Enterprise 48 PoE, 1/2.5/10G core, small PoE powered 5 port switch behind TV, etc. for fan out where needed (I did rewire house with Cat6A). 3 of the latest APs that support 6E. NVR for the cameras. TrueNAS MiniXL+ for NAS that I can run K8s on if needed (cheaper to pay IXsysten for the setup then build myself, including time). AppleTV for media. Leviton Wifi switches for remote control power/lights. 5 gig symmetric fiber internet with a Comcast backup that auto fails over (UDMP-SE for firewall). It all just works and I spend zero time on it.
About the only tech head thing I still have is a stand alone 4 generation old CPU (i7) running ... gentoo targeted at that CPU for speed, as a Minecraft server. And honestly that is going to be replaced with a 2 year old SFF box soon with a less complex version of Linux :)
Are there any building code considerations (e.g., chimney effect for a fire) regarding a multi-floor wiring chase? Isn't this why laundry chutes are no longer permitted in some places?
Riser or plenum rated cables are required by code if you go between floors, except in plenum rated conduit. Conduit is usually more expensive though.
Non rated cable more easily catches fire and makes a lot of smoke. Plenum rated cable is about twice the price, but ethernet is pretty cheap for a single home.
I've also been curious about this in the context of making a dumbwaiter, but not enough to do the research. I would think having passive automatically closing fire rated doors would take care of it, but I don't know.
You could look into it but I think the reason laundry chutes aren't made anymore has more to do with kids getting stuck in them and less to do with fire code (after all, stairways are chutes connecting floors).
Of course there was another solution (I’m assuming you’re not still in the chute, by the way) - which is to make them so big that even a large adult male can’t get stuck.
But the main reason they’re gone now is that people put the laundry on the same floor as the bedrooms in modern homes.
I've been fascinated by home labbing for a while now, and maintain something with comical simplicity.
ESXI 8.0.1 worked out of the box with the 7090 I was given.
I'm migrating it from 6.7u3, which is running in 14 year old hardware (injecting the drivers was easy btw - finding the drivers was difficult).
ESXi is not managed with vCenter, it's a standalone host. I'm hosting a handful of VM's that do everything I need.
Critical items are never sorted solely on the server; they're stored on three remote sources, with integrity checks , and mock restorations.
Fortigate 140d PoE, which handles a phone system and all of the PoE I need.
Backups are simple. No RAID, again, critical stuff is not solely on the server.
I use enterprise disks, and SMART - I'll get a notification with ntfy if something gets wonky.
Server is consuming 60-160w. 16gb of RAM, non ECC, 5th gen i5, 12TB of storage.
Homebridge, home assistant, scrypted, Asterisk, Samba, Timemachine backups (all containers configured with compose, checked into Git) windows 10 and 7 VM's
All of the hardware is second hand, with exception to the disks. Ambient in the closet is 29, no dedicated cooling.
UPS with cyberpower.
Glad to see you're using solar, and that your hardware too is used. When I worked in the industry they'd toss stuff I could never afford...
FWIW: Proxmox seemed like something new to learn entirely, ESXi seemed similar for me; I don't mind the RAID stance, or the licensing...
Looking at this setup makes me think it would be interesting to have a public-facing chat service on a 30-second clock where the user has to pad tweet-sized messages with a pre-determined number of zeros and complete sending them off before the deadline is up:
1g lounge: must pad message with 1gig of zeros
10g lounge: must pad with 10 gigs of zeros
100g lounge: must pad with 100gigs of zeros
1t lounge: must pad with 1 terabyte of zeros
100t lounge: intelligence community meet-and-greet
I’m in Spain and get 10Gbit symmetric which comes down to about 7000mbps up and down on speedtests to my airbnb (in a small town 30min away from a touristic hotspot) for 30 euro’s. My home in Barcelona gets 1gbit symmetric for 25 euro.
Competition here has gotten pretty good with lots of virtual carrier being allowed by law to use telefonicas ftth which has driven prices down.
I’m the opposite to you where I’m always surprised when I hear people in fairly big towns/cities in US/UK etc are still on less then 100mbit symmetric!
I'm in the UK (Oxfordshire) and have a 1gb symmetrical connection at home. It's provided by Gigaclear - there's a handful of other similar operators that do fibre in more "rural" areas. It costs £79 a month, so it's not cheap to be honest, but I love it.
Ring em up and tell them you’re leaving, they’ll drop you down to the new customer pricing. I’ve got the 1G up and down for something like £38 a month now?
Should hopefully change in the next few years, about 54% of houses have fibre to the property and the plan is over the next five years to expand that to nearly 100%. I have had 100/1000 for a while now but symmetric is still a rare product that only smaller competing fibre companies are rolling out.
I had 1g symmetric in the bay area (thanks at&t) and it was nice, but am now around your speeds (85m/13m) and it's clearly worse, but not really terrible. Certainly not terrible enough to pay $50k+ install to get munifiber, even though I'd enjoy it a lot. Maybe if one of the ISPs on munifiber starts offering 10g to residences. Not that I need it, but it'd be fun.
he said solar, which is pretty nice if (1) he is generating excess solar in the day (2) he cannot sell it back to the grid (3) it's not cost effective to store in batteries for night use (4) there's nothing else to spend the energy on.
otherwise, i can imagine it'll still be cheaper to not run things than to run things. but it's a hobby, people spend energy and resources on more frivolous things, and those frivolous things make us human.
> The generator is a 27kw Generator which powers everything in the house. This means the UPS's only need at most 10 seconds of runtime, as that's how long it takes for the generator to start and switch
That Generac generator though: it powers everything but only in case there's a power outage right?
P.S: there's a slight typo in TFA: it's Mellanox, not Mollonox.
I just bought a Ubiquiti Dream Machine SE along with a U6 Enterprise a few weeks ago and so far it’s my favorite tech purchase ever. The management interface is years if not decades ahead of everything else I’ve used before.
My only complaint is that there’s no public API and thus no official Terraform providers.
I would take Ruckus Unleashed over unifi all day long. Ubiquiti, unfortunately, feels like everything is constantly in beta, both hardware and software. Wait until they release a UDM SE v2 and abandon firmware on the UDM SE.
The UDM in particular is a masterclass in how to upset all your customers. (coming from a previous all-in UBNT customer that had a first gen UDM Pro).
Ebay Ruckus + OPNsense and my network has never been more stable and performant.
Yeah I've heard a lot of mixed things about their software support but I did a lot of research and people seem to say it's gotten a lot better over the past couple years. I certainly wouldn't say any of it feels like a beta, it's currently been up since I bought it two weeks ago with no issues.
I actually moved away from ubiquiti stuff to OPNsense + TP link AP. The firewalls rules on OPNsense makes a ton more sense and the plugins are pretty awesome.
You can run a UniFi hardware controller through a plugin on OPNsense. Worked pretty well minus mongoDB conflicts with zenarmor. But I switched to omada APs because they had better range and speed.
I love my Dream Machine too. I bought it about six months ago and it's been solid and was really easy for me, a novice with advanced networking, to get going with vlans and firewall rules.
I don't need this setup at all, like I don't need cat6 ports in every room, there's definitely an element of it being a hobby. Maybe I just like flashing LEDs.
Sonos gear still using STP was an unwanted pain, all part of the fun I guess.
I wanted to replace my EdgeRouter Lite with one of these but sadly Unifi hasn’t reached feature parity with their Edge line. The Dream Machines for example do not support multiple VLANs on the WAN port, when many ISPs use that configuration to deliver internet, VoIP and TV.
I also read some bad stories of the Dream Machines unable to restart after a power loss.
The Unifi software is pretty incredible. I am not using their router though (well... an ER-4 but it does not share the same management mechanism) so I am missing out on a lot of the goodies.
I keep meaning to produce a post like this. My setup is not nearly as pretty though. Love to see the cannabis grow operation exhaust fan, haha. Very well done!
Thank you for encouraging others. You’ve earned a loyal fan.
And thanks for sharing your knowledge. Infrastructure has been an off and on hobby, but it’s hard to know how to take it to the next level. Or absurd levels (in the best possible way), in your case.
I didn’t see a filter on the exhaust fan. He is so thorough I’m sure he thought of it but I think he needs a filter to keep the fan running long term unless he is filtering the air into the room and that is good enough?
What are the plans for the Meshastic mesh? I think having one hooked up to a server to do responses could be interesting. Or have you actually convinced other people to carry one around for messages?
My first thought is - imagine hiring this guy to architect your corporate network. The cost would probably be 10x for hardware, for the "just in case" factor and coolness to the setup...
For a corporate network I would love to have more "just in case". We had a day long wifi outage at the office once, productivity plummeted and the cost of that far outweighs the cost of buying and setting up back-up hardware.
I am reading this during a network outage thats been going on for half of the day now. I Wish we had some of that "just in case" but the higher ups get what they paid for I suppose.
This is a pretty cool setup! I'm personally working on getting a backup ISP connection going this week as well (two different isps with wildly different upstreams), since poorly configured ISP network issues have been plaguing me (random packet loss out of nowhere, PPPoE issues, scenic routing to companies that have POPs on my city, etc..) and I've pretty much exhausted all ISPs in my area and they all have some manner of weird issue. So two ISPs it is!
My setup currently consists of a single server with an i3-12100F, which does all of nas + hosting (9TB array for windows image backup of all local machines over smb + Home Assistant + AgentDVR) and networking (dual ISP in, single gigabit out, all handled by systemd-networkd and ip + firewall), and several 1G-CAT6 runs connected to an unmanaged switch. Wireless as several random brand APs scattered around the house, all running on low power mode and being pretty much room-confined, 5GHz everywhere, had mesh before and my experience is it just sucks!
Your post made me interested in those RIPE Atlas boxes.. I might see if they are willing to send one down here for me.. with two different ISPs on different upstreams I might have some interesting data.. and maybe use that raspberry-pi I have for a stratum gps server like you did as well... and maybe a weather station too.. down the rabbit hole I go again!
I love looking at other people's setups, especially if they have done a clean-job on it.
Currently I have four small racks:
Rack #1 & Rack #2 dual side-by-side (custom built)
Rack #1
NAS x1 4U
NAS x2 4U
Patch panel 1U
Rack #2
PDU 1U
UPS 4U
UDM Pro 2U
24 Port switch 1U
Rackmount cable modem 1U
Rack #3 (it's an arcade cabinet)
Workstation 4U
Rackmount Mac Minis (Intel & ARM) 2U
UPS 4U
Rack #4 (custom built)
Supermicro 4U server (330TB) that doubles as the VR computer
Supermicro 4U storage box (700TB)
PDU 1U
UPS 2U
I like tinkering, but I don't rely on my racks for anything other than compute and storage. Technically I have a partial rack in a data center that I have never physically seen, where I self-host all the usual services, but as I only rent that rack, I don't consider it "mine." Tinkering with a homelab is a useful IT skill to have, even if you don't work directly in an IT only area, but you shouldn't burden yourself with an overly complicated setup. Keep life simple (and organized).
Nice setup! I have a very similar Homelab minus the Generac (I regret not getting one before inflation kicked in, especially since I already have LNG to the home.)
My only recommendation would be switching your virtualization over to Proxmox (LXC / KVM) and setting up an HA cluster with Ceph and MLAG. It's relatively easy and free and will give you a lot more features than plain ESXi and even free vSphere/vCenter.
Yeah, the price on this genset I think has gone up around $4000 since I bought it, not including the install
I've been meaning to try Proxmox, but my day job heavily relies on ESXi, so its nice having something to mess with at home. I am also running vSphere with an Enterprise licence, so I get all the fancy stuff
Eh kinda... priority traffic shows the maximum potential speed of your link. The max speed of your link has nothing to do with the speed you'll get on any particular node on the internet. For example if the ISP has too little peering bandwidth to some other speed test.net host you will still be capped by that link.
These days I rarely need large bandwidth to any single host, but have a lot of bandwidth use between many hosts including streaming applications. Bufferbloat is far more apt to be problematic under this workload than ISP priority, unless there is a bandwidth shortage on the local links.
fast.com is the only browser based speedtest that shows the accurate results for me. The results from there resembles the download speeds I get when downloading files from different sites or installing upgrades through my package manager.
Cloudflare's speedtest often performs worse for me.
iperf3 - https://iperf.fr
Is the most accurate CLI speed test I've found (depending on which server you choose).
What's "accurate" is going to be different for every website / destination network anyways. Unless you suspect there's some issue with the speedtest software running on the browser / target, it's showing you the accurate speed you get to Cloudflare or whatever server you're trying to pull from. If you just want to see what your ISP is limiting your bandwidth to, sure, you need to find a server that can hit that.
The best speed test is to download some large linux ISOs with 100+ peers, so you max out your connection to tons of networks.
"accurate" is for me the speed I have on average throughout the day no matter what endpoint and not the max througput.
If I'm doing a speedtest, most of the time I just want to know if I'm now downloading file $x from $y what download duration can I expect.
Downloading packages with $PACKAGE_MANAGER, downloading files from $CLOUD_STORAGE, downloading videos from youtube, downloading linux ISOs, receiving files through $INSTANT_MESSENGER, ... they all are very close to the speed I get with fast.com.
Not surprised. I have long noted that when my my browser is strugeling to load tabs all i have to do is try connecting to speedtest.net for all open tabs to suddenly load and i have decent speed for the next 15-30 minutes.
Not coming in with proof but I have personally seen this- a buddy and I did some testing and the Ookla tests were the only ones we did that were much higher than the others.
I'm sorry for not contributing with hard evidence but I've seen this behavior (though I never heard about it from anyone else before now)
I'd start with something like this[0]. The only search term I used was "xeon-d". You can easily find other form factors and combinations. This line of CPUs is lower power (the linked one is rated at 45W), so should be trivially cool-able with a very quiet system. They support ECC RAM. You're likely to find mostly Mini-ITX and uATX boards, so they will fit in just about any case you want.
I've done this a few times, and the killer has always been the RAM. By the time you're buying decommissioned enterprise hardware, it's hard (or just expensive) to source RAM from its QVL list - even harder so when you want ECC, which not all of the QVL'd SKUs will be... And, in my painful experience, if it's not QVL RAM there's a good chance it just won't post.
I've had no problem with getting the RAM (either new or just more eBay stuff) - and even got a replacement mainboard for relatively cheap when one burned up (it was easier to swap the mainboard than reconfigure everything for a new server).
The killer is the power consumption. It's better now compared to old enterprise gear of 20 years ago, but you can sometimes still pay for a brand new low-power system just in power savings alone.
Of course, if you have solar onsite that doesn't matter.
The power is part of why I prefer the embedded boards like the one I linked.
Especially putting one of these not in a rackmount chassis, you can opt for quiet cooling and fans that do not draw 10s of watts each. The idle loads can get fairly low, depending on your additional peripherals.
For very low power, I actually prefer small form factor machines, either used 1L SFF machines from the big OEMs (see STH's Project TinyMiniMicro[0]) or one of the many AliExpress mini PCs[1]. These sorts of machines tend to use low-power CPUs and a lot of the AliExpress ones are laptop parts.
As with all things, there are always tradeoffs. If you enjoy researching and bargain hunting for hardware, then this is a viable path. If you just want a machine that works, then this is onerous work.
The top spec 32GB RAM on the qualified list for the specific board I linked earlier runs $40-$41 on ebay. The same speed and capacity for generic DDR4 runs in the mid $30s. The board has 4 RAM slots. So the premium for QVL RAM for that linked board is $20 over generic if you want to max out at 128GB.
Supermicro did have RAM issues in the early teens.
This sort of research is the non-dollar price of finding deals on server-class hardware.
If you want to get something pre-validated, and usually with a warranty, it can be worth it to look for resellers for the major OEMs that also offer refurbished hardware. It's more expensive than the typical options available on eBay, but still much cheaper than new hardware.
Some Ryzen chips (I think- might be TR) often support ECC- see if you can find a compatible mobo that supports ECC modules and Ryzen sand! When I last checked there were a few products out there
Without looking at the listing I can't earnestly say what you're seeing isn't fake but I know back when I had the same question you're asking here I saw similar offerings. Ultimately I bought a used ITX supermicro mobo+CPU combo unit from eBay for like $200
I'll have to go over the whole post, I'm terrible at writing! Quite a few errors have been pointed out. But I can just edit them and pretend they never happened
The Pis are using GPS receivers to serve as NTP servers on the local network. The specific appear to use GPIO pins. You would not be able to wire these to the ESXI machines as readily. You typically also want time-sensitive workloads running on metal, as there are all sorts of potential complications with virtualized time.
I am sure there are PCIe solutions that could easily go into a more standard form-factor machine, but perhaps these are more expensive or less readily available; this is just conjecture. The ESXi machines have minimal PCIe connectivity. Perhaps such a card could be put into one of the Supermicro chassis.
GPS modules use SPI which you can get a standard USB serial adapter for $2 on eBay. As the title suggests, it's overkill. Basically it's using the Rpi as SPI adapter.
Edit: USB to TTL costs like $3.5, it's like a standard RS232 serial port but much simpler and faster.
If he's using the timing pin on a GPS module, you want it on GPIO or an actual built-in serial port, because SPI or USB or other adapters can't be configured to trigger a CPU interrupt within a few CPU cycles.
I wish every apartment would have at least some units of at least half-depth of rack space. Fitting anything (even a basic setup of a PaspberryPi + the PON modem + a minuscule switch + a PoE inhector) into the tiny space behind hatch where the cables from all the wall sockets end in my condo apartment is extremely hard. And I am still looking for a 5-port switch which would be manageable, support vlans, handle gigabit ethernet and inject PoE if possible while being really small.
Looks great but I woud prefer to avoid Ubiquiti because I don't want to involve a 3-rd party cloud to manage it.
But I really appreciate your suggestion, perhaps I am going stick to it as I've seen no alternative so far and my previous experience with Ubiquiti makes me confident it will do the actual job great.
This one seems suiting my preferences perfectly. I even wonder how did I overlook it. Now I'm looking for a WiFi6 access point which would work on the passive PoE it would feed.
You really helped me a lot! I wish you could feel how my mood has improved. Now I feel empowered to implement the set-up I wanted. This is a minor home improvement task but it sitted in my to-do list (after numerous attempts) for over a year.
I wish 10" half-width racks had taken off. They are available in Europe but not the US. They would be good size for home networking gear, if anything supported it, and gear would fit in normal rack with bracket.
Is there a place nearby for cabinet, table, or shelves? Those are good for hiding network gear.
To say "couldn't live without it" would be an exaggeration, but we do use it extensively for work, and having it at home makes WFH much easier. End-to-end connectivity, like in the old days, is very nice!
Looks like they've got dual-wan (at&t fiber + verizon 5G backup). It's pretty easy to do ipv4 failover with NAT as long as you've got an indication that it needs doing; ipv6 failover is a lot harder; unless you're going to NAT the v6 traffic too. I don't know about Apple devices, but I couldn't get Android or Windows to play nice with two v6 ranges in radvd. They wouldn't listen to priority, and they would do derpy things like pick an address from one router's advertisement and then send the traffic to the other router.
In some magic happy land where you can use your IPv6 prefix over both WANs, then yeah, things can work, but that doesn't work for residences, generally.
Yeah, even with Comcast/Xfinity I run a HE.net tunnel for IPv6, just so I can have truly static allocations and not goof around with their DHCPv6 prefix request system.
It's been around for a long time! If you do the IPv6 training certification, they do eventually send the t-shirt :P
EDIT: They also provide free DNS for up to 50 domains at dns.he.net which is also worth looking at if you don't want to run your own for every little thing.
I haven’t used it in a few years, but one downside of an HE tunnel was that Netflix blocked traffic from it (since people were using it like a VPN to avoid geoblocks).
I had to run a DNS proxy to filter out AAAA records for any Netflix domains.
HE and Cogent are still feuding over IPv6 peering, too. Using HE for regular web browsing will work fine, but if you want to host through HE your services might not be reachable to visitors from a Cogent network, e.g. office buildings.
The norm seems to be for hosting services to contract with both HE and Cogent if they decide to go with one of them. This is why for typical client browsing you wouldn't notice--the other side has ensured you can reach them. HE seems to be the cheaper option, so people will route most of their traffic through HE, and then separately contract with Cogent for the smaller amount of traffic originating from/to Cogent networks. Alternatively, you contract with another network provider who handles that idiocy for you.
That is true, and a bunch of folks (including me) were experiencing issues with Google-related sites blocking with a 403 FORBIDDEN on the initial /64 static allocation from HE.net. Oddly enough, any subnet out of the optional routed /48 you can enable worked fine.
For sites that use HE.net and have more than one IPv6 subnet, I use the initial /64 for the DMZ, so that didn't matter on most of them. Small sites had just been using the /64 though, and I had to renumber those. Total pain.
Probably not supplied, I’ve got no IPv6 on any connection I have access to other than a few Azure VMs (and it’s a PITA to set up so we haven’t bothered)
there's no kill like overkill! i thought this was going to be pretty meh, but indeed it was amazing. my favorite:
> Raspberry Pi 3B+ NTP server with GPS/PPS time source
Just one though? I have a triple LeoNTP device. Terminated my own cables to outdoor GPS antennae. Been working on designing my own but it's slow going.
I also have UPS power routed elsewhere in the house. But just to 1 room, where I had a powered ISDN phone. I couldn't find an ISDN phone I liked that took power from the line, if that's even a thing. I don't remember as that's ancient history now and I don't use the remote 240v power at all, just PoE everywhere.
The one thing missing is EPO (Emergency Power Off). I built my data closet into the garage, stealing part of a garage bay. It is the size of a smaller sauna room. I have an EPO on the outside, wired to the UPS.
I added card access to the data closet last year, because I also added it to the outside entries of the house, and I had one controller port left over.
I don't have nearly as much switching, and no fiber. But many servers. I would buy an 0xide rack if they weren't $500k.
I don't know if it was just a lucky guess, but as soon as I saw that first pic of the house exterior with the BMW, I said "that looks like it is in Houston", and sure enough...
Can someone with more knowledge comment on whether that particular design is more common in Houston or was it really just a lucky guess? I kinda feel like that carport-style design is very Houston, haven't seen it as much in other cities.
Overkill indeed - got just a single router but it is so flexible (TP Link Omada) and runs various VPN tunnels without rebooting or hanging.
No fancy CPU just a dual core MIPS based MediaTek 7621AT - the key difference is the software, stable and well designed and not some hacked together version of OpenWRT and some old version of Linux from some random OEM that never get's updated after release.
>The Hubitat hub has now almost entirely been replaced by a HomeAssistant virtual machine. The only reason it is here is to connect the Z-Wave and Zigbee devices I have. However, I replace those devices with WiFi versions wherever possible
There's cheap and reliable ZigBee/Z-Wave/433MHz USB dongled antennas you can replace the Hubitat with.
what no 100g or IB? Those price point have come to a point where for home lab is bearable. Well we are talking about overkill. If one want to play around with network do it virtually is a much cheaper way to go: https://www.gns3.com/
This is beautiful and inspirational! I am about to redo my messy homelab and have two questions:
1. I don’t see any labels attached to the cables, how do you know which-is-what?
2. I am on pfsense too but I was considering a move to OPNSense, would you explain why you choose pfsense over opnsense?
Thanks
Very nice setup, thanks for documenting it! I used to have a much more involved at-home network, but since moving the business out of the garage, I've moved most of the network stuff with it. Still think I'm going to have to pay pole attach fees and string my own fiber if I ever want it at the house...
If only I could find a way to get someone else to pay for it :P I tried to convince one of the local electrical co-ops to do it, and use my day-job's building (we own it) as the CO for the area. They actually sounded interested but then went radio silent.
Brightspeed did basically the same thing a few months ago, but they kinda suck.
It’s a black hole for your wallet. Once you start. You can’t stop.
But I also now have a bunch of electrician tools for running wires, some private cameras that aren’t google or Amazon spying on everyone that walks by, and a network that isn’t bogged down by a couple people watching Netflix.
But, its all used and all over a number of years, so not as much as you'd think. If you went out and tried to get it all at once, it would be crazy. Most things I spent a good while waiting for the right deal to come up
Like the UPS alone, I saved probably close to $3000 off MSRP
If you're crafty you can work around this- three of my servers (2 NAS boxes and a Proxmox box) were custom built from commodity hardware. I literally bought plain ol uATX and ITX decommed business gear from my local uni surplus store and a P-Link chassis for each used from eBay. I spent more on drives than I did machines (Y'ALL WANNA KNOW HOW MUCH IT COSTS TO BUY
16TB WORTH OF SSDS FOR ZFS MIRRORS? TOO. DAMN. MUCH.)
You really gotta do some due diligence to make sure you're not buying lemon parts but it's very possible to get a beefy homelab at fair prices!
I’m in a very similar situation when it comes connections, FW and needs. I’m about to get a backup line for internet and it’s between Verizon (just like you) or Spectrum (same price, 500mbit down).
What was your reason for VZW? Was ATT your only wired option?
This doesn't seem that overkill to me other than using fiber optics, although I've done that in the past, unless you're doing exceptionally long runs copper cabling is sufficient even at 10Gbit within a home scenario. Seems like a decent setup.
True. That said, fiber can be painful to deal with. Electrical isolation is one big benefit, as is power savings, and of course forward-compatibility. I still prefer running copper for residential, especially because you don’t have to be as careful about bend radius and such.
Within my rack I use DACs to connect all the gear, which works well since it allows maximum performance over short runs with minimal effort.
Nice setup though, my comment wasn’t meant to criticize, I think having a proper network at home is a big boost to quality of life that most folks don’t realize because they’ve never experienced it.
Surplus 10G switches at good prices are likely to be sfp+ and not 10g-baseT at this point in time, too. DAC for same rack is a no brainer at that point. To the rest of your house depends on what you've got in the wall and how easy it is to put new stuff, I guess. I'm running 10G-baseT between my two network cabinets, because there's no economical way to pull anything else, and there's no reason to go for more than 1g other than fun.
Cool setup. I have worked in Telecom industry before and going to build something similar from second hand devices, but then switched to dev and now mostly wires left (sold all this telecom stuff to pay expences) :)
Here you would need a breaker after the UPS before going into the wall because UPS is not considered up to code for breaking leaking current and all outlets must have protection. That's why I ended up not having a central UPS. How is it in USA?
Ok seriously jealous of the setup. Have always wanted a rack - just because.
I have way too many power cords under the desk but use fused plugs and boards. Want a small cabinet just to keep cables and devices away from small curious people.
When I worked with racks in datacenters, we usually put the network gear facing the other side of the servers, so there was no need to cross cables for the rack depth.
There are 2x Internet connections + racks - one in a house closet rack, the other in the garage.
Garage rack has a NAS that mirrors storage of the closet rack.
Garage rack has one Raspberry Pi GPS for time services.
Closet rack has two (different) Raspberry Pi GPS for time services.
Looking at it that way, if you accept the 2x Internet connections each being independant and acting as cross backups in case of ??? then it's just the second GPS on the closet rack that's "extra".
That can likely be written up as a home experiment in messing about with a different Pi setup.
It's overkill .. but often these home setups are there to keep current on new technology and different ways of doing the same thing .. in that light it's not uncommon to have multiple ways of achieving the same end just to gain knowledge on the pros | cons of different approaches.
I'm sorry- when did they start doing this? I've been mulling making the switch to OPNSense. If pfSense is phoning home aside from the update check I might have a busy weekend ahead of me
I dunno if this is entirely true- my current pfSense router gets as close to line speed as I'll ever be able to get with my setup (~900 both ways to a test server in Chicago and on LAN) and I'm running an (overkill) i5 in my generic Chinese minipc. It's possible that I don't have Intel NICs but I swear I do
Got anything more I can read? Dunno if I'm just blind but I couldn't find anything on this. I'd like to learn more!
We had no problem routing gigabit on an Intel Atom D525 years and years ago at a previous job. Consumer Mini-ITX board, onboard gigabit NIC, and an Intel gigabit NIC in the one expansion slot. It did require minor tuning, but nothing that couldn't be done thru the web UI.
Everything I have to manage runs plain OpenBSD managed with Ansible now, so I don't know what the current state of pfSense/OPNsense throughput is.
To add to my other comment: AFAIK Ruckus is not targeting consumers. They sell to managed services providers and probably even then not directly but through a reseller. The MSRP for APs in this sense is much like MSRP for rack servers - it's huge on paper but in reality the price is chipped away by volume discounts, long term agreements and licenses. Retired(yet perfectly capable) APs go through ITAD and end up on ebay/craigslist for much better prices.
I have been running Ruckus I got off ebay for quite a while now. Flashed it into unleashed mode and it has been rock solid. In fact I am typing this comment while being about 100 ft away from the AP.
One of the main temptations to having a house built (probably partially by me) is that I would be able to do all the wiring correctly (not just conduit in the usual places, but decorative trim in each room that easily removes to reveal the wiring cavity in the wall behind it, etc).
These days i'm more pragmatic. I have exactly 2 wired machines, and everything else runs Wifi. The network "rack" is simply a firewall and a 16 port POE switch, everything else network related has been retired to the closet.
My NAS is gone. With electricity frequently going above €1/kWh last winter, it seemed like a bad idea to have a NAS sitting there, consuming ~35 kWh/month. Instead i spent about €20/month on cloud storage, and simply upload everything to the cloud (encrypted with Cryptomator), and i have redundant cloud storage (different provider) included in the cost which is where my backup goes.
All that's left at home is a small ARM based machine that handles making daily backups of my cloud data locally, as well as run a small Plex library on a single USB3 external drive.
Besides saving about half my previous electricity cost (as well as all the hardware cost!), i have gained SO MUCH spare time. My firewall is completely closed (save a wireguard VPN), and i no longer have to maintain anything except the normal patch routines.