>I always felt, and still feel, that applied Linux networking is difficult to get started with, mainly due to lack of good guidance. Most of the time I had to dig through small pieces of documentation scattered throughout the internet, trying to put them together to form a systematic overview of the network stack in Linux.
...
>It is extremely frustrating when somebody interested in setting up their own network infrastructure has to at some point get stuck at some convoluted networking concepts, intricate and abstract tools, mysterious errors here and there, or lack of systematic documentation. I wish everyone has some choices other than spending days and weeks trying to figure these out alone, so I decided to write down what I have done, what I have learned and what I have to share with the rest of the internet. I sincerely hope that some day IT operations would be more beginner-friendly, and hosting one's own network infrastructure no longer means headache and mess.
These are exactly my feelings. Stinky.fish makes more sense for a Linux blog. Everything about it stinks until it actually works. :)
I experienced this back when I configured my home Linux boxes as a router, VPN server, firewall, media server, etc. Since I had the time, compiled all of the info I found on random blogs and sites and added them to the Ubuntu Community wiki. That was the 12.x days, when Ubuntu was in its prime and the distro to use.
While these blogs were a great resource, I often found the commands outdated or applied to a different distro. A distro specific wiki solves both those issue. While I don't get the glory of a blog, I just checked an it's nice to see my notes still there for future Denvercoder9's.
The Gentoo Wiki was a great resource for many networking questions, even when I wasn't using Gentoo.
A major hurdle is simply learning how to describe what you want to do "in the industry terms" - "I can't access my server from my computer but it works from the Internet" is a lot easier to resolve when you learn what "hairpin NAT" is.
Ahhh, it sounds like you've only done this once. I started off with ipfw, then ipchains, then iptables and now whatever firewalld supports. OK that's roughly 25 years so not too much firewalling churn! I stopped hand rolling my own rule sets with ipchains and switched to generators and there are loads of them.
For me some of the problems nowadays are caused by search engine manipulation. Up until around five or so years ago Linux concept searches would get you pointed at the usual big hitters - Arch/Gentoo/Ubuntu/etc wikis and useful and quite well known blogs. My modern block list for ublacklist is huge and barely scratches the surface.
Now I come to think of it, we now have ChatGPT and I bet it can roll a decent ruleset without hallucinating madly. No doubt someone will soon be Showing HN: their smart new firewall prompt generator language for <insert AI here>. It will make the LLM use Rust as an intermediary for extra safety.
I found WireGuard to actually be especially frustrating, because it doesn't really log anything. If you do a single configuration misstep the packets just won't flow. Then good luck figuring out whether it's the authentication that's wrong (or any of the cert checks in the chain), rounting, MTU, firewall or anything inbetween.
It's kinda horrible. Not that IPsec is any better. OpenVPN at least yells something at you.
I'm wondering whether one can get these notifications through netlink. Not having any way to get the kinds of feedback you're mentioning from a stateful thing is horrible user experience. It doesn't need to cover the firewall part, mtu rejects, etc. Because if it's actually modular, you're supposed to be able use the usual tools (/proc/net and /sys/net for stats and netlink for firewall logs?), hopefully they're usable...
We recently switched a bunch of stuff from OpenVPN to Wireguard. A number of the links were OpenVPN layer 2 tunnels to pass, of all things, Novell Netware running on IPX (the particular situation precludes switching to TCP/IP for those customers). Now, layer 2 tunneling is being performed using RFC 3378 EtherIP, and it's much more performant, not to mention easier to manage.
Good grief - NetWare with only IPX! Presumably you'll be installing their Y2K patches any day soon 8)
I'm pretty sure NetWare can natively tunnel IPX/SPX over TCP/IP, assuming you are not stuck on 3.12. I don't think you have to fork out for the multi protocol router thing. They will crash regularly until you get the magic combination spot on and then run forever.
A NW 6.5 box will run quite happily as a pretty tiny VM - you could scatter them around as routers to support whatever nightmare of an app you need to run over IPX. Tunnel them over IPSEC or whatever floats your boat.
That particular client is stuck on NetWare 5.1 and due to how bad their maintenance has been I dare not touch the running instances. Virtualization didn't work out as there's some issue with the current patchlevel that dies on Intel CPUs greater than Pentium 4 (including hyperthreading P4s). It's...a stupid story.
On VMware you can hide CPU features - ie backrev the CPU and I'm sure HV can do the same, and no doubt KVM/QEMMU too.
I used to run a lot of NW back in the day. I remember deploying a NW 5 cluster of three Compaq boxes with six NICs each (for each VLAN) to do just DHCP/Dynamic DNS! I also ran up four HP boxes a year later with single ATM cards in them with a lot of VLANs to replace a load of 4.11 jobbies. The autoexec.ncf was a masterpiece! The cluster hosts had 6GB RAM each and despite being 32bit had quite a lot of cache which nss absolutely loved. I can't remember when NW 32 bit managed to devote >4Gb to cache but it was handy. As a file server it was absolutely unmatched. Apply an ACL and it simply worked - none of that marking subfolders and files thing that MS and Unix need. NDS/eDir was streets ahead of that weird LDAP n Kerberos thingie that MS "invented" and frankly still is.
I would suggest that you virty them as a matter of urgency. Presumably you have limited and dwindling hardware resources available and at some point something will go pop that can't be fixed. Once you have them as VMs then you can snapshot and all the other lovely things that virty brings to the game and you will never run out of hardware!
Pretty cool to hear about big NW installs! Thanks for sharing!
We're actively working on moving that customer off Netware entirely. The main reason they can't get away from it is they're running a custom god program that manages the entire business, and it's tied in pretty tightly with their old configuration. The whole thing is in Delphi 7.
W.R.T. old hardware, they are actually some of the newer old boxes we support! My main line of business is keeping old industrial control systems online and reliable. On the PC side, the oldest stuff we have in 24/7 operation is 286-based. Pre-PC, we have a few customers running CNC stuff on PDP-11s. I don't know if we still have any PDP-8 customers, I think most of them closed up shop during the pandemic.
You beat our DOS 5 booting control system into a cocked hat, that a customer is running. You've got to love cough technical debt. Why on earth is "Enterprise gone awry" considered the right way to go for your IT strategy over that boring old open source bollocks that has a nasty habit of still being supported or at least working decades later? To be fair, open source wasn't a thing in the 80s for most people (nor the 90s ... ).
PDP-11s are my uncle's era and I'm 53! My first real PC was a 80286 (don't forget the 80) I bought a '287 maths co pro for about £115 so I could run a dodgy copy of AutoCAD on it. I do still have my old Commodore 64 which now has a USB interface. I assembled a ZX80 or two ...
I basically just followed the OpenBSD documentation! One of the big advantages of OpenBSD is that pretty much everything you need to know is contained in the manpages.
As I'd said above, we ended up using RFC 3378 EtherIP to link the two layer 2 broadcast domains across the Wireguard tunnel. OpenBSD supports this with the etherip interface. You end up creating a bridge with the etherip interface and whatever physical Ethernet interfaces you want to bridge, on either side of the Wireguard tunnel.
I also tried VXLAN but did not have good results. I'm not entirely sure it wasn't a problem with my configuration. Traffic often went one-directional, where broadcast packets from Site A made it to Site B, but they did not come from Site B to Site A. EtherIP worked right off, so I didn't investigate further.
The docs are indeed great, but to me it seems like they are recommending GENEVE (RFC 8926):
> Generic Network Virtualization Encapsulation (GENEVE) supports all of the capabilities of VXLAN, NVGRE, and STT and was designed to overcome their perceived limitations. Many believe GENEVE could eventually replace these earlier formats entirely
I'm bit surprised that they didn't have section on vxlan there considering it is pretty popular afaik?
Anyways, I think tunneling GENEVE (or any other Ethernet-over-IP protocol) should work fine over WireGuard, same as using regular network interfaces.
Since WireGuard is Layer 3, what would is everyone's use case of doing Layer 2 on it? Or, what can it improve over existing solutions? I have tried to do the same for a bit while still learning networking, but ran into Layer 3 limitations.
Frustratingly enough, apparently not as I could never get it to work. It is pretty easy to set up a vxlan tunnel over wireguard if you absolutely need stuff like that though.
Oh hadn't thought about that, thanks. 'need' is a big word here but sometimes you can't change the client and server apps so, having support for the basic (although niche) features in the lower layers helps migrating smoothly.
Bonjour is built on top of DNS. You don't need a layer 2 tunnel to make it work.
However, it normally does rely on multicast. Rather than trying to bridge broadcast domains (which is going to cause performance issues), a more efficient option is to setup an Avahi mDNS reflector on either end of the tunnel to rebroadcast mDNS packets.
Alternatively, there's also a Wide-Area Bonjour service that works over unicast and doesn't need any special packet forwarding, provided you run a Bonjour-aware DNS server:
You are technically correct (best kind of correct) however, in reality, I see folks using L2 tunnels to solve for bonjour etc all the time. Usually those without networking knowledge to solve the forwarding.
Yeah, you can do it the right way...or you can just tunnel layer 2 and forget about it. I see it done a fair bit for both Bonjour/Avahi and DHCP (why?).
The best way to perform something like this on Layer 2 is to use Shortest Path Bridging (SPB) based on IEEE 802.1Q-2018. However the Linux kernel does not yet fully supporting this feature natively although the standard has been out for quite sometime and already being supported by commercial network solutions and the popular Open vSwitch (OVS) [1].
[1] Ask HN: Project ideas for a Linux kernel module:
Meta: Huh. I don’t use “shallow dive” enough. “Deep dive” of course, everyone loves a good deep dive. But what about a shallow dive, or even just “getting your feet wet”? These are useful concepts too. This headline alone revealed a blind spot for me.
> We will be using the local IPv6 addresses fc00::/7 for routing inside our WireGuard network. For example, if we use the subnet fc00:0:0:160::/64, this can be the new Address config on Server:
I just moved from OpenVPN to tailscale, which uses Wireguard, on my personal stuff. I have a similar situation as OP describes at first where my residential account has the ports blocked.
I am quite happy so far, just wish it was innately supported in my consumer grade router, which support vanilla wireguard.
Dumb question but the frp example shows op forwarding to port 443 on local, how then did the spammers get access to port 25 or was there a separate forward rule setup for 25?
It was incredibly so much easier when popular search engines did search instead of "recommendation." The hardest thing is actually finding the thing that exists, in multiple forms, that you set out to find.
For people using wireguard, it was not designed to provide anonymity. Otherwise it is fine for use in Countries with decent protections for their citizens. If you need privacy, you should use OpenVPN.
Quote:
>WireGuard is highly secure, but it’s not designed with privacy in mind.
> WireGuard is highly secure, but it’s not designed with privacy in mind.
I'm sorry, but I must inform you that the Toms guide contains affiliate links to OpenVPN services. However, it is important to note that neither OpenVPN nor WireGuard can guarantee your safety if you are being targeted by government agencies. The guide's attempt from TFA is to promote these VPN services as a solution for anonymity and censorship (deep packets inspection can block all VPN protocols) avoidance is misleading. VPNs are primarily useful for accessing corporate or home resources and viewing geo-blocked streaming content (say from your home network) on insecure networks like hotel or cafe WiFi.
At time of writing, the biggest privacy weakness that WireGuard has is how it assigns IP addresses. When you connect to a VPN service using OpenVPN or IKEv2, you’re assigned a different IP address each time. WireGuard instead gives you the same IP address each time. This is faster, but it means the VPN server must keep logs of your real IP address and connection timestamps.
The address assigned inside the tunnel has nothing to do with your real address, and definitely does not have anything to do with whether or not the VPN server is keeping logs of your real IP address and timestamps of your connection.
OpenVPN and charon keep far more logs of those things by default that wireguard and you have to trust your VPN provider turned them off.
Almost nothing was created with privacy in mind. Security and privacy are different things.
I hate that people think that a VPN is private as in anonymous. But then again, those providers had great marketing.. So now devs and sysops need to call VPNs "tunneled networks".
You're not wrong, but there are VPN services that add on privacy to their wireguard offerings, such as PIA (private internet access). They open sourced the connection code so you can see how they do it[1] using an API that initializes a temporary wireguard connection for you. I've been really pleased with PIA's wireguard setup, which even includes forwarding of an incoming port!
...
>It is extremely frustrating when somebody interested in setting up their own network infrastructure has to at some point get stuck at some convoluted networking concepts, intricate and abstract tools, mysterious errors here and there, or lack of systematic documentation. I wish everyone has some choices other than spending days and weeks trying to figure these out alone, so I decided to write down what I have done, what I have learned and what I have to share with the rest of the internet. I sincerely hope that some day IT operations would be more beginner-friendly, and hosting one's own network infrastructure no longer means headache and mess.
These are exactly my feelings. Stinky.fish makes more sense for a Linux blog. Everything about it stinks until it actually works. :)