>"You can make any protocol work with a custom proxy. Take DNS: your edge servers listen for UDP packets, slap PROXY headers on them, relay the packets to worker servers, unwrap them, and deliver them to containers. You can intercept all of UDP with AF_PACKET sockets, and write the last hop packet that way too to fake addresses out. And at first, that's how I implemented this for Fly."
This is really interesting. I looked at the linked blog post and was hoping there was more implementation details. Does your Fly pi-hole use HAProxy and the PROXY headers then? Is the config for that available anywhere i could see?
No, the Pi-hole example uses the XDP UDP scheme this blog post talks about: DNS packets arrive on edge servers, XDP intercepts them before they reach the IP stack, puts a proxy header on the message (we don't use HAProxy's proxy protocol, to conserve space), and relays it out WireGuard; TC BPF attached to the WireGuard interface on the other end (the worker server) strips off the header, fixes the addresses accordingly, and relays to the tap interface for the right worker.
The first cut of this feature I built, without BPF, used NFQueue (diverting packets based on iptables rules to userspace), did a sockets-based proxy from edge to worker, and used a simple raw socket to fix the addresses and write the packet to its destination. NFQueue was annoying to work with, I looked at BPF filters instead, and ultimately wound up just doing the whole thing in BPF.
You don't need to know anything about this to use UDP on Fly.io; you can just add UDP ports the same way you'd add TCP ports (the `fly.toml` in the Pi-hole blog post shows an example).
XDP UDP mapping to firecracker vms via WireGuard is really interesting! I have a question a bit before UDP is landed on the NIC, assuming the NICs on the edge servers is connected to multiple transit providers for incoming and outgoing traffic. This mean from the VM perspective, you can have incoming/outgoing tap/tun inside the VMs able to receive packets from difference transits or outbound, did you do anything with this aspect? and if so do you also deal with ECMP inbound in such that you can have the same virtual IP receiving UDP on multiple edge servers?
There's not much to know! "XDP" is really just the Linux term of art for "BPF running directly off the network driver". Your BPF program --- ordinarily, just a C program you compiled with clang --- is given a struct with pointers to the beginning and end of a packet, and your program can return OK, DROP, or REDIRECT, in addition to modifying the packet.
The XDP project itself has a pretty excellent tutorial:
>"You can make any protocol work with a custom proxy. Take DNS: your edge servers listen for UDP packets, slap PROXY headers on them, relay the packets to worker servers, unwrap them, and deliver them to containers. You can intercept all of UDP with AF_PACKET sockets, and write the last hop packet that way too to fake addresses out. And at first, that's how I implemented this for Fly."
This is really interesting. I looked at the linked blog post and was hoping there was more implementation details. Does your Fly pi-hole use HAProxy and the PROXY headers then? Is the config for that available anywhere i could see?