I'm curious about why you're using lxd. Is it just that you wanted to try something different from Docker and its rivals? Or is there a reason you think lxd is better for your setup? For a service per container, I figured minimal, immutable containers, rather than containers running full distros, would be better.
The primary reason is that LXD has an indisputably better overall security policy than Docker. They support isolated user namespaces (containers running with different userns mappings), user namespaces are the default, they make use of far more new kernel hardening features than Docker, and so on. If I'm going to self-host something at home and expose it to the internet, I'm simply not going to use Docker.
I used to run Docker containers several years ago, but I found them far more frustrating to manage. --restart policies were fairly hairy to make sure they actually worked properly, the whole "link" system in Docker is pretty frustrating to use, docker-compose has a laundry-list of problems, and so on. With LXD I have a fairly resilient setup that just requires a few proxy devices to link services together, and boot.autostart always works.
Personally, I also find it much simpler to manage a couple of services as full-distro containers. Having to maintain your own Dockerfiles to work around bugs (and missteps) in the "official library" Docker images also added a bunch of senseless headaches. I just have a few scripts that will auto-set up a new LXD container using my configuration -- so I can throw away and recreate any one of my LXD containers.
[Note: I do actually maintain runc -- which is the runtime underneath Docker -- and I've contributed to Docker a fair bit in the past. So all of the above is a bit more than just uneducated conjecture.]
Have you taken a look at podman / buildah? My understanding is that podman resolves all of the security concerns you highlight above while mostly maintaining compatability with the docker cli and existing docker images. It gets rid of the docker daemon so your containers (and restart policy) can just be managed by your existing service manager.
I only just recently discovered podman and I've been pretty excited. Having never used LXD and only understanding the high level differences between the two, I'm curious how it compares with regards to security and usability.
I'm a little bit too familiar with podman. LXD is more mature and actually implements all of the hardening features I mentioned. podman could implement them in theory, but doesn't. Its default security posture is very similar (though not the same as) Docker. Don't get me wrong, I do want to see podman succeed -- but I don't like the amount of unneeded hype around it. It's effectively a Docker rewrite by Red Hat (and other folks) that has some fairly important improvements, but it's not a revolutionary new concept. As for buildah, I am too biased to respond to that question.
Oh, and most of the Docker CVEs found in recent years -- including those I've found -- have also impacted podman. The most brazen example is that podman was vulnerable to a trivial symlink attack that I fixed in Docker 5 years ago[1,2]. It turns out that both Docker and podman were vulnerable to a more complicated attack, but the fact that podman didn't do any special handling of symlinks is just odd.
I have to admit, the biggest selling point to me for podman is the removal of the central docker daemon. For my use case (personal workstation and home lab), it seems strange to me that I need essentially another service manager for these processes just because I want to slap them in a container. It definitely makes sense that there would still be some gaps though as it's a less mature product.
You've definitely convinced me to take a good look at LXC/LXD though. Thanks for the thorough response!
It should be noted that (unlike Docker), LXD can be safely killed and upgraded without your containers dying -- which is the main problem most people have with Docker's container liveliness model (even with Docker's --live-restore there are many issues). The main reason why LXD has a daemon is that it supports lots of management features (such as live migration and clustering) which cannot easily be done without a daemon.
You can use LXC directly if you want to avoid a long-running daemon.
