In my experience, "slim" Debian images (like `python:slim`) aren't significantly larger than Alpine-based ones, but save lots of time and headache when something assumes glibc and breaks (or, worse - subtly breaks) with musl (or doesn't have a binary distribution for musl so every time image's built you have to build from source).
Also, I'm not sure what are the benefits going `FROM alpine` and installing nginx, than just starting `FROM nginx:alpine`. The latter has benefit of a more straightforward update logic when a new nginx version is released - `docker build` will "just" detect this. It won't notice that Alpine repos have an upgrade, though, and will reuse cached layers for `RUN apk install nginx`.
> In my experience, "slim" Debian images (like `python:slim`) aren't significantly larger than Alpine-based ones, but save lots of time and headache.
I came to pretty much the same conclusion too.
For years I was using Alpine but as of about a year ago I've been going with Debian Slim and haven't looked back.
I'd much rather have the confidence of using Debian inside of my images than save 100MB on a base image (which is about what it is in a real web project with lots of dependencies).
In my experience, the difference is sometimes even less than 100MiB (which is quite a lot). For the current ("real-world") project I'm working on, it's about 25MiB - something like 325MiB for Alpine and 350MiB for slim Debian base images.
Either way, it's not 1.12GiB I was getting with a fat `FROM python:3` base image.
Exactly. The author considers the fact they the official images are built from source as a negative. Though that’s what allows them to actually generate updated images.
They mention trust of who’s building the images, which is valid, but then using `apk install nginx` means you still have to trust that package maintained. It’s really just moving trust from Docker to Alpine.
It’s fair that it’s a reduction of entities that need be trusted since they are using Alpine as their operating system already, however they are still running Docker binaries...
It’s pretty neat to be able to say they don’t use Docker Hub for anything, but it doesn’t seem to offer any advantage.
Nah, that would require going completely `FROM scratch`. Otherwise that won't be true as `alpine` base image is still hosted on Docker Hub: https://hub.docker.com/_/alpine
Oh that is excellent news, thanks for the tip! I spent a bit of time trying to get Alpine working for a Python application with a relatively complicated set of dependencies, and Alpine was really not cooperative so I think I gave up and went with a fat image. I'll try it again with slim.
I would like to switch to a dockerized setup, but running everything on Debian/stable has the advantage of unattended-upgrades (which has worked absolutely flawlessly for me for years across dozens of snowflake VMs). Not going back to manual upgrades.
I tried a Registry/Gitea/Drone.io/Watchtower (all on same host, rebuilding themselves too) pipeline and it worked, but felt patched together. Doing it wrong?
If you are building everything on the same machine you could get rid of the registry and watchtower, mount the host machine docker socket into your pipeline, and build and start the image on the host machine from inside your pipeline.
I use dockerhub/github/drone/watchtower to automatically publish images and update services. I use dockerhub and github to avoid self-hosting the registry and version control. I also have about a dozen servers (as opposed to single-server in your example). This works really well for me and does not feel patchy. Yes there are moving parts, but fewer moving parts than a full-blown orchestration system.
> An aspect of Docker that I don’t really like is that inside the container you are root by default
PSA: everyone should turn on the userns option in docker daemon settings. It messes with volume mounts but you can turn it off on a per container basis (userns=host) or arrange a manual uid mapping for the mounts.
> The cornerstone of Docker is in its ability to use Linux control groups, namespace isolation, and images to create isolated execution environments in the form of Docker containers.
If you want to run dockers at home, i suggest you give it a try. All you need is an old computer and a USB (it runs in RAM). Unraid basically is Linux with a (happy little) webinterface for NAS shares + apps.
This only works if you trust all your docker images and audit them for root processes. Even then you still have setuid binaries and other privilege escalation risks, if you inherit from dockerhub images etc.
I'd suggest people looking to do something similar to this to check out Caddy as a reverse proxy for your services. It'll manage grabbing SSL certs for you and some people have already wrapped it into a nice docker container for you [0].
I recently had the pleasure of using Traefik[0] as my reverse proxy, which similarly handles SSL automatically via LetsEncrypt. Lovely piece of software!
I recently upgraded my simple web facing CV site over to a couple containers and front-ended with Traefik. What's really nice about it is if you run the host as a single-node Swarm you get a lot of freebies with regard to service discovery. I also use Traefik in my internal network to front-end Heimdall [0] in a container. This affords me a very elegant internal network dashboard, bastion host (proxy) and presents all my internal services with LetsEncrypt valid certificates (no more internal self-signed cert warnings).
I've been meaning to start blogging again and use this as a first topic.
>Note that Alpine Linux doesn’t use systemd, it uses OpenRC. This didn’t factor into my decision at all. systemd has worked well for me on my Arch Linux systems. [...]
How does systemd, or any init for that matter, come into the picture if you're running everything inside docker? Containers don't use any init, right? They just execute the binary in the host environment (but containerised). Or am I missing something?
Edit: nevermind, OP is using Alpine as the host OS as well.
Correct. Docker containers typically only contain one process, and that process runs as PID 1. If you need an init system, tini is very popular, and is now built in to docker itself[1]. Systemd is way heavy and overkill inside docker.
They don't always have one process though. And this is actually a significant good case for OpenRC, as it is far more lightweight than systemd. I haven't used tini.
Primarily, you would run an init system in a docker container in order to correctly proxy signals to the one process in question, which would otherwise run as PID 1 - for example, sending SIGTERM to docker run running a PID 1 with no registered handler will result in nothing, because Linux won't use the default handler (killing it).
Secondarily, if you want to be neat and save some pids and kernel memory, you need an init system to wait(3) on orphaned zombie processes.
These are the only two use cases AFAIK, which a small init system such as tini satisfies, without the complexity and size of systemd.
That doesn't answer the question: what does systemd provide which isn't needed?
I have multiple network devices. I want some to be controlled by processes running in a container; effectively I want some processes to run under a user account but still provide root (root-like?) access to the specified network device(s). I want to be able to give a specific (containerized) user full control over one or more specific network devices. My (naive?) understanding is that the init daemon takes care of bringing the network online and then subsequent management of it. For systemd, that would be Network Manager? Or do I misunderstand?
In my experience I had to update a bunch of scripts and configs which used systemd commands as well as using GNU versions of other commands (with some basic programs like `find` or `grep` this turned out to break a lot of stuff). That was my only big annoyance.
Plus the package manager and system structure was a bit of a learning curve.
But otherwise I was very happy with it. The lack of portability with my Linux ArchLinux/Debian desktop/servers can be overcome with time and experience like anything.
I used some basic Docker images as well but they always needed some config work, as they mostly installed the base software but didn’t get your apps running. Other people may be using more sophisticated images though.
You can run systemd inside a systemd-nspawn container. I've found this quite useful for debugging/testing custom Raspberry Pi disk images. You can simply import the rootfs into the container and have something that behaves almost like the real system, including startup, but right there on your development machine. I don't think you can do anything like that in Docker. But I do appreciate most people probably don't want/need that systemd support.
I just ran `docker run -it alpine /bin/sh` and when inside, I ran top. It looks like `/bin/sh` ran as PID 1. No init, just sh (busybox in this case).
Edit: Ah you meant the host OS. I can't reply to wezm down below for whatever reason (there's no "reply" button), so I'll just edit this to say I didn't realise that he was using alpine as your host OS as well. I haven't seen many people running it outside Docker, so it's quite interesting.
Host operating systems need a init, of course. I was just startled by the irrelevant discussion on alpine's init system when it isn't involved anywhere.
I haven't really used docker - so here is a dumb question; suppose one makes a setup like the author here, then what does a deployment of new version look like?
Suppose the author updates one of his rails apps and there some database schema modifications.
Is that handled by docker?
How long does a deployment take? (Minutes, seconds ... basically is the tool able to figure out what is changed and only apply the changes or does it remove the old installation and build the new from scratch?)
Deployment of a new version would depend upon your setup. Assuming a setup similar to the author, you can have a new Docker images with the new version of your code and run it in parallel. All you have to do after that is point the traffic from the old version to the new version (By just running `docker compose`).
If you have a more complex setup, e.g. if by using Kubernetes, you can do things like run both the version at the same time, person A/B testing or have canary deployments to ensure the new version works .
Time for deployment would be most likely in seconds unless the setup is complex/convoluted.
Schema modifications are another beast. For small use cases, you could run a specialized one time container that performs the modifications, but once you need high availability, you'd have to consider a more complex approach. See https://queue.acm.org/detail.cfm?id=3300018
Docker works in layers. Depending on how your layers are set up, it may be just a single diff from the existing image you already have, which can be quite fast. However, if you change one of the base layers, like Alpine Linux it will rebuild all layers above that. Good container design should make that rare.
This article was interesting to me because a lot of my personal infrastructure runs on FreeBSD. While I don't host anything publicly accessible, I do have some similar needs.
The author mentions the Docker port for FreeBSD. According to the FreeBSD Wiki, it's meant to run Linux Docker images and relies on FreeBSD's Linux ABI layer to do so. To me, this is the wrong approach.
FreeBSD already has good container technology, what it really needs are good tools around that. Since the author ended up building his own Docker images, I suspect that he'd be happy with a FreeBSD-equivalent way to declarative build and manage jails.
Gitlab offers free private docker registry with your non-commercial projects and it's fairly easy to build and deploy containers using their CI when hosting the Dockerfiles there.
I would love to Alpine all the things, it's just so fast to work with, BUT musl makes things hard, especially if you're over in Pythonland - packages with C dependencies, if built, are only built for glibc, so when installing them on Alpine, one has to have a compiler, development headers etc. Makes things too slow and error prone.
Otherwise it's a great distro and I use it for non-Pythonic stuff or for Python with no dependencies.
True, but it's still a pain having to do that, all I want is to install a package in an interpreted language, I'm not building a project in a compiled language.
I think they specifically mean python packages with c dependencies that are built for musl. For example, if you "pip install", is it going to have to rebuild completely from source?
Question for those who have migrated to Kubernetes - at what point did you look for something bigger/better than what Docker Compose (or Mesos) can offer?
Once you need more than one machine, more than one engineer, and a desire to use existing tooling.
K8S is a platform to build things. Because of this most of the amazing features you have access to are built by the community (service mesh for example).
Docker Compose is a mess once you have 10+ services with each having different containers backing it.
I don't necessarilly agree with his decision to build all images by hand as opposed to using those available on Dockerhub, however it's a personal choice and I respect it.
Given that he's using docker-compose, I wonder why he's chosen to host his images in a repository at all, instead of just specifying the Dockerfile in the yaml and having them locally.
It's all fun and games using alpine linux until you run into weird networking issues that are caused by slim images.
I was a big fan of slim images until unsolvable bugs started popping up. Like others have said, not much benefit shaving off a few hundred mbs in the age of fiber.
I am in remarkably similar state to this 'prior' state being FreeBSD 11 hosted, with elements of other distributed service.
I also looked to docker and gave up. I like bhyve, and have considered a low-pain migration to bhyve instances to package things into Linux, and then (re)migrate into Docker. A way to avoid pain, and cost of a duplicate site, to build out and integrate.
I wish something as logistically simple as docker compose was in a BSD compatible model, to build packages to. I'd like the functional isolation of moving parts, and the redirection stuff.
Nice write up. I wonder how many other people are in this 'BSD isn't working for me as well' model?
I use FreeBSD for my NAS/utility server at home, and am considering switching over to Linux now that ZFS seems pretty stable there. NixOS is my happy place these days.
I'm half-yes half-no on this. I have successfully carried non-root ZFS partitions into Debian. But, I just lost 15TB to an unexpected multipath/iscsid zpool import so now.. I am unsure how I think this story goes.
Debian ZFS is not easy to install as root FS which is .. disappointing. It would be nice if it was integrated into the net install .iso as a legit disk install option.
I've been using ZoL git on Arch Linux (several systems and configurations) since 2017 without any significant issues. It's probably more stable than mainline Btrfs.
I think the FreeBSD guys did not fully realize how linuxlator would be fundamental a few years back, so they never really fix it up. Which is a shame, because it kinda ruled out the advantage FreeBSD had with jails when the containerization frenzy started.
He finds ansible annoying — anybody suggest a good alternative?
I need to re-spin a set of standard utilities on local hardware from time to time so am looking for the best way to manage the config files (bind, Apache, and the like)
I'm with you. I've been using Ansible for all my work the past 3 years and I haven't seen any thing that it hasn't been able to handle. It's incredibly flexible and if you use the inventory and variable precedence correctly then it can be very deterministic.
I've seen examples of how Ansible will keep recreating instances but that's only if you don't define what your infrastructure should look like in the inventory.
I have lately been thinking an Ansible-like approach but using compiled code, maybe using Go, could be a way to go. So I went looking for that, and found Sup. It was not what I had in mind but worth a closer look.
I have been contemplating adding a small DSL that compiles to Ansible, as i find writing Ansible config files very tedious and boring. As its just YAML, it should be quite easy to generate the output.
sorry, I call the YAML playbooks config files sometimes. but yes, something that can compile down to the YAML files or other tools as well. Abstract away the abstraction basically.
Same here. We should create an efficient compiled version of Ansible, same feature set much faster execution, single configuration file flavor (only yaml)
My idea is different. Instead of playbook yaml, playbook go source, that compiles to a fat binary that is transferred over ssh. That decreases the dependencies to ssh only on the target hosts (no python). The framework would include an idempotent API that matches all the tasks that Ansible provides.
I use a somewhat similar setup, although I kept FreeBSD ala FreeNAS.
I use a FreeNAS server to manage storage pools and the bare metal box, run Ubuntu vms on top of it, and then manage top level applications in Docker in vms via Portainer.
This is nice because the vms get their own ip leases, but can still be controlled and very locked down (or not) depending on their use.
Docker volumes are mounted over NFS from the underlying NAS, and the docker level data is backed up with the rest of the NAS.
There is also another option - Consul/Nomad/Vault/Terraform. I only wasn't able[1] to figure how to setup private Docker registry with Terraform/Nomad. The rest, including GitHub organisation, DNS zones, etc - can be defined as code.
That's an orthogonal option in my opinion. It still allows you to run a full docker setup. It just influences how you run manage everything. That being said, I really like the hashicorp stack and more people should run it.
What about performance hit now that all of the services are running on the same server? Might be interesting to monitor and put resource limits on specific services.
This is pretty classic amateur hour. He basically says 'I now know docker and everything looks like a container! Look at how productive I can be by discarding this set piece environment with out of fashion, slow and deliberate provisioning for docker images that I can create once and run forever with auto provisioning and dutch oven magic + an editor!'
Docker is just setting a few kernel parameters, VMs run a full OS over the host, there's a cost to that. If you don't need the extra flexibility, there's no point.
Mind that you don't need to use Docker to use containers, there's always LXD and others.
Many virtualization technologies use hardware acceleration supported on modern server grade CPUs. It's not like running on "bare metal" but pretty close. One use case for VM's is running obsolete software, like some old proprietary OS, it's a weird feeling when the software run 100x faster then the hardware it used to run on.
VMs and containers have pros and cons. Not sure what do you mean by freedom. If I need secure isolation or non-linux guest I would choose VM, but now I use containers mostly because lightness, speed, registry and snapshots.
This is what existential terror looks like. In the face of something incomprehensible and impersonal, OP decides to spend a while rearranging the deck chairs. Nothing has changed, but perhaps OP felt a little better about himself for a while.
What is the "incomprehensible and impersonal" thing that the OP is facing? I skimmed the post and I have no idea what you are talking about. If I was feeling adventurous, I might claim you are projecting a tad :)
It's fine to spend time learning with personal projects that aren't measured by forward progress on features. Many people rebuild their blogs every year just to try out new things.
Still have to decide on a single OS to reduce maintenance problems. Could just have installed all the services (which are all available as packages) and handled the configuration files instead of configuration files+docker file+s3 costs of docker image with nothing but the base os + one package and a configuration file.
That's not a fair assessment and I'm surprised this is the top comment. In this case, the huge advantage to a containerized setup is that everything is now easily portable. If his server goes down, or he just decides to move, OP can now deploy all of his websites onto another server instantly. He also quotes the ability to build (and test) locally before shipping images to production, which is a really neat workflow. Improved security comes as an added bonus.
As for the "s3 costs of docker image", it's a few cents per month.
> Without adding all that docker specific complexity described in the post:
My guess is that you never used containers at all, let alone Docker.
A Dockerfile is just a setup script that installs the needed services on an image, which you can run on any machine. That's it. There is no added complexity.
This is not about a single Docker file vs a setup script. If you read my post you will see that I describe the steps the author took. And they are plenty.
My guess is that you did not read the article at all.
He was "building Docker images for each of the services". So not a single one. 10 of them. And he signed up for a commercial registry to host them. An additional service he depends on now.
Yet even a single Docker file would not be as simple as a setup script. A setup script on the host OS would install some packages that the host OS will keep up to date. Using a Docker image instead puts the burden on you to keep it up to date.
I agree with this statement that Dockerizing creates more dependencies that you need to track. But...
A setup script on the host OS would install some packages that the host OS will keep up to date.
This is simply not as easy as you make it out to be. Installing dozens of services from the OS, is inherently creating a nest of dependencies which is hard to explicitly reproduce on other systems.
Whereas Docker provides explicit isolated environments for each service so it's far easier to reproduce on other systems. This appeals to me for cloud environments but Docker on the desktop might be a bit too far for me...
Yes, isolation is a big win. It means I can update the “os” each service resides on independently of each other, so I don’t have to tackle 10 upgrades at once.
It also removes attack vectors and weirdness that happens when a package sees optional dependencies on the system. I.e,, if I need ldap for one thing, I don't have services in other containers trying to work with ldap.
Now every time a package in Alpine gets an update you have to update all 10 containers. Because you will have no way of knowing if that package impacts the security of the service running in that container.
Yes, most docker enthusiasts don't do this. They run a bunch of containers full of security holes.
I expect this to become a hot topic as soon as we will start witnessing data breaches that have outdated containers as their source.
> Now every time a package in Alpine gets an update you have to update all 10 containers. Because you will have no way of knowing if that package impacts the security of the service running in that container.
That's pretty much the baseline when dealing with any software system, whether it's a bare metal install of a distro, a distro running on a VM, or software running in a container.
> Now every time a package in Alpine gets an update you have to update all 10 containers.
All it takes is inheriting the latest version of an image and running docker build prior to redeploying.
I mean, this stuff is handled automatically by any CI/CD pipeline.
If you don't carr about running reproducible containers you can also sh into a container and upgrade it yourself.
Do you also complain about package managers such as deb or rpm because most debian and redhat users run a bunch of software full of security holes?
Software updates is not a container issue. It is a software deployment issue. I mean, when you complain about keeping packages updated you are in fact complaining about the OS running on the base image.
That's pretty much the baseline when dealing
with any software system
Exactly. And now instead of one system, he has 11.
All it takes is inheriting the latest
version of an image
He is not using "an image". From the article: "After the Alpine Linux 3.9.1 release I noticed the official Docker images had not been updated so I built my own."
I mean, this stuff is handled
automatically by any CI/CD pipeline.
He has not describe any CI/CD pipeline involved in his infrastructure. Yet another aspect he has to build.
you can also sh into a container
and upgrade it yourself
> He was "building Docker images for each of the services". So not a single one. 10 of them.
10 services, 10 installers, 10 installations.
Where exactly do you see any problem or issue?
> even a single Docker file would not be as simple as a setup script. A setup script on the host OS would install some packages that the host OS will keep up to date. Using a Docker image instead puts the burden on you to keep it up to date.
That's simply wrong on many levels. Yes, a single Dockerfile is as simple (if not simpler) than a setup script. A Dockerfile is a setup script.
And yes, you can update individual containers or even build updated images.
Again, you seem to be commenting on stuff you know nothing about.
> Yes, a single Dockerfile is as simple (if not simpler) than a setup script. A Dockerfile is a setup script.
Sure, but:
a) you have 10 setup scripts rather than 1. This would make sense if you actually wanted to have different dependencies/OS setup/whatever for your 10 services. But if you've decided to standardise on a common baseline set of dependencies for the sake of consistency (which is a valid choice) then why repeat them 10 times over?
b) You have the extra intermediate artifacts of the images which just give you one more thing to get out of date, go wrong, or slow down your process. Rather than run script -> get updated things, it's run script -> generate images and then deploy those images. Sure, it's all automatable, but what's it gaining you for this use case?
If you have a single setup script to build, package and deploy all 10 services, and you can't build and/or deploy each service independently, then you have more important things to worry about than figuring how containers are used in the real world.
Actually, it is, because your criticizing proper deployment strategies, which are not specific to containers, with a usecase that has many technical red flags. You can't simply criticize deployment best practices by giving an blatant anti-pattern as an example. And do note that this has nothing to do with containers at all, because this applies equally well to VM and bare metal deployments.
To have a productive discussion you have to actually engage. If there's really a "blatant anti-pattern" then it shouldn't be so hard to explain what's wrong with it. Your replies so far have been no more substantial than "you're wrong".
You're giving the credit of automation to docker, which isn't where the credit belongs. It's pretty easy to get the same portability and testing without containers (this is what was happening long before docker was launched). Not to say that the OP shouldnt have done it, but I'm kind of tired of seeing the whole portability thing still being put up as if it's only viable with containerised solutions.
To be fair, docker itself introduced no radical new technologies, but it did introduce a lot of convenience. Containers had been available for a long time, but the convenience of a Dockerfile + docker hub made at accessible for the non hard code linux/bsd people.
What other solution for the easy portability do you know? Or how would you propose to handle this?
If it is easier then docker build && docker push and docker pull on the other side I'm all ears!
The main benefit docker introduced, was leading developers to at least consider "configuration injection" and "stateless installs" ("12 factor apps").
If upstream supplies a decent docker image, chances are that means the package is more amenable to scripting and running in a chroot/jail/container - and documents it dependencies somewhat.
That said, snaphotting the "state" of your container/jail can be nice. Recently I used the official ruby images, and could build a custom image to use with our self-hosted gitlab (with built-in docker registry) that a) got a standard Debian based ruby image and applied updates, and b) built freetds for connecting to mssql.
Now I can quickly update that base image as needed, while the CI jobs testing our rails projects "only" need a "bundle install" before running tests.
And the scripts are largely reusable across heterogeneous ruby versions (yes I hope we can get all projects up on a recent ruby..).
> What other solution for the easy portability do you know? Or how would you propose to handle this?
Puppet for the server-automation part. Languages that make it easy to produce a "fat binary" for the isolation part.
Docker solves a real problem for languages where deployment is a mess, like Python. It just grates on me when the same people who spent the last 10 years mocking Java (which does essentially the useful parts of docker) are suddenly enthusiastic about the same kind of solution to the same kind of problem now that it has a trendy name.
> portability thing still being put up as if it's only viable with containerised solutions.
You're arguing a point never made. That containers make things portable is not saying that's the ONLY thing that makes things portable.
I find using containers a lot easier to be portable when I have multiple apps that bizarrely require different versions of, say, python, or python libs and the same version of python.
You get portability by using any provisioning system: Ansible, Puppet, Chef.
Although, it's not exactly the same thing, because with Docker you have everything already installed in the image. I've only used Ansible and I was never happy with its dynamic nature.
You don't get portability from chef and others. You get a framework where you can implement your deployment with a case for each system you want to target. Past some toy examples, it's on you to make it "portable".
This is 100% correct. If you go look at the Chef cookbooks for any popular piece of software, say Apache or MySQL, the code is littered with conditional logic and attributes for different Linux distributions (not even considering entirely different operating systems). Every distro has different packages as dependencies, install locations, configuration file locations, service management, etc.
Docker (all container solutions really) aren't a panacea, but they solve a very real problem.
I was referring to the language you write playbooks in (YAML). There are no static checks, other than a dry-run that only tests for syntax errors. Frankly, I haven't heard of any provisioning system written in a compiled language. I wonder why.
NixOS kind of fits the bill (it can generate complete OS images from a recipe which is IIRC statically typed and "compiled")
If it looks waaaay different to puppet, ansible and chef, there's a reason for that :) Doing provisioning "properly" means managing every file on the drive...
I know about nix, but I'm referring to the language you use to describe the final image. Actually, this part is not a problem. The problem comes in the deployment part.
For example, there's no concept of `Maybe this request failed, you should handle it`. So when you run the deployment script, the request fails and the rest of your deployment process.
Defining the possibility of failure with a type system would force you to handle it in your deployment code and provide a backup solution.
Concerns about server going down or changing cloud provider imo is not particularly interesting or even useful advantage to mention for personal infrastructure. Considering that it's likely we might change our personal infrastructure less than one every year and I've never got a case when an unmaintained docker setup can run 6 months later, I'm not sure if the value for portable is that high.
> Concerns about server going down or changing cloud provider imo is not particularly interesting or even useful advantage to mention for personal infrastructure.
Why? Personal projects aren't more stable or bound to a single provider. If anything, personal projects may benefit more from a deployment strategy that makes it quite trivial to move everything around and automatically restart a service in a way that automatically takes dependencies into account.
> Considering that it's likely we might change our personal infrastructure less than one every year
In my experience, personal projects tend to be more susceptible to infrastructure changes as they are used to experiment stuff.
> and I've never got a case when an unmaintained docker setup can run 6 months later,
The relevant point is that the system is easier to maintain when things go wrong and no one is looking or able to react in a moment's notice. It doesn't matter if you decide to shutdown a service 3 or 4 months after you launch it because that's not the usecase.
> I'm not sure if the value for portable is that high.
That assertion is only valid if you compare Docker and docker-compose with an alternative, which you didn't. When compared with manual deployment there is absolutely no question that Docker is by far a better deployment solution, even if we don't take into account the orchestration funtionalities.
> Concerns about server going down or changing cloud provider imo is not particularly interesting or even useful advantage to mention for personal infrastructure.
I look at this from a different perspective: I have plenty of actual things to do, personal infra should be the least of my concerns and I should be able get them up and running in least amount of time.
> I've never got a case when an unmaintained docker setup can run 6 months later
It really depends on the well-being of the host and the containerized application. I have plenty of containers running for more than a year without a single hiccup.
I've been upgrading my OVH dedicated server once in a year. So far it has been possible to get a little bit better server for the same price from their black Friday sale. Thanks to a simple bootstrap shell script, docker and docker compose I'm able to migrate my ten random services and two KVM VMs in two hours (copying /data directory takes most of the time obviously)
> Concerns about server going down or changing cloud provider imo is not particularly interesting or even useful advantage to mention for personal infrastructure.
I’ll let you know my kids personally disagrees with you on this one if Plex on the TV or iPad suddenly doesn’t work.
Being able to easily migrate apps is super nice to when changing hardware/server.
But he had to migrate away from FreeBSD to use Docker, so that doesn't sound like a portability advantage at all, in the usual sense of the word "portability". "Improved security" is also a red herring. You can't make something more secure by adding an additional layer of abstraction. He even mentions that he found it problematic that inside the Docker container, everything runs as root by default. Plus the dicker daemon must needlessly run as root on the host.
> he found it problematic that inside the Docker container, everything runs as root by default
That's technically right, but not what you'd expect. Docker runs root in a few restricted namespaces and behind seccomp by default. The syscall exploits people are worried about are often simply not available. Even then it's easy to take it another step and create new users. You could even have root mapped to a different user on the outside if you prefer that.
Last I checked, Docker containers are not hard to break out of unless you go to extra lengths (SELinux, AppArmor, etc--not entirely sure; I'm not an expert). Most people use Docker as a way to avoid their programs accessing the wrong version of a shared dependency or similar. I believe there may be other container runtimes with stronger guarantees or better defaults, some of which are even suitable for running untrusted code (or so they advertise).
The advantage is not that huge if you compare to the author's previous setup, which was based on Ansible.
Unless you like to run a different OS per virtual machine, moving your websites to a new machine (or add a new machine to the bunch) is as easy as with Docker, and you can test your setup locally too (though you will need a VM running on your computer).
The biggest advantage of Docker in my opinion is that it makes it much easier to make conflicting software coexist on the same machine (for example two websites requiring two different versions of node.js or php). Also it is nice that you can build the image once and then deploy it many times. Ansible's equivalent is of rebuilding the image every time you want to deploy it.
Also I find it a bit easier to accomplish what I want with a Dockerfile than with an Ansible script and if you make some mistakes it is easier to rebuild an image than "cleanup" a VM instance.
So, Docker smooths many edges compared to Ansible, but I wouldn't consider that a _huge_ advantage, expecially in the context of a personal infrastructure.
Another advantage is that you can run more up to date packages than your distro would allow, or different versions of the same one.
The downside is that you should rebuild the containers daily to be sure to have all the security patches. Not as convenient as apt-get. Maybe it's more cost effective to run another VPS or two.
Yeah but you can also use Ansible or a comparable tool. Moving with that is equally easy. Also without always rising storage usage that results from configuration tweaking, which can be especially problematic if you deploy heavy-weight Java servers.
Sure the containers he can just re-setup, but what about all the DATA.
Where is all the data for his mattermost instances being held? You still have to back that up somehow/somewhere and feed it into your "new containers" that are on another server "instantly"
It's not over-engineering; it's quite a lot easier to get Docker up and running and run things in it rather than dealing with init systems, package managers (and dependency conflicts), ansible, etc for every app. You get a sane, consistent, and standard interface to logging, networking, data management (volumes), declarative infrastructure, packaging, etc and Docker swarm makes it relatively simple to scale up to multiple nodes.
I would disagree with this even if I had a really nice cloud operator with great interfaces and utilities for logging, networking, monitoring, image management, volume management, secrets management, declarative infrastructure, etc and you can afford to run that many VMs... I'd still probably be running lots of cruft (SSH servers, process managers, etc) in each VM (or I'd have to go through the trouble of opting out) and I still need to get logs out of the VM and into the logging service, which usually implies faffing with ansible and friends. Nooooo thank you.
Also, `docker commit` is pretty easy, and you can also just back up individual volumes.
I disagree -- with traditional VMs, you have to deal with multiple mutable systems. In the Docker/OCI container world, containers are immutable, so you can manage all your changes atomically, from a single source of truth (your Dockerfile collection).
In my view, LXD/LXC splits the difference pretty nicely between VMs and Docker.
Portability with LXD is even cleaner as all the data is in the lxc container. It's not immutable, and the initial setup is a little more involved as you have to set up services on each container, eg no dockerfiles, and you need to figure out ingress on the host often less declaratively, with normally routing 80/443 via iptables to an nginx or haproxy container to then reverse proxy to the relevant container per domain-based ACLs.... etc.
But, I still prefer it to Docker. I rather don't mind initially setting up and configuring the needed services the first time on each container... And for me that's a good way to get familiar with unfamiliar packages and services/applications that I might want to try/play with, rather than just go find a dockerfile for Application X or Y and not actually learn that much about the service or application I am deploying. Speaking for myself only-- obviously there are the gurus who know common services and applications inside and out already, and can configure them blindfolded, so Dockerfile everything would make sense for them.
Fully agree and pretty much exactly my setup. A haproxy container which directs traffic (not only websites, but also syncthing, caldav/carddav etc.) and renews all Let's Encrypt certificates.
It's fun, easy to backup, easy to migrate, easy to just test something and cleanly throw it away. And in practice the containers are pretty much like VMs (talking about personal projects here, corporate is more complicated of course).
And the upfront work is not that much. Do the quick start guide and one or two things. Maybe you don't even need to configure iptables manually, "lxc config device add haproxy myport80 proxy listen=tcp:0.0.0.0:80 connect=tcp:localhost:80" does a lot for you.
Just like how most programs are available as packages in your favourite distribution's package manager, most programs are available as Docker images on Docker hub, which is as gratis as packages built for distros. And for the ones that don't have an image on Docker hub, you can include the Dockerfile to build it on the spot in the production environment.
The real benefit comes from having the _ability_ to use a different base distribution if required. While alpine is suitable for most applications, sometimes you might be required to run a different distro; perhaps you're packaging a proprietary application compiled for Glibc.
Also, networking isolation is straightforward in Docker. Let's say there's a serious security bug in Postgres that allows you to log in without any password. If someone could perform a RCE in a public-facing website, this would be a catastrophe as they'd be able to exploit the database server as well. With Docker, you can easily put each service in its own network and only give access to the database network to those services that need it. Or you can be even more paranoid and have a separate database and a separate network for each service.
One more feature I'm fan of is the ability to run multiple versions of a software. Maybe some applications require Postgres 9.x and some require Postgres 10.x. No problem, run both in separate containers. You can't do that with regular distributions and package managers (at least in any I know of).
Nix is... tough. I really want to like it, but every time I pick it up I end up having to write some custom package definition for some obscure transitive C dependency with its own snowflake build system. Couple that with poor documentation for existing packages, the terrible search engine experience ("Nix" and "Nix packages" usually turn up things about Unix or "Congress Nixes Aid Package"), and a thousand other papercuts and it just seems to create more problems than it solves. I dearly hope this changes.
Guix is like Nix, but I don't think Nix has an equivalent command to bundle a bunch of packages into a self-contained file for running on arbitrary distros (without Nix/Guix installed).
That's a bit like saying "ORMs are too complex" before proceeding to eventually build your own crappy ORM. I think by accepting docker you gain a lot of support from a developer community which you'd otherwise not see if you assembled your own.
Before proceeding to eventually learn the modern SQL finally and write a couple of stored procedures to stop pumping terabytes of almost-raw data to your "application server" code to be filtered by ORM that is too complex to understand, and in the same time still too stupid to use half of the features your DBMS provides.
You're complaining about a scenario where you might have picked an ORM that didn't met your personal performance requirements. Even if you ignore that you are free to pick any ORM you choose and even if you assume that you have all the time in the world to refactor your project to an experimental setup that was never tested at all, your comment still sounds like a revamp of the old argument on how hand-rolled assembly is always superior to any compiled or interpreted code.
Anyone building out a setup based on cgroups and namespaces will eventually arrive at a poorly specified, bug ridden mini Docker. Might as well get with the program early.
Anyone calling bc from bash script will eventually arrive at a poorly specified, bug ridden mini Mathematica?
Wrt "bug ridden"
> if I were having trouble with something Docker-related I would honestly feel like there was a 50/50 chance between it being my fault or a Docker bug/limitation
Sounds like you don't understand Docker or containerisation in general
> Still have to decide on a single OS to reduce maintenance problems.
No you don't, you can run various distros in docker containers. We're using a mix of Debian (to run legacy services developed to run on old Debian LAMP servers) and Alpine (for our sexy new microservices) at my current job.
> Could just have installed all the services (which are all available as packages) and handled the configuration files
Then you would have a system dependent on the volatile state you configured by hand, meaning the system configuration is not declarative or reproducible.
> No you don't, you can run various distros in docker containers. We're using a mix...
I think you are missing the point. For each distro base you introduce via docker you must track and update the security releases. Standardizing on one base distro absolutely reduces the ongoing maintenance work.
> Then you would have a system dependent on the volatile state you configured by hand, meaning the system configuration is not declarative or reproducible.
If you're using Ansible as the author already was, you essentially have this already. I can do a one command deploy to cloud servers, dedicated servers and colo boxes with just Ansible. Docker gets you a slightly more guaranteed environment and an additional layer of abstraction (which has its own set of pros and cons), but that's about it.
Slightly more guaranteed is something of an understatement; if you specify base images with specific hashes and pin package versions, you can get quite close to reproducible builds of the environment.
In support of your argument; Look for example at the Dockerfile for the official Golang container. They pin exact sha256 hashes for each architecture, and the source release in case you're on an un-binary-released architecture.
Pin specific versions of your packages, coupled with caching and you're sitting pretty.
Yes, but you still can't guarantee anything about the host the docker container has to run on, so you're still impacted by host configuration and therefore still need to provide a good base environment for running your containers. This is fairly simple with most providers, but in such a case, using Ansible or similar to deploy directly to the host has similar results.
The problem that docker solve for me is reinstalling.
Some software, like RDBMS, are heavy to setup. You can do once and is not that hard, but then when the day come to reinstall, upgrade or move it to another version:
- You need to remember how do that
- If you upgrade OS, the step have changed
- If the install (of the OS) go kaput, you can lose the work of setup everything so far -the steps- (ie: I have a few times manage to ruin some ubuntu install. Instead of wasting time fixing it, I spin another server and redo)
- If the install (of the app) go kaput, you are left with a inconsistent state (I manage to broke a PG database in a upgrade, because was looking to the wrong tutorial, Instead of wasting time fixing it, I spin another docker image and redo from backup)
- I try to upgrade everything fast. A new ubuntu version? upgrade. A new major PG version? upgrade. Not always to production but I don't like to wake up 2 years late and face a BIG upgrading cycle. prefer to spread the pain in the year. If something wanna break, I want to see it coming. So I reinstall a lot of times.
And it work across OS. So all the problems of above, was in my dev machine!
P.D: I wanna something alike docker, I don't care for orchestration (so far) only something that allow to package apps/dependencies, work in my small serves, allow to (re)build as fast as possible. What other option can work? (P.D: My main deps are postgress, nginx, redis, python3, .net core, rust (coming). Wish to setup android/ios toolchains too)
> s3 costs of docker image with nothing but the base os
An alpine container is in the middle double digit MB range and will be reused in all other images using it. The space costs are trivial. Furthermore the base image is hosted on Dockerhub and last I checked you can host up to 3 images privately on Dockerhub and unlimited in public.
Clearly you don't understand the advantage of containers. You can swap them without causing problems to other services, in contrast with traditional installation on a single OS.
It is a surprise, since he was already a FreeBSD user, that he did not use its facilities like for example jails + perhaps a program to manage them or bhyve for VMs.
Which features Jails are missing? You have to orchestrate Docker just as you have to orchestrate Jails, there is a reason why Kubernetes is on the rise.
He's taken the first level step of containerization. What he gets is a deterministic way to build his environment.
You could just install the services and handle the configuration files, but then you suffer from replicability and deterministic build problems:
- The configuration files are all over the place, and while you can remember most of them, you can't be 100% sure that you got everything in /var/lib/blahblah/.config/something.conf, /etc/defaults/blahblah, /etc/someotherthingd/blahblah.conf, etc.
- Rebuilding the server becomes a problem because you don't remember a year and a half later everything you did to set the damn thing up. Even worse, you've been tweaking your config over time, installing other packages that you can't remember, putting in custom scripts for various things...
- Recovering from a catastrophic failure is even worse, because now you don't even have the old configuration to puzzle through.
- If you set up another failover system, you can't be sure it's configured 100% the same as the old one. And even if you did get them 100% the same, they WILL drift over time as you forget to migrate changes from one to the other.
You can mitigate this by using ansible or chef or docker or the like.
The other handy thing with container tech is that you can tear down and rebuild without having to wipe your hard drive and spend 30 minutes reinstalling the OS from the iso. Iterating your builds until you have the perfect setup becomes a lot more appealing when your turnaround time is 2-5 minutes.
Damage control becomes a lot easier. A bug in one program doesn't damage something in another container.
After this level, you can move up to the orchestration layer, where each container is just a component, and you tell your orchestration layer to "launch two of these with failover and the following address, link them to instances of postgresql, do an nginx frontend with this cache and config, pointing to this NAS for data", and it just works.
It's a lot like programming languages. You COULD do it all in assembly, but C is nicer. You COULD do it all in C, but for most people, a gc functional language with lambdas and coroutines etc gives you the sweet spot between cognitive load and performance.
With a shared language for doing these powerful things, you now gain the ability to easily share with others (like docker hub), increasing everyone's power exponentially. Yes, you need to vet whose components you are using, but that's far less work than building the world yourself.
Yes, there's a learning curve, but the payoff over time from the resulting power of expression is huge.
I blacklist Vultr/choopa in every environment I manage. They make zero effort to deal with botnets and bad actors. They are the first org where I went to the trouble of blacklisting the ASN so new ranges get blocked as soon as maxmind updates.
Everything else matches patterns we use. WebPageTest is probably the most hilariously janky application that’s the best at what it does that we use. Standing it up locally so you can test internal stuff is a revolting experience I’ve had several times.
TBH I found kubernetes easier to use than docker compose. Mainly because I saw little reason to learn the syntax when I was already using kubernetes yamls and kubeadm makes it so easy to stand up. What you have looks pretty simple though so I may take another look at it.
You can actually ship images as tarballs and reimport them. That’s what I do on personal stuff instead of standing up an ECR/registry. As long as you version tag your containers it should be fine.
HTTP/2 is pretty much a nonissue for everyone except some janky java clients at this point. We turned it on a couple of years ago with only minimal issues.
Maxmind (and others probably) regularly publish lists of IP blocks mapped to the owning ASN.
Basically geoip but instead of mapping to geographical regions, maps it to ASN. If you want to roll your own, you can probably start from a public route reflector and go from there.
This, fastly makes this easy to do as well (I don't work for them, just use the product).
Maxmind dbs are... awkward to get access to. They are expensive as hell if you buy them directly but usually built into edge service products. F5 had them as well I think.
The real sauce in my infrastructure is simply 'generational' storage. No more clean installs with a backup dir of the system. All my data volumes get mounted into my machine/s as needed. I can swap out all my hard drives with new ones without any downtime too which is nice. I could reinstall my desktop each morning and it would barely set me back 15 minutes. If I also properly stored my general system settings as code (cfengine, puppet, chef, etc) I could practically just launch ephemeral desktop instances and have basically a native experience.
For me a long time ago an upgrade meant buy new hard drives, do a reinstall, and copy either the entire old drive or main data dirs onto the computer.
Now my data volumes are separate from the system. I used raid for redundancy on the physical machine with another drive for differential filesystem level backups (snapshot every day for 60 days with only differential storage cost, see rdiff-backup). When I upgrade I just add new drives, assign them into the raid, wait for them to sync, and remove the old drives. By generational I mean my storage and how I do things aren't going to change when I reset things like they used to. All my data is outside the system I'm using and must be attached.
I can for example, spin up an AWS machine, run the package install commands for the apps I normally use, VPN to my fileserver and mount my home directory and project volumes, and in quite literally about 15-30 minutes have the exact same environment as I do at my house. CFEngine et al would speed that up quite a bit.
You are welcome. I'd have to say the biggest benefit for me personally is the distinct separation and permanence leads me to keep it very organized. When I had my files 'in' my system the lines blurred and things got spread all over the place. I still keep my download dir on the base system and only things i really care about make it to the data drive so there isn't much pollution. I'm a bit scatterbrained in that regard and this method accidentally solved that.
Software raid with LVM. For my primary storage pool I created a raid1 pair, encrypted with an NVME cache drive, then created an LVM PV nested on that, LVM within LVM to dish out volumes that all share those lower level benefits.
And after all that, RedHat dumps Docker and Docker adopts Kubernetes. He should have stuck with the stability of FreeBSD--which he prefers--and used jails.
Jails are just the runtime mechanism, equivalent to cgroups/namespaces/etc in Linux. What Docker provides is the tooling for repeatable container setups, so you can have a single configuration file that you can deploy locally for testing and then on the server.
It can be quite complicated if you use a docker version without containerd's CRI. And you do if you follow the version recommendations (because docker has a lot of regressions). GKE does it, so we do it.
Kubelet actually has a translation layer baked into it that it starts in-process when detecting docker, which provides the gRPC CRI interface on a real filesystem socket.
Also, I'm not sure what are the benefits going `FROM alpine` and installing nginx, than just starting `FROM nginx:alpine`. The latter has benefit of a more straightforward update logic when a new nginx version is released - `docker build` will "just" detect this. It won't notice that Alpine repos have an upgrade, though, and will reuse cached layers for `RUN apk install nginx`.
Just saying.