We’re currently consolidating all container images to run on Debian-slim instead of a mixture of Debian, Ubuntu and alpine. Sure, alpine is small, but with 70% of our 500 container images being used for Python, R or node the final image is so large (due to libraries/packages), that the difference between alpine (~30 MB) and debian-slim (~80) is negligible. We’ve been experiencing the weird DNS behaviour of alpine and other issues with musl as well. Debian is rock solid and upgrading to bookworm from bullseye and even buster in many cases didn’t cause any problems at all.
I will admit though, that Debian-slim still has some non-essential stuff that usually isn’t needed at runtime, a shell is still neat for debugging or local development. This trade off could be considered a security risk, but it’s rather simple to restrict other stuff at runtime (such as run as non-privileged, non-root user with all capabilities dropped and a read-only file system except for /tmp).
It’s a balancing act between ease-of-use and security. I don’t think I’d get popular with the developers by forcing them to use “FROM scratch” and let them figure out exactly what their application needs at runtime and what stuff to copy over from a previous build stage.
My biggest beef with the apt/deb-based distros for container images is that apt-get updates installs take frustratingly long, whereas apks always tend to be near instant. I wonder what it is in the postinst and related scripts that just take so long and can't be parallelized!
Most of the reason I switched from Ubuntu -> Arch is that working with alpine allowed me to realize that installing packages and their dependencies don't have to take so long.
Have you tried building the images with eatmydata library?
Essentially, dpkg is hilariously slow due to its approach in issuing fsync calls for every file - eatmydata wrapper injects a shared library which turns that into a no-op.
I haven't timed it rigorously, but I experienced up to 10x speedup in container builds by just wrapping dpkg with eatmydata.
[Edit: modern debian images have “force-unsafe-io” turned on in /etc/dpkg/ so eatmydata shouldn’t make a difference.]
I’m not the OP, but in an example I just tried it took build-essential from 48s to 39s (timed on the third step, without and then with the call to eatmydata):
from debian:sid
run apt update && env DEBIAN_FRONTEND=noninteractive apt-get install --yes eatmydata
run apt update && eatmydata env DEBIAN_FRONTEND=noninteractive apt-get install --yes build-essential
Can you replicate something faster? Was I doing something wrong?
Yes, the second invocation will have hot caches. Basically the standard is to either test in a clean environment (e.g. immediately after a reboot) or to run the command until the times stabilize, and then take, say, the middle 3 out of 5 measurements.
What is cached though? DNS? IP routes to the Debian mirrors?
I ran it three more times. Plain install was 50s 48s 47s; eatmydata was 46s 46s 46s.
Installing build-essential probably does enough non-dpkg IO to account for the difference (install scripts etc, for example when libc changes and the system scans for running daemons) but it isn’t much and the delta may well be statistically insignificant.
This here. Honestly most orgs with uhh.. Let's say a more mature sense of ROI tradeoffs were doing this from pretty much the very beginning.
Also, Ubuntu 22.04 is only 28.17MB compressed right now so it looks equiv to debian-slim. There are also these new image lines, I can't recall the funky name for them, that are even smaller.
I'm pushing to go back to Debian from Ubuntu. Canonical's making decisions lately that don't appeal to me, and especially on the server I don't see a clear advantage for Ubuntu vs good ol' rock solid Debian.
I've done this multiple times when new Debian Stable was released (Bookworm just got released). Its great, until it isn't (ie. you need more up-to-date software). Although to be fair, you might be comparing Ubuntu LTS with Debian releases and the minor Ubuntu releases get short support. Ubuntu is akin to a polished Debian Testing snapshot. I don't like them swapping to Snap either (snappy it ain't).
Yes, that's it! I've been working with .Net a lot recently and recalled Microsoft's announcement about them. Specialized runtime images for JRE, .NEt, Asp.Net, and etc.
Fewer features (including less "we've supported doing that for 20 years and we're not cutting it now"), packages separated into parts (ex. where debian might separate "foo" into "foo" for the main package and "foo-dev" for headers, alpine will also break out "foo-doc" with manpages and such), general emphasis on being small rather than full-featured.
In the same boat here as well. Especially when you’re talking about container images using JavaScript or other interpreted languages that are bundling in a bunch of other dependencies, the developer experience is much better in my experience given that more developers are likely to have had experience working in a Debian based distro than an Alpine based one.
Especially when you’re also developing within the container as well, having that be unified is absolutely worth the convenience, and honestly security and reliability as well. I realize that a container with less installed on it is inherently more secure, but if the only people who are familiar with the system are a small infrastructure/platform/ops type of team, things are more likely to get missed.
Can you point me on where to look for more details on securing a container? I'm a developer myself, and for me, the main benefit of containers is being able to deploy the app myself easily because I can bundle all the dependencies.
What would you suggest I restrict at runtime and can you point me to a tutorial or an article where I can go have a deeper read on how to do it?
What you want to read is the kubernetes pod security context fields[1].
In your Dockerfile, add a non-root user with UID and GID 1000, then in the end of your Dockerfile, right before a CMD or ENTRYPOINT you change to that user.
In your kubernetes yaml manifest, you can now set runAsNonRoot to true and runAsUser & runAsGroup to 1000.
Then there’s the privileged and allowPrivilegeEscalation fields, these can nearly always be set to false unless you actually need the extra privileges (such as using a GPU) on a shared node.
Then there’s seccomp profiles and the system capabilities. If you can run your container as the non-root user you’ve created, and you don’t need the extra privileges then these can safely also be set to the most restricted. Non-privileged non-root is the same as all capabilities are dropped.
The tricky one is the readOnlyRootfilesystem field. This includes /tmp, which is considered a global writeable directory, so the workaround is to make a in-memory volume and mount it at /tmp to make it writable. Likewise, your $HOME/.cache and $HOME/.local directories (for the user you created in your Dockerfile) are usually used by third party packages, so creating mounts here can be useful as well (if for some reason you can’t point it to /tmp instead).
> In your Dockerfile, add a non-root user with UID and GID 1000, then in the end of your Dockerfile, right before a CMD or ENTRYPOINT you change to that user.
I'm not certain, but would it be more secure to create a non-root user in the Dockerfile, but _don't_ use UID 1000. Use volume mounts to grant host disk access instead.
If you want to use the runAsUser and runAsGroup options in kubernetes that user needs to exist, hence why creating it is needed.
In our case, we have our own base container images that include this user. The base images are either just plain copies of official ones (such as dotnet or curl) or also includes stuff that almost everyone needs (such as MSSQL ODBC drivers, libxml2, curl etc) in the case of python/R/node.
Basically you need to work as unprivileged user and with immutable file system (of course you can have ephemeral /tmp or persistent /data, but generally the entire system should be treated as read-only).
You might want to check out Wolfi and Chainguard Images. Wolfi is a Linux distro that we use to build minimal images that are roughly comparable to Alpine in size but, everything is compiled from source against glibc.
Our images come without a shell or pacakge manager by default, but there are -dev variants that include these.
I was using wolfi for an oss project, but the recent attempts at forcing payments when pinning version was a huge red flag, as was the messaging around it, and I’ll be migrating away from it.
Pinning a language version (say python 3.11) isn’t an optional thing its a best practice, and the notion that its because of security seems intentionally misleading as the images should be refreshed in place on the tag along with signatures.
To be clear, nothing has changed with Wolfi. Wolfi is an open source community project and everything is still available there: https://github.com/wolfi-dev/.
We have made changes to Chainguard Images - our commercial product built on top of Wolfi - which mean you can no longer pull images by tag (other than latest). Chainguard images are rebuilt everyday and have a not inconsiderable maintenance cost (and the money we make here directly helps us support Wolfi).
I agree that pinning is a best practice. The above blog explains that you can still do it using a digest, but I accept this isn't the simplest solution.
If I can help any more, please feel free to get in touch - you can find me most places including twitter https://twitter.com/adrianmouat
The built Chainguard images are all on the cgr.dev registry.
Wolfi is completely OSS.
The policies regarding Chainguard Images have changed over time, so if there are docs that don't properly reflect this, please let me know and I'll get them updated.
let me rephrase that, wolfi is an oss container image distro without an image, you get to pay us for the image. but here's a our tooling cause we like to call it oss, have fun building. either have those oss images on a non commercial registry, or let's stop misleading folks about what this is, aka pay to play/use oci images. that all wolfi docs / blog posts reference a commercial registry while touting it as oss, is the definition of bait and switch.
Exactly. Part of the appeal to consolidate all of our container images to use Debian-slim is the ability to optimise the caching of layers, both in our container registry but also on our kubernetes cluster’s nodes (which can be done in a consistent manner with kube-fledged[1]).
And even if it did, in an ancient data center that only uses gigabit ethernet, that's only a .5s longer download. And even a $4 DigitalOcean server comes with 10GB of storage, so that 50MB is only 1/200th of the instance's store. (I'd also bet that nearly no one uses instances that tiny for durable production work where 50MB is going to make a difference.)
I've run into the same thing for large dev images, but using pure rust often means that musl allows for a single executable and a config file in a from scratch container for deployment. In cases where a slab or bump allocator are used, musl's deficiencies seem minimized.
That means duplication of musl in lots of containers, but when they are all less than 10MB its less of an issue. Linking against gnu libraries might get the same executable down to less than 2MB but you'll add that back and more for even the tiniest gnu nase images.
By pure, I just meant no dependencies on c or c++ bindings other than libc. If that is the case you can do a musl build that has no dynamic dependencies, as all rust dependencies are static. So then your only dependency is the kernel, which is provided via podman/docker. A decent sized rust program with hundreds of dependencies I can get to compile down to 1.5MB. But that is depending on gnu. So if you had 4 or 5 of those on a node, it might be less data to use one gnu base image that is really small like rhel micro, and build rust for gnu. But if you have cpu hungry services like I do, then you usually have only a couple per node, so from scratch musl can be a bit smaller.
Very few system calls in the main loop. Not even close to being io bound, and as I said memory is all preallocated, so allocator efficiency isn't a factor.
Edit: realized I didn't answer your question, I ran flamegraph both ways and it was completely dominated by tight vectorized loops and a couple sadly necessary memory copies.
I made the switch too around 4ish years ago. It has worked out nicely and I have no intention on moving away from Debian Slim. Everything "just works" and you get to re-use any previous Debian knowledge you may have picked up before using Docker.
R is kinda difficult and I haven’t cracked this one. Currently we’re using the rocker based ones[1] but they are based on Ubuntu and include a lot of stuff we don’t need at runtime. I’ll look into creating a more minimal R base images that’s based on Debian-slim.
This has been an issue plaguing Alpine for years where the musl maintainer basically said the standard says may fallback, not must fallback. Let the rest of the internet change, we don't feel this is important. We're standards complainant.
It gained traction for the change with the latest RFC. for dns last year which made TCP fallback mandatory [0]. The cloak of standards compliant could no longer be used.
Once I started reading about these issues a few years ago, I stopped using Alpine as a container base image, and started using Debian (the 'debian-slim' variant). Slim is still larger than Alpine, but not by a lot, and does contain some extra functionality in the base image that's useful for debugging (most of which can be fairly easily removed for security hardening). Debugging random DNS issues is difficult enough; there's no need to make it harder by using intentionally-faulty software.
While I wouldn't call myself a fan of Postel's Law (I think "being liberal with what you accept" can allow others to get away with sloppy implementations over long time periods, diluting the usefulness of standards and specifications), I think at some point you have to recognize the reality of how things are implemented in the real world, and that refusing to conform to the de-facto standard hurts your users.
The fact that the maintainer only caved because the TCP fallback behavior is finally being made mandatory, and not because he's (very belatedly) recognizing he's harming his users with his stubbornness, also speaks volumes... and not in a good way.
A gentle reminder that you're not entitled to other people's free labor. It's one thing to disagree with technical decisions made by the project, but the shaming and name calling is a bit too much.
This is a ridiculous take. Nothing in the parent comment even remotely hints at feeling entitled to the labor of the Musl maintainers — they just disagree with implementation decisions that have been made.
And while I’d agree in principle that nobody is entitled to anyone else’s labor for free, I’d also add that nobody is entitled to be shielded from criticism of work they have chosen to publish, or to have that criticism be watered down to a sanitized, milquetoast version of the writer’s intended message.
This shouldn't have to be spelled out loud, but you don't ever need to misrepresent the maintainer's position to criticize a technical decision. "They're harming everyone by intentionally shipping crippled software!" Really?
One could say the same thing about your previous post, falsely claiming the person you replied to felt “entitled to someone’s labor,” when they said no such thing.
Now, that is a ridiculous take. By your definition, entitled people have to describe themselves as such? The rest of us use "entitlement" as a term to describe the action of others. People aren't making "false claims" when they do that.
Resorting to personal attacks because someone refused to implement and maintain your pet feature for free fits the description of "entitlement" in my book.
See "strawman fallacy".
People absolutely are making false claims, including about entitlement. Your imagining someone to be entitled does not make them so, and your accusation belittles their actual, valid reasoning. It actually shows you to be the one "entitled" to your projected redeclaration of their identity and rationale - implying the accuser (you) is more likely to have narcissistic tendencies than the person holding a potentially shared value-based opinion about the facts being criticised.
Furthermore, it's about irrellevant as calling out someone for being "entitled to breathe air" if you catch them criticizing the air's quality.
> See "strawman fallacy". People absolutely are making false claims, including about entitlement.
You certainly demonstrated the fallacy. You took the most absurd interpretation of my words that is possible in a literal sense which happens to be easier for you to mock.
Describing someone's behavior as entitled doesn't amount to making a false claim. That's the most obvious interpretation of what I actually said. But you took it to mean that any claim of entitlement can't be false.
> Your imagining someone to be entitled does not make them so, and your accusation belittles their actual, valid reasoning.
There's no imagination or accusation involved here when the very comment I'm referring to is right here in this very thread. It's a statement of fact.
The parts I quoted are ad hominem attacks that accuses someone of neglecting an obligation that objectively doesn't exist. That's the dictionary definition of entitlement.
> It actually shows you to be the one "entitled" to your projected redeclaration of their identity and rationale
Like misrepresenting my words or calling me "narcissistic?"
> potentially shared value-based opinion about the facts being criticised
The parts I quoted isn't a fact-based criticism. It's an ad hominem attack.
> irrellevant as calling out someone for being "entitled to breathe air" if you catch them criticizing the air's quality
Everyone is entitled access to clean air. People have an obligation to keep the air clean.
No one is entitled access to free labor, including having your pet features implemented and maintained in a software project.
But overall, gosh. How many fallacies, misrepresentations, and personal insults can you pack in such a short comment?
is strong language and not productive. If someone at my company sincerely criticized my work in this way I'd find it unacceptable and managers would get involved. There's criticism and there's polemics, and this is the latter.
Fair, but I'd hold HN to a higher standard. We're peers and we choose to be here. We should treat each other with kindness and respect, not contempt. I read a fair amount of contempt in OP's characterization, and I hold that separate from respectful criticism.
I looked up a thread about this from 2020 [0] and it actually seems well reasoned, calm, and respectful. No one's slinging around words like "faulty", "broken", or "harm" because those tend to insult people and derail technical discussions.
And FWIW, that's why I would get managers involved if this happened in the workplace. Calling people's work "broken by design" or "harmful" is how to start an argument, not how to solve a problem. If someone's acting this way, there's no way things are going well.
Now, that is a ridiculous take. By your definition, censorious people have to describe themselves as such? The rest of us use "censor" as a term to describe the action of others.
No, to censor means to remove speech and I have never defined it any other way. Calling out abuse for what it is hardly fits that definition.
BTW go take my words verbatim and use it against me if my actions are genuinely hypocritical. But to do that for my actions you made up is abusive and makes you look bad.
we are not entitled to it, but it was the sole reason why our team (and it looks like others did too) stopped using alpine and i believe that should be valid feedback to the maintainer.
Yikes. The first time I ran into an issue where email delivery for a major provider was impossible without TCP fallback was 23 years ago. To treat this as optional this long is ridiculous.
For all the good things Musl does, its authors can often be downright ridiculous out of a dogmatism which is very unpleasant to deal with as a user. See also their stubborn refusal to give you an easy way to detect that the libc is Musl while compiling.
Well, if musl is standards compliant, then wouldn't it make sense to detect when you're building on something different rather than musl? Like, if the standard didn't require TCP fallback, then in theory the application should handle that and only skip it if it detects itself being built on a libc with non-standardized extensions to obviate the effort.
Standards are not laws, they are reasonably well-written documents describing behavior that is supposed to be interoperable. If, for some reason, that is not true (perhaps because the standard is ambiguous, obsolete or some other reason), ensuring that there is interoperability is more important than just keeping it to the letter of the standard.
Yes, it has to work. That’s exactly the point the person you’re responding to is making. There are plenty of things libc doesn’t provide. Software authors find way to provide functionality to make their software work regardless. How is DNS different? Just because you’re used to one specific implementation?
Because as we all know the standard never changes potentially introducing new features, allows no variability and bugs don’t exist.
It must be nice living in your. theoretical world. Here in the real world, I like libraries to actually help me do what I want to do rather than be needlessly annoying to take a stand no one actually cares about. But that’s me. Clearly it serves Musl right given the massive amount of use it’s seeing. Hmm, wait a minute…
Well I would like to use it. It’s code is mostly sane, easier to understand than glibc and it would make sense in a lot of places where performances are not critical. That’s why I am so annoyed by the non sense its developers keep pulling out. It’s an incomprehensible situation because they get absolutely nothing out of it. They are basically annoying people for the sake of it. It’s incredible that changes like the one we are commenting on which are just sanity being restored have to come from an evolution of the standard.
> See also their stubborn refusal to give you an easy way to detect that the libc is Musl while compiling.
I suspect that the primary use cases for that would be applying half-baked workarounds and refusing to work with an "unsupported" libc, much like HTTP User-Agent.
Or, you know, taking into account the stupid things Musl does like not having TCP DNS fallback.
I’m only half joking. Features detection in Musl is a generally pain. I don’t blame developers for sometimes taking the path of least resistance and not trying to support it.
Musl also includes res_send() etc., which can be used for MX records, and e.g. Qmail did use that back then. Here's the commit adding TCP support to those functions in Musl.
AOL was the specific case we ran into that at the time (ca. 2000) returned either MX or A records (can't remember which caused the problem) that required more than 512 bytes.
The point wasn't that this would have helped Qmail - Musl didn't exist back then for that matter -, but that running into the 512 byte limit and needing TCP fallback was already a thing. Bumping a buffer size is far less effort than having to replace the resolver.
gethostbyname() only returns an address, it can't be used to e.g. query for MX records. You also lose control over retries to different addresses if you use gethostbyname(), which some mail server software will also care about.
However as I noted in my other reply, musl also implements the res_* functions, like res_send() etc., and those can be used to address both.
> This has been an issue plaguing Alpine for years where the musl maintainer basically said the standard says may fallback, not must fallback. Let the rest of the internet change, we don't feel this is important. We're standards complainant.
I've never really understood the underlying mentality that causes maintainers to bend over backwards to _not_ provide popular functionality like this.
Reminds me a bit of the strlcpy/strlcat debacle with glibc.
In this case it's not so much "popular functionality" as "core mechanism of the protocol". TCP DNS isn't a quality of life feature; it's necessary in order to look up record sets with e.g. lots of IPv6 addresses, because UDP DNS has a sharply limited response size.
I'm sure they have their reasons. Some might even be good. Here's something I know I want: for my resolver to resolve valid DNS queries, no matter how stupid I believe the remote zone's maintainers to be.
Yes. And I think anyone who thinks 10 IP6 addresses in a DNS response is useful for anything is making a huge mistake, let alone cramming 20+ on a long domain name to smash the response limit, but I am fully prepared to be convinced I was wrong if you had a compelling use-case.
Once upon a time, the DNS spec required requests come from port 53. This was terribly insecure, so many sysadmins patched their DNS server to not do this-- most servers at the time didn't care (including the incumbent), and the few that did were easy to convince of the error of their ways with a simple demonstration.
Then one day, a DNS server became quite popular that didn't do this by default, and perhaps because the DNS spec was written by a company that sold consulting for their (buggy) DNS server, they used this exact argument: that it was wrong because the spec said it was wrong. They patched their server to reject these messages and encouraged users to block this traffic.
That was wrong of them, and eventually, due to enough pressure they eventually caved. That's because the reality of good engineering doesn't give two fucks what the spec says.
Client side load balancing across a universe of 20+ disparate nodes sounds unnecessary. Why is 20+ better than just giving out 5 random members of the universe for your use-case?
Is this HTTP/S? SNMP? Something bespoke? Why doesn't the client already know where all the servers are? How far away are the servers from the client? Can you put code that lives in the client, or is this existing client software you're trying to trick into supporting client-side load balancing?
Also, and only slightly related: Do you have a different definition of "use-case" than me that involves being as vague as you possibly can?
> I've never really understood the underlying mentality that causes maintainers to bend over backwards to _not_ provide popular functionality like this.
Probably trying to prevent feature creep; musl's home page describes it as
> musl is lightweight, fast, simple, free, and strives to be correct in the sense of standards-conformance and safety
And every single feature they add makes those things harder.
Okay, then they haven't been standards-compliant for a very long time, which is a much more compelling point than arguing that they should be doing things beyond the standard.
The reason the DNS TCP requirements RFCs were written was because of boneheads who don’t believe people who tell them it has been a non-optional part of the protocol since the 1980s
"It's a catastrophe, but of little importance" -- Zizek
"RFC" is Request For Comments. Some RFCs become standards. But there are no internet police, notwithstanding that the internet is a fascist state (run by corporations for the benefit of corporations). Understanding it as a fascist state is the rational way to view "enforcement" as well as the natterings over whether something is a standard and who is going to enforce it anyway.
Given the choice between maintaining a fork of a C standard library implementation and switching to an implementation that doesn't have this issue, the choice is pretty clear.
It was pretty clear that musl was unwilling to support adding any TCP fallback code path regardless of a patch existing.
Anyways, your comment is inflammatory and is full of straw-men. Try being less of a dick.
I don't use musl, so I don't have a horse in this race, but I wouldn't use it anyways because of issues like this one.
Do they owe me anything? Of course not. Is anyone here claiming anything to the contrary? No. So what are you on about, exactly?
This is the comment section on a website. Here we are discussing that the project maintainers kinda cocked this one up from a reputation and user trust perspective. If reputation, user trust and thus adoption are not a concern or goal of the project, then cool, good for them. It doesn't mean, however, that there isn't some reputation or user trust that was eroded. And it certainly doesn't mean we can't discuss it.
For you to reply "what are you a child, just fork it and grow a pair, fix it yourself" is not constructive nor does it contribute to the discourse. Or did I put words in your mouth? Hm I wonder how that feels.
Our CDN provider ended up having a shedding mode in some hot areas that made the chain exceed the limit from time to time. Our multi CDN set up saved us so we could do geo specific failovers.
While I'm glad this is finally addressed, this limits the usefulness of one of my favorite interview questions.
Asking about Alpine in a production environment was always good way finding who has container experiences of watching C-Beams glitter in the dark to those who only just read a "10 Docker Container Tricks; #8 will blow your mind!" blog post from 2017.
It's still going to be pretty common for at least a few years, and the now incorrect assumption that it is still broken I'm sure will persist for a decade or more among those who have been burned and thus moved on from Alpine and do not follow it.
DNS is a fun rabbit hole for interviews, for sure.
My favorite one to see on a resume is NIS. If you are listing NIS and don't have horror stories or other things to say about NIS, that's a really good indicator of the value of your resume.
I intentionally list NIS on my resume because it is such a fun conversation topic to go on about how security models changed over time, all the ways NIS is terrible, but also how simple and useful it was.
After constraint 1, this devolves to a weird game of "does the interviewee realize I don't consider `getent hosts` to be a 'common networking utility' so it's still available?"
Except it's... not quite correct. getent performs NSS resolution, not DNS resolution, so its output depends on the value of the hosts field in your /etc/nsswitch.conf, which could be set to something fun like LDAP or something else.
I don't know Kubernetes, so it's very possible that Kubernetes enforces that name resolution is always DNS and only DNS (a "hosts: dns" line in /etc/nsswitch.conf, without any additional stuff like files which would mean resolution via /etc/hosts), in which case getent is indeed correct, but in the general case, there is no system-intrinsic tool that tests DNS only, and you have to use something like dig.
This must be an IT thing because a developer would go, "ah, what is this shit? Hang on, I did this once back in 08', let me being up the documentation' and proceed to bootstrap a production-fix by EOD.
DevOps/SRE. Our technical interview is pretty regimented and cookie cutter. Over the course of one hour we cover a wide base of practical knowledge to validate the candidates existing knowledge or their ability to quickly acquire the needed knowledge (use of google/stack overflow is encouraged).
As an icebreaker, I pick out items from their resume that stand out and look like strengths. My goal is to play to those strengths and get them comfortable talking about their past battles.
> It's still going to be pretty common for at least a few years, and the now incorrect assumption that it is still broken I'm sure will persist for a decade or more among those who have been burned and thus moved on from Alpine and do not follow it.
I'm still not going back until I can configure for TCP to be the default.
> Asking about Alpine in a production environment was always good way finding who has container experiences of watching C-Beams glitter in the dark to those who only just read a "10 Docker Container Tricks; #8 will blow your mind!" blog post from 2017.
I dunno, I've been running containers in prod for a while now and I don't recall Alpine being a problem. Maybe it varies by your workload?
It's usually the case that everything works until it doesn't. When it's DNS that doesn't work, good luck debugging it unless you've got war stories to tell.
glibc also has some fun behavior that few people know about because (1) distributions have been patching it and nobody ever actually ran the upstream version and or (2) downstream software is papering it over:
This seems pretty neat. We’ve been getting reports from a lot of people where they seem to be having random DNS failures, and from what we can tell it’s because they’re on an IPv6-only network (seems very common in developing countries with infrastructure that can’t hand out IPv4) and the UDP DNS fails. Hope this helps resolve it for them.
Doesn't distroless bring in a lot of complexity when you need something as simple as ca-certificates?
IMO Distroless or even scratch is nice for statically complied binaries or self contained deployments, but if there's a dependency on user space then it becomes complex.
You can copy the CA bundle in the last step of the image build (along with the required libs and their assets/dependencies), or am I missing something?
Did the musl people never get around to implementing EDNS0? There's lots of talk here about not implementing DNS/TCP, but the musl people could have ameliorated that, and side-stepped the years of complaints to an extent, by implementing EDNS0 and allowing up to 64KiB datagrams.
Yes, some (I hope more rare now than when I wrote the FGA on it almost 20 years ago) firewalls cannot handle large DNS/UDP datagrams, but this code isn't for traffic that crosses firewalls. It's for traffic that goes only as far as the local resolving proxy, or at least goes somewhere "nearby" like 9.9.9.9. EDNS0 firewall worries are a lot less here.
And EDNS0, at least for the bit that enables >512 byte datagram sizes, is a lot less complex to implement than full DNS/TCP fallback. It's just proper buffer size handling and an extra resource record set.
I think there's also still some potential problems because it still does some things differently than glibc. Musl defaults to parallel requests if you define more than one nameserver (multiple --dns=, for example, for the docker daemon)...where glibc uses them in the order you provide them.
To be clear, that's not "wrong", but just different maybe from what docker was expecting.
I will admit though, that Debian-slim still has some non-essential stuff that usually isn’t needed at runtime, a shell is still neat for debugging or local development. This trade off could be considered a security risk, but it’s rather simple to restrict other stuff at runtime (such as run as non-privileged, non-root user with all capabilities dropped and a read-only file system except for /tmp).
It’s a balancing act between ease-of-use and security. I don’t think I’d get popular with the developers by forcing them to use “FROM scratch” and let them figure out exactly what their application needs at runtime and what stuff to copy over from a previous build stage.