Personally I feel like shrinking images by guessing unused parts is an a good way to have an image explode in your face randomly in the future. (Probes and heuristics missing critical but rarely used parts and more) Also wouldn't it hurt reproducibility? Temporary runtime monitoring doesn't exactly sound like a deterministic metric.
A containerizable project probably has its requirements known and well-specified?
I think building on top of a base with a smaller unused surface is a better idea than using analysis that might backfire. These days I am using apko + melange for my personal images and they are super neat.
There's always a trade off. You are willing to do more low level work "manually" assembling container images your way with apko. You are also willing to accept Alpine randomly exploding in your face :) Different people have different preferences in terms of what risks they will take. For example, I wouldn't use Alpine even if somebody paid me money to use it :) That's because Alpine is not a standard Linux distro and you need to be able to "own" it with all of it's gotchas and incompatibilities. Not everybody is capable of doing that. I know I'm not that good :-) In other cases people can't change their base images.
Some form of tree-shaking type of thing would probably be quite handy for images, but yeah I'm a bit wary here as well. First thought would be what happens when it hits Out-of-Memory, DNS timeout, or loses network connectivity or another edge case that totally happens in Production.
Removing those code paths would not be a good thing, but I guess if you build your apps right you could just have your container orchestration system recover by replacing the Pod.
The commercial version could include a small filesystem shim that tracked references to missing files and maybe even did some work to fill them back in over the network!
Isn't this incredibly dangerous? I know everyone likes to pretend they have perfect code coverage, but just ripping stuff out that wasn't called during 'probing' feels like the perfect way to make rare code paths even more dangerous.
kkrieger (https://www.pouet.net/prod.php?which=12036) is an impressive 3D shooter in only 96 KiloBytes. As one of their optimization techniques they recorded all code paths and discarded unused parts. At least in the first version this was why you could only use CursorDown in the menus and CursorUp did not work.
My guess is: praise for kkrieger for its tiny size, and condemnation by demonstration because kkrieger was known to have bugs because of the usage of that technique.
kkrieger was my first thought too - I think there was some other important piece of functionality that got stripped as well (I want to say something to do with hitboxes or collision detection).
I guess it only seems dangerous to me if you blindly follow it's recommendations. Feels like it could generate a list of "things you may want to consider", that you'd then be able to use to take a look at your container.
it sometimes doesnt work sure but thats why we have tests and test i minify all my containers nowdays and in most cases it works in those that it doesnt i figured out the pattern when and why for my apps and use include flags to ensure things remain inside
Even prior to docker-slim there were tools like Quay.io that "did the right thing" by squashing images to just the contents of the final image layer.
The best thing you can do is use minimal images and multi-stage builds. This should help you immensely to reduce your attack vector and do standard software bill of materials, too.
The quay.io squashing optimization is a lot safer though, right, as it doesn't remove anything that should be visible to the container?
I agree that the multi-stage builds are the best option, but it can be hard to know if you've included everything that is required or if you've accidentally excluded something that is important in rare cases.
I guess the question is in which way dangerous?
It might lead for crash for sure, but is that crash controlled? If it is, then it is just a crash. Stability vs. minimal attack surface
But I agree, this is just bandaid for lazy bois. Better use Bazel etc. for distroless builds
This is dangerous in that it strips assets, resources, and files from your app without understanding how they are used.
If you forget a critical code path when you build using Docker-Slim, and a resource file is not used, that resource will be stripped. The feature which depends on it will be broken in production.
i would disagree i use em in production apps, i configured it and it works if you do it blindly it happens that sometimes things break but if you configure it, it will work
There is no guarantee that a blind code shaker will leave in everything important while stripping out everything that isn't. How could it possibly know?
If Docker-Slim is working for you in production apps, you are either getting lucky or your app is trivial enough to lack unseen code paths.
Sandstorm does something similar in its packaging process. In practice it works pretty well; besides integration tests exercising functionality, the packaging generates a manifest list people typically check into source control, making it easy to see a diff between versions. I’m sure I’ve had a bad build at some point, but on the whole it works fine. Most dependencies tend to get referenced up-front anyway and any issues that arise are usually from shelling out to binaries in rare code paths.
It depends on how comprehensive they are, and how important it is that your container operates correctly.
For example, even the best integration tests (for small/mid-size companies) don't always include tests that exercise weird paths around dates/times - leap years, leap seconds, daylight savings time, etc. We often trust that our datetime library or code will handle these for us, but what if the configuration is stored in a file that isn't accessed during your integration tests?
Best case scenario is you hit the error-path soon in production and your code either crashes or does something correct-enough with a fallback path, but a worse scenario is you start losing critical information and don't realize it/fix it until it's gone on for a while.
In a non-trivial app, you can never guarantee that your "good integration tests" cover every edge case. If you could, we wouldn't have outages in production.
Not really. There's a few common problems doing this with Go and for example, Alpine as a base image. You'll have to manually grab something like sqlite.c or some openssl stuff (its been awhile) but after that it works just fine.
Its actually a good way to find dependencies you didn't know were there. As long as you're diligent this isn't something you should be shaking in your boots over.
If you have a good pipeline to prod, should be okay. You should hopefully have plenty of automated tests to ensure it doesn't get to prod if there are errors.
I think a "good pipeline to prod" with sufficient automated tests to ensure nothing is broken is the exception not the rule. Even in places that think/say they have a "good pipeline to prod". It's something that takes a shocking amount of engineering effort to do well, and tons of discipline to maintain.
Totally agree. I've joined companies where there was literally zero test coverage. All tests were done manually. But then if you have bad test coverage, then you shouldn't be using something like this.
should be okay is definitely not enough for me to ship things to production.
And while I do have automated tests, they might sometimes stub system calls as I'm mostly testing my code to keep things stable and fast.
I'd rather explicitly declare my dependencies and use the same container for development, test and production to feel much more confident that it includes actually everything that's needed.
thats why you should test and if there are stuff htat needs to be included but arent and you know wont work fail the test and add --include-path to your docker-slim command to ensure something is added
We build our binary first with one image as the builder image, then use `copy` to copy the binary from the builder to the final executable image like alphine.
an example Dockerfile likes:
FROM golang:1.18.1-alpine as builder
# RUN apk add, wget, etc, and build the binary
FROM alpine
# or FROM scratch
COPY --from=builder builder/binary /binary
ENTRYPOINT ["/binary"]
How do non running utilities increase attack surface? If you're able to execute inside the container couldn't you just write whatever utilities you want in?
There are many exploits that may give one the ability to execute shell commands. If there is no shell or commands to even write a file in the first place, mobility becomes limited.
once in awhile i use 'kubectl exec ' to run some commands against my running prod containers to debug something or extract an environment variable. can i still do that without a shell or anything?
That sounds sweet! But I guess DigitalOcean doesn't support it? I'm getting "error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource")."
You can’t, no. One approach is a “debug sidecar” container in the Pod that has your desired tools. Linkerd, for example, will add such a container if an annotation is present on the Pod: https://linkerd.io/2.11/tasks/using-the-debug-container/
I usually just include a shell, but to each their own.
I've had great success with reducing image size by running docker-show-context (https://github.com/pwaller/docker-show-context) and eliminating big and unnecessary files that it reports. This seems to go just a bit further than that with what seems like more complexity. I got timeouts when following their instructions to run it on two different containers, one of which is just a very simple web server.
This is interesting for optimizing build time. But I think it works a bit different from docker-slim, which is focused on the final resulting image size.
If you want a quick double check to verify a handful of common cleanups, check out https://github.com/viraptor/cruftspy, which reports cruft like logs, caches, and git repos.
docker-slim is the wrong solution for container optimization. You can't just have a program rm -rf files that it didn't think were in use. What if you missed a code path?
I know your question was rhetorical, but probably depends on how mission-critical the code path is (and the consequences of hitting those missing files).
If you're running a website and the removed dependency is related to a feature that is uncommon enough that isn't covered by your automated tests, maybe .1% of your users experience a broken page.
If you're running critical infrastructure and the removed dependency has to do with leap-second handling, maybe eight months from now, everything crashes and you lose millions of dollars.
Please explain how it's wrong without simply saying you prefer other solutions, which was the case with Ariadne :) Dead code elimination is a common construct in software engineering. There's nothing magical about apps and their dependencies. They are relatively straight forward to identify and for the web apps with static assets there are helpers to help you ensure you got everything.
Why would someone want to use this instead of say base images made specifically for containers? like alpine for eg.?
And for languages like golang (in their examples) - why/how would anyone get such huge container images in the first place? Doesn't go give a neat statically linked binary?
One word... libc :-) There's tons of info on the internet about the gotchas in Alpine. Even fly.io gave up on using/supporting Alpine based container images and they are probably one of the most competent and capable engineering teams out there.
Honestly I didn't know that. I have only used alpine based images for some pure python webservers and was fairly happy with the size and no bugs so far. So what is the go to distro for base image for containers these days? Ubuntu-minimal?
And yeah libc was a pain for us even for AppImages. You'd think that something as fundamental as C library would be standardized on Unixes...
Alpine's libc is intentionally non-standard* and Alpine (at least, some of their members) explicitly state that Alpine is not GNU Linux while most other distros people use are :)
distroless is picking up a lot of interest especially with its recent uptick in its adoption in the kubernetes community.
Good question. Alpine is already small enough that it seems a little odd to go to elaborate measures to reduce image size further. Seems better to me to start with a minimalist image and only add what you need to make your app work than to start with a huge image with everything, install your app, and rely on something like this to find only the things you don't need to remove and not make any mistakes.
you can use whatever base image you want lets say ubuntu:latest (i dont like alpine) and normaly base images tend to include a lot of stuff that doesnt have any place in container think why do i need a tool for ext4 managment inside contianer makes no sense ok for production throw it out thats what docker-slim does
and gets rid of vulnarabilities in programs that are not used by your program by simply getting rid of them
this removes far more then multistage docker build ever would, do you need bash dash or passwd or many other binaries and files in image that are in by default no you dont only way to do anything simular to what docker-slim does is with scratch image which doesnt work if you dont copy everything you need in
Still seems kind of silly. If you base everything on ubuntu minimal, you'll only have the one copy of that base image, which is a fraction of the size of the `docker` and `dockerd` binaries added together. No server running docker will have a problem keeping one or two versions of ubuntu minimal on it.
But if you go around "minifying" all your applications independently, you won't have that shared base layer. One application needs `sh` and another doesn't? Now you get two entire base layers, one with it and one without. Sure, each image's total size will be less, but the size of all your different images added up will be greater because you killed the sharing.
If for some reason the 29 megs of ubuntu minimal (or even fewer for alpine) are a problem (which they aren't on your server that already has over a hundred megs of `docker` binaries), then the right solution is to better control layer sharing. Ensure that you don't have different base layers between your applications. And then--strictly for kicks and giggles--you could minify that base layer to the minimal set of what all your images require. To save a 51K `passwd` binary (woohoo!).
one question is is possible in any kind of way that that passwd or any other binary that stays that you dont need has a security vulnarability that could if someone got into the container in one way or another(most likely your app) cause trouble on the host.
hint yes it is and that could be a problem a giuant one
The problem is not about removing though. The problem is what/who guarantees that nothing broke after all these files are removed? Especially in obscure code paths in nested dependencies?
With something like alpine linux/ubuntu minimal, you trust the package maintainers to make sure that if you use python in your docker image it would work like it worked for them. Out here, it just says "Yes (it is safe)! Either way, you should test your Docker images.".
As a bad example, if a library used by your application uses a different "theme" requiring different files at night and different files during the day, you might still say "it worked during my tests" but things definitely broke and the only thing you can blame is this overzealous tool.
That bad example was from back when i was trying to make AppImages for an application we used. At first all we did was recursively collect all the libraries reported by ldd. Then it turned out some libraries were only being dlopen'ed by other libraries under specific circumstances and we missed them. So we manually added those libraries. Then it turned out that we missed the config files and other resources used by those libraries. Eventually we shipped all the files belonging to all the distro packages used by the libraries we used and left it at that.
in some cases i essentialy ensure my whole app remains using --include-path flags so that i get a removal of you know things that i absolutly dont need.
They don't even contain a kernel - Docker containers use the host kernel. Container runtimes based on VMs like Firecracker's firecracker-containerd typically supply the kernel themselves.
On the other hand, scratch is a special image that contains literally nothing, 0 bytes. Docker doesn't have to talk to the network to download it, it doesn't have to be built, it's just a tarball of nothing.
scratch is thus infinitely smaller than distroless since it has no size.
It's also not suitable for rust since rust applications will commonly link against openssl for tls, against a libc implementation for things like network operations and threads, and so on.
Rust can link to rustls instead of openssl, musl targets are very convenient (and can statically link musl), and https://crates.io/crates/trust-dns-resolver can be used instead of doing dns via libc
Because of such tooling / ecosystem, it's more convenient to use scratch from Rust than from most languages (including C)
This kind of looks like a tool that does the reverse of what scratch does. Instead of only including the binary and any dynamically linked dependencies, it tries to figure out a minimum set of dependencies based on access.
In practice, I'm curious how error prone the result is.
it is error prone somewhat but it has flags to allow you to fine tune what gets added back in.
great thing is you can work with any base image and language including those that wont work with scratch
For workloads where the image size was critical, I have achieved a similar result with using strace to collect the required files and then limiting the image to only those files in the build process.
It's a neat approach, but ultimately brings non-negligible amount of uncertainty as you can never be 100% sure your test set of inputs did not miss a particular edge case which will require to have a file present in the container that no other input does.
yes that tends to be the problem with docker-slim as well that is why it includes flags like --include-path with which you can easily achive such fixes
personaly i highly recommend as it works in most cases and gets rid of those vulnerablities that come with things like bash or passwd that you dont need in prod apps
I feel like if you get to the point of considering docker-slim, you should consider alternate build strategies like compiling dependencies from scratch with minimal functionality (like with buildroot)
Container optimization also seems to border on premature optimization. If you're not hitting issues with your container size, don't worry about it. People may scoff at a 10GB container but for something that doesn't scale or get built a lot, it may be fine.
There are also other options like pre-pulling containers on the machines running them (which you could build into a golden image) that can get you similar startup/pull speed gains for something like host auto scaling use cases
Instead of going extreme with coverage analysis, it shows places that can be manually cleaned during the build process. Maybe someone will find it useful. Smaller space gains, but gives more confidence.
i have docker-slim in CICD completly automated seems to not have a problem as i ahve configure it per pipeline, maybe check out examples for docker-slim
I have a minor in math, and I don't know what "shrinking by 30X" means. To me, decreases always start from 100%. So I think we are talking about a ~97% decrease in size?
It's nice because they're inverses - if you shrink by 30x, then grow by 30x, you're back where you started, whereas a 97% decrease in size followed by a 97% increase in size leaves you at ~6% of the original size.
I think if you want to say "1/30th of the size", you should say that. Growth is usually measured as a difference. For example, a 200% increase means the value has tripled.
But "30x" is just another way of saying "3000%". Or, "3000%" is just another way of saying "30x". "Shrinking by 30x" means the same thing as "shrinking by 3000%".
Thanks I hate this, but seems to be everywhere now. "This products is now 3 times cheaper!", WTF. They still haven't got to percentages yet, like 200% off!!.
If I say "image A is 1 MB, image B is 30 times bigger", what is the size of B? I don't think many would respond "31 MB". If you consider it improper to use "bigger" in this way to express a multiple, how would you express that?
I'm no mathematician and it wouldn't often be my choice of phrasing, but it seems unambiguous and clear to me; your definition is much stranger/less intuitive to me.
We're clear on 'x is two times larger than y', right?
x = 2*y
An equivalent statement is 'y is two times smaller than x', but it conveys a construction more like:
y = x/2
Which, since we're speaking English sentences, might change the emphasis/implication.
This is really just an English expression that doesn't map perfectly to the actual mathematics it's intended to describe. 2 times smaller does usually mean 1/2 by convention in English and I don't think there's anything more profound than to just say it is what it is.
Everyone is super negative about this. Use what works for you.
It's kinda like tree shaking in JS. Super standard practice and hugely deployed everywhere. I don't care that zip or tar are removed. I care that my app runs, passes QA and gets deployed to prod.
I don't know what other containers people are running but if you have Lotus Domino running in a container, fine, don't minimize that, I'm super happy to minimize my node process that collects HTML files from other sites.
The cert is usually the public key. The private key is usually named key. So it doesn't promote secrets being saved within a container as far as I can see.
Even further off topic but perhaps relevant: Chimera Linux, which consists of the FreeBSD user land ported to Linux. I wonder if q66 has OCI images published...
You're quite right, and to be honest at this point I've given up trying to keep track of where it's at. It's definitely something I'd like to see completed, but I've been too busy with FreeBSD/EC2 and speeding up the boot process to spend time looking at pkgbase too.
Seems like today is the day of misconceptions for docker-slim on HackerNews :-) Happy to answer any questions. AMA
One thing I'd like to mention right away is that docker-slim doesn't do function level code elimination and that simplifies what it needs to do in a significant way.
Not the same, but potentially similar in intent, I use aconfmgr to track what's installed, changes to configuration files, and any potential changes/whole files left behind by some quick test or since uninstalled software.
(Also, even primarily to me, but less relevantly, it's great for gitting configuration & its reasons, and syncing across machines.)
Docker's trademark guidelines say that products, services and technology that are not their own shouldn't use the Docker name so it seems it's a matter of time before they get a nice letter from some lawyer.
The product is a loose partner of Docker. They are an official Docker Desktop Extension. DockerSlim is an open source tool that led to the creation of Slim.Ai (and now Slim.AI and therefore Docker Slim) are loose partners.
A containerizable project probably has its requirements known and well-specified? I think building on top of a base with a smaller unused surface is a better idea than using analysis that might backfire. These days I am using apko + melange for my personal images and they are super neat.