If you follow the link to the mailing list posting, you will see I implemented a feature: search through multiple caches.
Say you have a build server which populates a cache. Your developers can point their ccache installations to that. However, ccache wants to write things there. The fix is to have a two-level hierarchy: ccache will look in their local cache first, and then the one from the server. If there is a cache miss, it goes to the local one; the server cache isn't polluted with development objects.
One could develop a whole package manager based on this. Add some standard build profiles to something like Gentoo's portage to maximize cache hits and you could have the best of both worlds: a fast source-based distribution. Specifying a custom build profile for a package would just mean it would take more time to install but would otherwise be transparent.
One thing missing from this is a trust framework where cache writers would sign their compiled code. There could also be a verification layer.
Warpforge -- project website: http://warpforge.io / https://github.com/warptools/warpforge -- is a project I work on that's heading a bit in this direction. Hashes in + build instruction = hashes out. It's really a powerful primitive indeed.
Building big sharable package systems is indeed a second half of the problem. Figuring out how to make the system have the hashy goodness AND be usefully collaborative with async swarms of people working independently and yet sharing work: tricky. We're trying to do it, though!
gittup[1] implements part of this idea, but without the distributed bit. bazel[2] implements the distributed bit (minus trust), but not the distro bit. What's really lacking is momentum around the idea to get a sufficient number of people behind it.
NixOS uses a cache that works like this; it's source-based like Gentoo but for packages in cache it will download the cache rather than rebuilding. (it will rebuild if you configure the package in such a way to invalidate the cache, or install a package not in cache)
Besides the global cache for the whole distro, you can also set up caches for other software. For example, if you build your projects with Nix, you can have a cache for your projects (so that new contributors won't need to recompile everything from scratch). That's the premise behind https://www.cachix.org/
The only difference is that Nix caches aren't fine grained like ccache and sccache
Gentoo supports building and installing binary packages from portage: You can compile once and then distribute the resulting package to other computers where they can be installed by Portage. Portage even verifies that the USE flags are the same!
Nix is also almost there - it happily uses binaries if they match the exact build inputs but compiles if no binaries match - but it's not nearly as fine grained (you'd need something like 1 package per object file).
I've spent some time to deploy shared ccache for our Azure Devops PR builds as well as locally.
The 1 hour build time went down to 5 minutes for release, about 9 for debug. It took another 2-3 minutes to download/upload the shared ccache, so it's still a good 6 time speedup, not to mention that it takes much less CPU.
The trick is to set --fdebug-prefix-map and -fprofile-dir to proper relative paths and then with some extra scripting caches will be reusable across build nodes even if the workspace directory is different for each build.
This and distcc (or IncrediBuild) are a game changer for every serious C++ workshop.
> The 1 hour build time went down to 5 minutes for release, about 9 for debug.
I kind of one in the future if there will be a publicly trustable, free to use "just plug and play, you don't need to set up your own" ccache
Think of the energy savings implications
Imagine every unoptimized build job for every microservice (Rust in Docker pulls the entire Cargo registry + rebuilds all dependencies every time you make a source code change of any kind if you don't go out of your way to prevent it)
Obviously trusting a public source to just compile something for you and give you a binary-like object is... probably a malware distributor's dream.
And I don't have data to prove the bandwidth cost would offset the energy savings of CPU cycles recalculating the same stuff over and over.
There are all sorts of scenarios where ccache is a major accelerant. “make clean && make” is the easiest way to force a rebuild all, with relink, and ccache will supply the object files quite fast. You may also be switching back and forth between debug and release builds; again, ccache keeps the penalty low.
Finally if you run configure scripts, all those “checking for printf…” messages are the configure script generating and compiling tiny C programs invoking those functions to make sure the compiler can find them. ccache can therefore shave a significant percentage of time off running configure scripts, which is welcome.
Ah the configure scripts and large c++ codebases, the bane of fast compile times. Configure scripts "checking useless stuff" isn't parallelized so it takes ages, even on a modern beefy processor. Somebody should write ccache-like thing just for autoconfigure, so it can instantly feed answer to those checks once it has once run and cached the result, maybe this could even be part of ccache (I don't mean just the preprocessing part, but whole thing).
Many years ago there was confcache¹, that for a time was integrated in to portage via a FEATURES flag for Gentoo users. It wasn't particularly useful in the general case, and never really worked that well in practice. I don't recall the idea really taking off anywhere else, and I can't remember when it finally disappeared in Gentoo either(but it was a looooong time ago).
It was strictly a cache, it didn't run parallel checks or make any other attempts to improve the run time.
It’d be great to have a notbraindeadautoconf that just deleted the macros for the bog standard autoconf macros, checking for compilers and cstdlib and posix functions, and then invoked autoconf to work over the actual logic of configure.ac. It’s 2022, we don’t need to worry about SunOS and Ultrix and AIX and their broken compilers.
Readers please note that sccache isn't just for Rust, and isn't just caching!
It has a number of features, combining capabilities of ccache/distcc/icecream for C, C++ and Rust...along with some unique things that I've not seen in other tools. My comment at https://news.ycombinator.com/item?id=25604249 has a summary.
icecream and ccache has been a killer combo for my development routine, even if I only connect my workstation with the laptop which I would otherwise almost never use.
distcc didn't for example handle the load balancing well in my experiments whereas icecream did much better on that front and thus resulting with noticeably shorter build times. icecream also comes with a nice GUI (icemon) which can come really handy, e.g. you can use it to observe the build jobs and debug when things go south.
But I didn't know that sccache also supports distributed builds. From your comment it seems as if this was a recentish addition. I wonder how polished it is but I will definitely give it a try at some moment.
Sounds like you invented your own Turbopack (by Vercel) [0], which basically does what you're talking about, it hosts a remote cache that parallel build servers can access, cutting down compile times by quite a lot.
Documentation is encouraging. I loved icecream when it was working but it had serious reliability problems. This looks worth trying as a replacement, thanks.
Note that sccache works slightly differently and has a different threshold for what is cacheable. I could never get it to match even 80% of ccache’s hit rate.
And don't forget distcc [1], together they can speed up compilations and recompilations. I remember using colleague computers to speed up a 30 min C++ compilation to just 5 min... to just seconds on second compilation.
Distcc is great when it works but just yesterday I had to make the admin reboot my VM three times before giving up: my VM used all its memory which made it unusable..
I use ccache and love it, but I keep wondering: wouldn’t it make more sense for compilers to do caching themselves? That would allow them to cache the results of much finer-grained operations than whole translation units. If two (or 500!) translation units generate the same template instantiations and then each optimize that code, that has to happen in each translation unit and ccache can’t help; couldn’t a compiler with internal caching do it once and be done with it forever? I’ve considered trying to add this to clang, but haven’t prioritized it.
I think most of the requirements of this are fully orthogonal to the job of a given compiler, so it's best solved at the level of the build system.
Bazel does this, so instead of needing to reimplement this for each compiler it's automatically done across all your languages, and even any random one-off build rules based on a shell script.
I think the hard part is to know when caching is not allowed, and for that the compiler is a good point to manage such a cache because it could add metadata such as compiler version (maybe even patches?), compiler flags etc which a wrapper may miss. This disintegration of ccache and the compiler makes me not really trust it because these two could easily get out of sync and you would not easily notice when/why the result is wrong.
the time you spend pulling your hair out figuring out why your program stopped working due to some obscure problem with the caching mechanism, is alot more than the time you save by doing the caching. at least for a lot of people. for others it might make sense
If you doing it locally only, I don't see the difference from make that will just keep your old .o files.
If you using it distributed between different developer, how do you make sure the cache result is secure? A shared cache where everyone can contribute to is really hard (impossible?) to make secure. Someone could add malicious code the cache that then everyone will use.
To combat this scenario, we only had build servers populate the cache. Clients were readonly to the cache, so they would benefit from anything that build servers had already built, which covered 95%+ of what clients usually had to rebuild.
Also release builds were excluded from caching, to prevent any form of poisoning there.
The threat is not that the cache contains builds of untrusted code but that it contains builds that do not match the code that they are associated with.
It uses a cryptographic hash of the _inputs_ to the compiler, but there is no way to verify that the cached artifact matches the _output_ of the compiler without actually compiling it yourself.
Let’s say you have 15 engineers and they each have their own laptop computer. Each of these engineers generates a pair of cryptographic keys, one public and one private.
Each engineer then gives their public key to the trusted authority that operates the ccache server. Only code that is submitted and signed by a respective private key is built and then distributed to the rest of the engineers.
For a public project you would only want the builds to be propagated out to other developers once the changes had been approved and then merged into a branch that triggers the CI.
Make uses timestamp, so if you checkout code or switch branches even if the files are the same, make rebuild them. Also when you switch branches the ccache cache still have copy of .o files.
Using it locally too, works great on Mac, but on Windows ccache has some problems caching debug builds. IIRC the embedded debug symbols use absolute paths, so the presence of this particular flag (/Z something...) disables cache eligibility.
ccache is fantastic for CI systems. It's very common for commits to only affect part of a build, and ccache allows for clean builds (tear down the build and rebuild from scratch) to still take advantage of previous compilation runs.
Just looking at a jenkins machine with ccache I see a >90% hit rate for the cache, with 440k compilations returned from cache in the last 3 months (when stats were reset last).
Specifically, once a semester the lab systems would become the build nodes. The gfx/opengl class always used a textbook that was written around libvtk. Each new version had a new set of minimum versions for its long list of dependencies. This allowed for building the entire system to the same dependencies (vtk was very concerned with the version of libjpeg/pcx/etc whereas the rest of the system (eg x11, gnome) were happy with whatever vtk dictated.
Ccache only requires r/ssh access from the controller to any remote build nodes and the ccache program. I've never heard of anyone using this program as a shared cache source. That would be, well, kinda dumb. :/
I used to use it for gentoo locally even, but I don't remember why and I don't think I only used it for stage1. I do remember that even locally it had a huge impact on build times.
I used it for normal package builds in Gentoo for a while but the hit ratio was not too great. Even small USE changes modify the command line (or a shared config header) for most compiler invocations in the package. Same for going from release version to release version. I think it makes more sense for development builds where the changes from commit to commit are smaller.
Oh yes. I meant: started a complete stage1-3 build. Building the compiler chain and the kernel was but a single can of paint to watch dry (among a display case of cans)
Hrmm.. More fun* to use a system designed to adapt itself to the specific requirements and much less frustrating. Gentoo required no hacks and (remarkably) delivered few unstable distribution builds
I used distcc with k8s on a medium sized C++ project, until I got a workstation suitable for the compilations (32 core AMD thread-ripper). With the new workstation in place, I changed the build-script for the project to use ccache by default for all builds, and mapped a docker volume from the build-container to the local disk to keep the cache around.
That is the compiler itself sortof creating "ad hoc" modules to cache parts of the source code, ideally even inside a single compilation unit, which especially helped with template heavy code. This way completely new code could be sped up, which with ccache would never get a cache hit.
However that project is based on an old version of clang and the changes were never upstreamed (initially it was a commercial product), so sadly this project is practically dead.
Is there a sane way of using ccache in Xcode projects generated from CMake nowadays?
For other generator targets, adding ccache was a single line in the CMake configuration, but for Xcode you had to bend over backwards. This was maybe 4 years ago.
It's a nice tutorial on how to set this up, thanks. But you still have to create two launcher script files, and then set some undocumented internal Xcode variables. So to me this is more a of "no" to my question: `set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "${CCACHE_PROGRAM}")` still doesn't work for Xcode, you have to do some weird things.
I wish I could clean cache objects related to a specific c++ file. Like a bad object got into the cache and there isnt a way to remove it unless I nuke the whole cache.
When I did C++ development, ccache and distcc were absolutely vital to keep build times manageable. I can’t imagine not using them, in that environment.
The commenter was referring to having compilation and linking in separate stages, as is standard in Makefiles to enable Multi-Threaded compilation. As in
cc -c main.c -o main.o
cc -c init.c -o init.o
cc init.o main.o -o final_binary
In that specific setup, ccache does indeed not provide a speedup, since those .o file are kept. The Makefile simply checks if the source file has a younger modification date than the object file and recompiles only if it's older. In that sense the Makefile does in fact cache the results. Once we go beyond single-user and onward to bigger projects, ccache starts making sense.
All rules inherently depend on the Makefile they are defined in. The second you touch the Makefile rules, you have potentially invalidated all the object files. So, ccache is great when working on the build rules.
Using ccache is also nice when you have generated files. If you edit the generator code but the output is identical, Make will needlessly rebuild everything and ccache will make it quick.
make recompiles source code files by detecting the last modified date on a file, hence it only recompiles source files as necessary. So if you have 10 source files with 5000 lines of C in them, and you only change one of them, it will not recompile everything, it will only recompile that source file which has changed.
Which makes me agree with the parent above, I don't see how exactly Ccache is supposed to be used. Maybe for a distributed source directory with many developers working on it?
On ccache site, there is section "Why bother?", the very first line:
>If you ever run `make clean; make`, you can probably benefit from ccache. It is common for developers to do a clean build of a project for a whole host of reasons, and this throws away all the information from your previous compilations. By using ccache, recompilation goes much faster.
Putting aside my tongue-in-cheek comment, honestly this argument does not convince me very much.
What is the purpose of "make clean" other than to invalidate the whole cache so that it is cleanly recompiled? In such a situation I would want to invalidate the cache from ccache also completely.
I'm sure there are legitimate reasons for using ccache but it is not very obvious to me what it is:
"Only knows how to cache the compilation of a single file. Other types of compilations (multi-file compilation, linking, etc) will silently fall back to running the real compiler. "
Well yes, traditional use of makefiles has been exactly to cache the compilation of single compilation units and trigger the compile of changed units - ccache does not help with granularity here it seems.
Distributed development might be a good argument for this, but then what does it offer to faciliate that? It seems to suggest using NFS - which I could do with a Makefile as well. So is the advantage that it uses hashes instead of timestamps? Timestamps work quite well for me, but maybe that is a valid point.
Another argument could be that is stores the precompiled units somewhere else and therefore doesn't clutter the file system. But is that really a good argument? Build directories exist, so even if you'd like to keep compiling several variants in parallel you could do so with a few different build directories.
And yes, there are quite a lot of newer alternatives to Makefiles as well, so it would have to compete with those alternative build systems as well.
I basically never `make clean` but ccache is a boon for `git bisect`. In theory bisect takes log time; in practice, without ccache, it’s slower because handwave build time goes by something like the log of the number of commits you jump across.
If it’s my branch, I often have a build of every commit in my cache. Even if not, each jump back and forth makes a bunch of new cache entries, many of which will be reused in subsequent bisect steps.
It was my experience that building a native Android project with older NDKs benefitted hugely from introduction of ccache. Especially if you had multiple branches of the same code (release/nightly) that shared a significant code base.
That was pre-Android Studio times. IDK what is the situation now.
Some changes of mine was reworked by someone and upstreamed:
https://github.com/ccache/ccache/commit/e8354384f67bc733bea5...
If you follow the link to the mailing list posting, you will see I implemented a feature: search through multiple caches.
Say you have a build server which populates a cache. Your developers can point their ccache installations to that. However, ccache wants to write things there. The fix is to have a two-level hierarchy: ccache will look in their local cache first, and then the one from the server. If there is a cache miss, it goes to the local one; the server cache isn't polluted with development objects.