clcache.py - a compiler cache for Microsoft Visual Studio

clcache.py is a little Python script which attempts to avoid unnecessary recompilation by reusing previously cached object files if possible. It is meant to be called instead of the original 'cl.exe' executable. The script analyses the command line to decide whether source code is to be compiled. If so, a cache will be queried for a previously stored object file.

If the script is called in an unsupported way (e.g. if the compiler is called for linking), the script will simply relay the invocation to the real 'cl.exe' program.

Installation

Python 3.3+ is required.

Installation via exe

You can generate a self-contained 'clcache.exe' file using PyInstaller. If you don’t have it already, installing PyInstaller is usually a matter of running

pip install pyinstaller

Afterwards, you can generate an executable by running

pyinstaller --onefile clcache.py

This will put the resulting binary and all the dependencies into a 'dist\\' subdirectory. You could then tell your build system to use 'clcache.exe' as the compiler binary, or you could rename 'clcache.exe' to 'cl.exe' and put it into a directory which comes first in the PATH. In my case, I went for the latter solution; I put the 'cl.exe' binary into '%HOME%\bin' and prepended that directory to the PATH. The 'cl.exe' wrapper binary will notice that it was renamed and forward all the arguments to the real compiler executable.

This way, simply running 'cl' will invoke the script instead of the real compiler.

Installation for Visual Studio

Some users have reported (see e.g. GitHub issue #18) that in order to make Visual Studio pick up clcache, the original compiler binary needs to be moved out of the way and replaced with the executable generated by PyInstaller:

1. Rename cl.exe to e.g. cl_original.exe

2. Rename cl.exe.config to e.g. cl_original.exe.config

3. Copy clcache.exe to cl.exe

4. Set CLCACHE_CL environment variable to point to cl_original.exe.

Alternative installation via clcache.bat

This method is especially handy when there are multiple Python installations on a system.

Create a clcache.bat and put it in a directory in PATH (e.g. %HOME%\bin):

@echo off
@setlocal
rem this is a good place for clcache environment variables
set CLCACHE_HARDLINK=1
C:\Python35\python.exe C:\clcache\clcache.py %*

Now set your compiler to clcache.bat in the build system, e.g. for cmake

set CC=clcache.bat
set CXX=clcache.bat
cmake ...
nmake

Check stats via clcache.bat -s, clear cache clcache.bat -C and so on.

Options

--help

Print usage information

-s

Print some statistics about the cache (cache hits, cache misses, cache size etc.)

-c

Clean the cache: trim the cache size to 90% of its maximum by removing the oldest objects.

-C

Clear the cache: remove all cached objects, but keep the cache statistics (hits, misses, etc.).

-z

Reset the cache statistics, i.e. number of cache hits, cache misses etc.. Doesn’t actually clear the cache, so the number of cached objects and the cache size will remain unchanged.

-M <size>

Sets the maximum size of the cache in bytes. The default value is 1073741824 (1 GiB).

Environment Variables

CLCACHE_DIR

If set, points to the directory within which all the cached object files should be stored. This defaults to %HOME%\clcache

CLCACHE_CL

Can be set to the actual 'cl.exe' executable to use. If this variable is not set, the 'clcache.py' script will scan the directories listed in the PATH environment variable for 'cl.exe'.

CLCACHE_LOG

If this variable is set, a bit of diagnostic information is printed which can help with debugging cache problems.

CLCACHE_DISABLE

Setting this variable will disable 'clcache.py' completely. The script will relay all calls to the real compiler.

CLCACHE_HARDLINK

If this variable is set, cached object files won’t be copied to their final location. Instead, hard links pointing to the cached object files will be created. This is more efficient (faster, and uses less disk space) but doesn’t work if the cache directory is on a different drive than the build directory.

CLCACHE_NODIRECT

Disable direct mode. If this variable is set, clcache will always run preprocessor on source file and will hash preprocessor output to get cache key. Use this if you experience problems with direct mode or if you need built-in macroses like _TIME_ to work correctly.

CLCACHE_BASEDIR

Has effect only when direct mode is on. Set this to path to root directory of your project. This allows clcache to cache relative paths, so if you move your project to different directory, clcache will produce cache hits as before.

CLCACHE_OBJECT_CACHE_TIMEOUT_MS

Overrides the default ObjectCacheLock timeout (Default is 10 * 1000 ms). The ObjectCacheLock is used to give exclusive access to the cache, which is used by the clcache script. You may override this variable if you are getting ObjectCacheLockExceptions with return code 258 (which is the WAIT_TIMEOUT return code).

CLCACHE_PROFILE

If this variable is set, clcache will generate profiling information about how the runtime is spent in the clcache code. For each invocation, clcache will generate a file with a name similiar to 'clcache-<hashsum>.prof'. You can aggregate these files and generate a report by running the 'showprofilereport.py' script.

Known limitations

• INCLUDE and LIBPATH environment variables are not supported.

How clcache works

clcache.py was designed to intercept calls to the actual cl.exe compiler binary. Once an invocationw as intercepted, the command line is analyzed for whether its a command line which just compiles a single source file into an object file. This means that all of the following requirements on the command line must be true:

• The /link switch must not be present

• The /c switch must be present

• The /Zi switch must not be present (/Z7 is okay though)

If multiple source files are given on the command line, clcache.py wil invoke itself multiple times while respecting an optional /MP switch.

If all the above requirements are met, clcache forwards the call to the preprocessor by replacing /c with /EP in the command line and then invoking it. This will cause the complete preprocessed source code to be printed. clcache then generates a hash sum out of

• The complete preprocessed source code

• The `normalized' command line

• The file size of the compiler binary

• The modification time of the compiler binary

The `normalized' command line is the given command line minus all switches which either don’t influence the generated object file (such as /Fo) or which have already been covered otherwise. For instance, all switches which merely influence the preprocessor can be skipped since their effect is already implicitely contained in the preprocessed source code.

Once the hash sum was computed, it is used as a key (actually, a directory name) in the cache (which is a directory itself). If the cache entry exists already, it is supposed to contain a file with the stdout output of the compiler as well as the previously generated object file. clcache will copy the previously generated object file to the designated output path and then print the contents of the stdout text file. That way, the script behaves as if the actual compiler was invoked.

If the hash sum was not yet used in the cache, clcache will forward the invocation to the actual compiler. Once the real compiler successfully finished its work, the generated object file (as well as the output printed by the compiler) is copied to the cache.

Caveats

For known caveats, please see the Caveats wiki page.

License Terms

The source code of this project is - unless explicitly noted otherwise in the respective files - subject to the BSD 3-Clause License.

Credits

clcache.py was written by Frerich Raabe with a lot of help by Slava Chigrin, Simon Warta, Tim Blechmann and other contributors.

This program was heavily inspired by ccache, a compiler cache for the GNU Compiler Collection.