Nop build is slow

[Timmmm]

A build of my (quite small) app with no changes using cross takes 5 seconds. Using cargo build it only takes 0.1 second.

It would be nice if a nop cross build were as fast as that, so that I can have subsequent steps in my build process depend on it. Currently it makes editing files further down in the build DAG very annoying because you always have to wait 5 seconds for cross, even if you didn't change anything it consumes.

[Timmmm]

Here's an example command:

$ cross build --target x86_64-unknown-linux-musl --release --verbose
+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
+ "rustup" "target" "list" "--toolchain" "1.36.0-x86_64-unknown-linux-gnu"
+ "rustup" "component" "list" "--toolchain" "1.36.0-x86_64-unknown-linux-gnu"
+ "cargo" "fetch" "--manifest-path" "/path/to/Cargo.toml"
+ "docker" "run" "--userns" "host" "--rm" "--user" "501:20" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" "-e" "CARGO_TARGET_DIR=/target" "-e" "USER=timh" "-e" "RUST_BACKTRACE" "-e" "RUST_LOG" "-e" "CROSS_RUNNER=" "-v" "/Users/me/.xargo:/xargo:Z" "-v" "/Users/timh/.cargo:/cargo:Z" "-v" "/cargo/bin" "-v" "/path/to:/project:Z,ro" "-v" "/Users/timh/.rustup/toolchains/1.36.0-x86_64-unknown-linux-gnu:/rust:Z,ro" "-v" "/path/to/target:/target:Z" "-w" "/project" "-i" "-t" "image:cross_capnp" "sh" "-c" "PATH=$PATH:/rust/bin \"cargo\" \"build\" \"--target\" \"x86_64-unknown-linux-musl\" \"--release\" \"--verbose\""
       Fresh unicode-xid v0.1.0
       Fresh cc v1.0.45
       Fresh semver-parser v0.7.0
       Fresh ppv-lite86 v0.2.5
...
    Finished release [optimized] target(s) in 5.80s

It's the bit between docker run and the first Fresh that takes all the time.

Docker is part of the slowness, but not all:

$ time "docker" "run" "--userns" "host" "--rm" "--user" "501:20" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" ... "-t" "image:cross_capnp" "sh" "-c" "PATH=$PATH:/rust/bin \"cargo\" \"--help\""

...
real    0m1.309s
user    0m0.041s
sys     0m0.023s

[Timmmm]

Ok I figured out the magic to show what cargo is doing (set CARGO_LOG=DEBUG, or INFO).

One thing that is slow is this:

[build-dependencies.prost-build]
git = "https://github.com/Timmmm/prost.git"
branch = "write_if_changed"

Means that for some reason Cargo updates submodules on every build...

2019-11-28T18:05:39Z DEBUG cargo::core::registry] load/missing  https://github.com/Timmmm/capnproto-rust.git?branch=writeifchanged#548580ea
[2019-11-28T18:05:39Z DEBUG cargo::core::registry] loading source https://github.com/Timmmm/capnproto-rust.git?branch=writeifchanged#548580ea
[2019-11-28T18:05:39Z DEBUG cargo::sources::config] loading: https://github.com/Timmmm/capnproto-rust.git?branch=writeifchanged#548580ea
[2019-11-28T18:05:39Z INFO  cargo::sources::git::utils] update submodules for: "/cargo/git/checkouts/capnproto-rust-f8e69caed208a9ec/548580e/"

Another slow bit is this:

[2019-11-28T18:07:53Z DEBUG cargo::core::package] byteorder v1.3.2 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] bytes v0.4.12 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] capnp v0.10.3 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] crossbeam v0.7.2 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] failure v0.1.5 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] log v0.4.8 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] once_cell v1.2.0 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] prost v0.5.0 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] rayon v1.2.0 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] serde v1.0.101 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] serde_json v1.0.41 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] iovec v0.1.2 doesn't need a download
[2019-11-28T18:07:53Z DEBUG cargo::core::package] libc v0.2.62 doesn't need a download
....

And this:

[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/backend-a72830e1a1e3bf1a/build-script-build_script_build-a72830e1a1e3bf1a
[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/capnpc-6c64ec6c9893a323/lib-capnpc-6c64ec6c9893a323
[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/capnp-6347cddcc05443b1/lib-capnp-6347cddcc05443b1
[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/prost-build-ceb00d518a2dba38/lib-prost_build-ceb00d518a2dba38
[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/bytes-b792317d722014f8/lib-bytes-b792317d722014f8
[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/byteorder-645a09eaac5177bf/lib-byteorder-645a09eaac5177bf
[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/byteorder-c1a7daf969c40e14/run-build-script-build_script_build-c1a7daf969c40e14
[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/byteorder-12980b5944ba5f95/build-script-build_script_build-12980b5944ba5f95
[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/iovec-7a3553a0d530b566/lib-iovec-7a3553a0d530b566
[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/libc-a80b83aa86cfb4bd/lib-libc-a80b83aa86cfb4bd
[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/libc-448b86d881f39794/run-build-script-build_script_build-448b86d881f39794
[2019-11-28T18:08:25Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /target/release/.fingerprint/libc-e5df394485c7591c/build-script-build_script_build-e5df394485c7591c
...

A non-cross cargo build still does all those, but for some reason it is way way faster. I wonder if it is just Docker for Mac being a bit shit.

I think those three things account for most of the cargo time.

[Timmmm]

Ok I managed to shave ~1s off the time by using the cached and delegated mount options:

"docker" "run" "--userns" "host" "--rm" "--user" "501:20" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" "-e" "CARGO_TARGET_DIR=/target" "-e" "USER=timh" "-e" "RUST_BACKTRACE" "-e" "RUST_LOG" "-e" "CROSS_RUNNER=" "-v" "/Users/me/.xargo:/xargo:Z,cached" "-v" "/Users/timh/.cargo:/cargo:Z,cached" "-v" "/cargo/bin" "-v" "/Users/me/project:/project:Z,ro,cached" "-v" "/Users/me/.rustup/toolchains/1.36.0-x86_64-unknown-linux-gnu:/rust:Z,ro,cached" "-v" "/Users/me/project/target:/target:Z,delegated" "-w" "/project" "-i" "-t" "image:cross_capnp" "sh" "-c" "PATH=$PATH:/rust/bin \"cargo\" \"build\" \"--target\" \"x86_64-unknown-linux-musl\" \"--release\""

Still pretty damn slow.

[Timmmm]

Aha! I found a solution to my particular problem. It turns out Cargo outputs a file in target/.../foo.d that lists all of the file dependencies for foo. This can be used with the dependencies' mtimes to work out if cargo needs to be run.

I'd recomment using the cached and delegated options anyway (on Linux they do nothing).

[reitermarkus]

Nice find, I guess in this case

delegated: the container’s view is authoritative (permit delays before updates on the container appear in the host)

would be preferrable in terms of performance for the container.

[reitermarkus]

Ah, you're using delegated for the target directory already, missed that.

[polarathene]

Is some of that network activity? If so Docker by default has some notable overhead with that afaik, even on Linux. You can disregard the safe default and use the host network(something like that, vaguely recall this being the reason docker network performance was slower than the host network directly).

[Alexhuszagh]

Note that this actually doesn't have any effect anymore with cached and delegated options (and have since been removed from the documentation). You can check my logic on macOS if you wish, since I've published a working version that will mount all volumes using either cached or delegated. These options only had an effect on macOS, and were removed when the backend was changed.