Tips to speed up compilation

[jiayihu]

Is there anything that can be done to speed up the cross-compilation? A native compilation taking about 1min can easily become 20mins within Docker. I usually go playing a blitz 5min+3s chess game meanwhile 😄

I'm currently using Docker Desktop on MacOS and I have a 15" MBP 2018. I've allocated 6 CPUs, 4GB RAM and 1GB Swap to Docker.

[jiayihu]

I have tried to add cached and delegated as suggested in #355 and by reading docker/for-mac#77 but it didn't change a bit. A debug recompilation without any file changed takes 24s on MacOS, 3mins with cross.

I guess it's some issue CPU-bound with the virtualization which can't be improved much

[ChzenChzen]

@jiayihu the problem is IO on Mac

[Alexhuszagh]

Cached and delegated are no longer supported on macOS Docker (see here). Unfortunately, IO on mac with Docker is quite slow, and this is likely the issue. Using remote docker on an SSH server might work better, which we recently added. Please note you'd have to use persistent data volumes to see any enhancement: otherwise it would be much slower.

We might need to limit the amount copied back, since you might just want the builds and not the incremental artifacts, etc.

[Alexhuszagh]

Closed since we now have full remote Docker support, which includes options to only copy changed files when using persistent data volumes. This might in some cases be much faster than running Docker locally on macOS, and if not, there's very little we can do. We can't use Podman right now (see #756, #757), and lima/nerdctl is very much so in beta, so this is outside the scope of cross.

[abcd-ca]

@Alexhuszagh would you mind sharing a brief note about why the need for remote docker and what kind of computer would be suitable to get? I have a pretty fast M1 Mac, not clear why it’s slow. Is a remote Intel PC with Ubuntu on the same LAN what is being suggested as a build server? Oh, maybe you mean setting up a container on the mac and the container is considered the remote?

[Emilgardis]

On the Mac M1 the docker images can be slow to run due to bad QEMU virtualization. using CROSS_REMOTE with the context being the local host would not solve the issue.

You might have better milage using an aarch64 native image (which we don't host right now, see #751 (comment) for an example)

[Alexhuszagh]

We're hoping soon to deploy native ARM64 images soon, the major obstacle currently is requiring self-hosted runners. However, I'll be re-integrating some automated testing for the process.

[abcd-ca]

Happy to test, thanks!