Make our CI faster

[gabrieldemarmiesse]

Currently our CI take ~30 min to run. After some analysis, here is the critical path and the big steps:

• Building the wheel on linux: 25m
  • pull the docker image: 3m
  • install TF and dependencies: 2m
  • build CPU and GPU ops: 8m (building CPU ops only takes 2m in another build, so we can assume 6m for building the GPU ops)
  • Run all tests: 6m, I should re-enable pytest-xdist here.
  • Rebuild all ops for the wheel: 5m (because the bazel cache was cleaned before)
• Test the release wheel on Windows: 4m10s
• Upload to pypi, but not really: 25s.

I'll work toward making this faster, help and ideas are welcome too of course.

[bhack]

Can you point on the CI scripts sources?
Can we take any speed-up with https://github.com/actions/cache?

[bhack]

See also this thread https://git.luolix.topmunity/t5/GitHub-Actions/Cache-docker-images/td-p/44978

[gabrieldemarmiesse]

Caching may help, but it may also make things slower. It's usually hard to know until we try.

[bhack]

I've not built the linux wheel Dockerfile with the progress option but I suppose that in 25m the main problem is that /root/.cache/bazel is an empty volume when you run the Action vs local build right?

[bhack]

I've asked if we there is a way to cache with action/cache the mount of buildkit RUN --mount=type=cache at actions/cache#260.

[bhack]

In the meantime I think that

--cache-from=type=registry,ref=$CACHE_TAG
--cache-to=type=registry,ref=$CACHE_TAG,mode=max

could work if we have a registry to push. See https://medium.com/titansoft-engineering/docker-build-cache-sharing-on-multi-hosts-with-buildkit-and-buildx-eb8f7005918e

[gabrieldemarmiesse]

I did that a while ago, didn't seem very fast to me at the time. If someone wants to try again, a pull request is welcome: #1043

[seanpmorgan]

@gabrieldemarmiesse Can this be closed or are there pending actions we need to do?

[gabrieldemarmiesse]

Let's close it as there is no obvios path right now

[bhack]

If you want you could subscribe to tensorflow/build#5 cause we are marginally talking also about the bazel cache for github actions.
That ticket was created yesterday by one of the SIG Build leaders after our chat on the SIG Build Gitter channel.