Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-8316: [CI] Set docker-compose to use docker-cli instead of docker-py for building images #6802

Closed
wants to merge 3 commits into from

Conversation

kszucs
Copy link
Member

@kszucs kszucs commented Apr 1, 2020

The build cache sometimes works sometimes doesn't.

The images pushed from the master branch were sometimes producing reusable layers, sometimes not. So the caching was working non-deterministically.

The underlying issue is docker/compose#883

@kszucs kszucs changed the title [WIP][CI] Use docker cli from docker-compose to build images instead of docker-py ARROW-8316: [CI] Set docker-compose to use docker-cli instead of docker-py for building images Apr 2, 2020
@apache apache deleted a comment from github-actions bot Apr 2, 2020
@kszucs kszucs marked this pull request as ready for review April 2, 2020 16:45
@github-actions
Copy link

github-actions bot commented Apr 2, 2020

@kszucs
Copy link
Member Author

kszucs commented Apr 2, 2020

I've tested it on my fork with the AMD64 Conda C++ image:

  1. Build and upload the produced docker images to my dockerhub account, the image was built without using any cache
  2. Trigger a new build to see that the cached layers are properly reused
  3. Pull the image locally and rebuild it so check that the cache is used. Previously my local docker daemon wasn't reusing the images pushed to apache/arrow-dev, but the newly produced image is properly reused locally (for me at least).

Theoretically the passed flag enables to both docker-compose and docker use the layer hashes.

@kszucs kszucs force-pushed the docker-compose-cli branch 2 times, most recently from 7df34d3 to 243b19d Compare April 3, 2020 10:55
@pitrou
Copy link
Member

pitrou commented Apr 8, 2020

Does something remain to be done here?

Copy link
Member Author

@kszucs kszucs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants