Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup intermediate images #1716

Merged
merged 2 commits into from
Dec 17, 2021

Conversation

devinrsmith
Copy link
Member

For our "two-phase" images (to compress the copy layer logic into a single layer), docker leaves behind unnamed images that we are unable to cleanup. For example, see the <none> from a clean build:

REPOSITORY                          TAG           IMAGE ID       CREATED              SIZE
deephaven/runtime-base              local-build   0523ecea89ee   3 seconds ago        814MB
<none>                              <none>        3f41fa1e353a   10 seconds ago       814MB
deephaven/server                    local-build   8f1ad6d63a87   24 seconds ago       920MB
<none>                              <none>        98b6f64e4325   35 seconds ago       920MB
deephaven/web                       local-build   bbeca5f4cf7b   39 seconds ago       233MB
deephaven/js-out                    local-build   94e40bc9db72   47 seconds ago       1.03GB
deephaven/server-slim               local-build   ae2bffe02178   54 seconds ago       658MB
deephaven/deephaven-jpy-wheel       local-build   ff50c2531fb9   58 seconds ago       931MB
<none>                              <none>        bf68bae15220   About a minute ago   658MB
deephaven/deephaven-wheel           local-build   9c70845cc85c   About a minute ago   925MB
deephaven/deephaven2-wheel          local-build   f19aba39dfc5   About a minute ago   907MB
deephaven/grpc-proxy                local-build   73823a5b495d   About a minute ago   21.2MB
deephaven/dhide                     local-build   1d1e52ebddfb   About a minute ago   1.01GB
<none>                              <none>        2b80b20cc233   About a minute ago   21.2MB
deephaven/proto-backplane-grpc      local-build   3b09001a6401   About a minute ago   272MB
deephaven/envoy                     local-build   921eda9d4d62   About a minute ago   132MB

This patch adds generic logic to better handle two-phase images by ensuring we specifically target the intermediate layer and apply a name to it, so we can successfully clean it up later if necessary. This patch also removes two-phase building logic from places where it's not necessary (images that aren't necessary to publish).

After this patch, a clean build looks something like:

REPOSITORY                          TAG           IMAGE ID       CREATED              SIZE
deephaven/runtime-base              local-build   16b1d55ff52c   3 seconds ago        814MB
deephaven/server                    local-build   3e2ed605cb2c   26 seconds ago       920MB
deephaven/server-install            local-build   2b25ffd0f600   39 seconds ago       920MB
deephaven/web                       local-build   f3afb7315878   42 seconds ago       233MB
deephaven/js-out                    local-build   b76afa949f68   51 seconds ago       1.03GB
deephaven/server-slim               local-build   601b657e2def   53 seconds ago       658MB
deephaven/deephaven-jpy-wheel       local-build   69c575ebb4d8   About a minute ago   931MB
deephaven/server-slim-install       local-build   165b83f9d940   About a minute ago   658MB
deephaven/deephaven-wheel           local-build   40fcbd9a534a   About a minute ago   925MB
deephaven/dhide                     local-build   a006735fdb1e   About a minute ago   1.01GB
deephaven/deephaven2-wheel          local-build   7f4d6b3135fc   About a minute ago   907MB
deephaven/grpc-proxy                local-build   2cc53bdc317c   About a minute ago   21.2MB
deephaven/grpc-proxy-install        local-build   b887daee43bb   About a minute ago   21.2MB
deephaven/proto-backplane-grpc      local-build   01f7519654cd   About a minute ago   272MB
deephaven/envoy                     local-build   21c113bf2868   About a minute ago   132MB

niloc132
niloc132 previously approved these changes Dec 17, 2021
static TaskProvider<? extends DockerBuildImage> registerDockerTwoPhaseImage(Project project, String baseName, String intermediate, Action<? super DockerBuildImage> action) {
// Explicitly target and tag the intermediate task; otherwise, docker will leave it unnamed, and we won't be
// able to clean it up.
def intermediateTask = registerDockerImage(project, "buildDocker-${baseName}-${intermediate}") { DockerBuildImage buildImage ->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unless you have a strong objection, don't mix camel and dash word joining? we're inconsistent in other places, but it does make telling another def what something is irritating (without having to spell it out word by word, qualifying how to join each).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see buildDocker as an "identifier" right now; and I don't want to go to build-docker-{x}-{y} to break that pattern. I have similar reservations about trying to go full camel-case (uppercasing the user inputs). But I agree we are inconsistent...

@devinrsmith devinrsmith merged commit d82d9b4 into deephaven:main Dec 17, 2021
@devinrsmith devinrsmith deleted the docker/cleanup-intermediate branch December 17, 2021 23:01
@github-actions github-actions bot locked and limited conversation to collaborators Dec 17, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants