Optimize size of default-cpu / gpu dockerfiles, update to latest ubuntu and cuda, remove python 2 #3557

jzwang43 · 2021-05-27T06:03:17Z

Reasons for making this change

Update (to latest ubuntu) and make default-cpu / gpu dockerfiles smaller

Related issues

fixes #3526

Screenshots

Checklist

I've added a screenshot of the changes, if this is a frontend change
I've added and/or updated tests, if this is a backend change
I've run the pre-commit.sh script
I've updated docs, if needed

epicfaace · 2021-05-27T15:21:03Z

How small are they now?

jzwang43 · 2021-05-27T23:20:44Z

How small are they now?

I think docker slim reduces the size based on what is used and not used during execution. But we are providing a general image which does not have a specific behavior since user can do anything with it. So I am not sure docker-slim can help much. I checked the images and looks like they are already removing the apt files.

epicfaace · 2021-05-28T02:06:37Z

Okay, sounds good. From my earlier comment "How small are they now?" I meant, how large are the images now (in bytes?) I know they used to be 5 GB, so I'd like to know how much space we saved in this PR.

jzwang43 · 2021-05-28T07:02:19Z

Okay, sounds good. From my earlier comment "How small are they now?" I meant, how large are the images now (in bytes?) I know they used to be 5 GB, so I'd like to know how much space we saved in this PR.

I am not sure why but locally if you do docker images, the reported size for the original image is 10+ GB. After my changes it seems to be 1GB smaller. I think on docker hub reports around 5GB for the size and it might be related to this issue: docker/hub-feedback#331

Just as an experiment, I tried removing python2, the docker image size is halved.

epicfaace · 2021-05-31T22:11:49Z

docker_config/dockerfiles/Dockerfile.default-gpu

+    wget
+
+RUN curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
+RUN python2 get-pip.py	


We should be able to remove python 2 now.

epicfaace · 2021-05-31T22:48:35Z

docker_config/dockerfiles/Dockerfile.default-cpu

-RUN add-apt-repository ppa:deadsnakes/ppa && \
-    apt-get update -y  && \
-    apt-get install python3.6 -y \
+RUN apt-get install python3.6 -y \


Let's combine this with the previous apt-get command

epicfaace · 2021-05-31T22:49:36Z

docker_config/dockerfiles/Dockerfile.default-cpu

-RUN add-apt-repository ppa:deadsnakes/ppa && \
-    apt-get update -y  && \
-    apt-get install python3.6 -y \
+RUN apt-get install python3.6 -y \
        python3.6-venv \
        python3.6-dev \
        python3-pip \


At the end, we should also add && rm -rf /var/lib/apt/lists/*, which will save more space -- see https://stackoverflow.com/questions/61990329/dockerfile-benefits-of-repeated-apt-cache-cleans. let's also add this to the end of all apt-get install commands.

epicfaace · 2021-05-31T22:50:19Z

docker_config/dockerfiles/Dockerfile.default-cpu

@@ -60,26 +54,11 @@ RUN wget http://scala-lang.org/files/archive/scala-$SCALA_VERSION.deb && \
    apt-get clean && \
    apt-get autoremove && \


We should combine all apt-get install commands into one single command if possible. I believe this should save a bunch of space (as we'd store far less layers).

epicfaace · 2021-05-31T22:53:07Z

docker_config/dockerfiles/Dockerfile.default-gpu

-RUN pip2 install -U \
-      tensorflow-gpu==1.12.0 \
-      tensorboard \
-      keras
 RUN python3 -m pip install -U \


For both dockerfiles: we should combine all pip install commands into a single command, and also use the --no-cache-dir option to save space. See https://stackoverflow.com/questions/45594707/what-is-pips-no-cache-dir-good-for (we can probably keep the pip install command at the end with apex as-is, though)

jzwang43 · 2021-06-01T07:16:17Z

@epicfaace looks like the newer nvidia/cuda image is much larger than the older version we used before. The old base image was under 1GB (link) and if we upgrade the newer image is 2.61GB (link), this will increase our default-gpu image size by a lot. What should we do in this case?

epicfaace · 2021-06-02T21:01:06Z

As per our discussion, let's try the following:

clearing out disk space (

codalab-worksheets/.github/workflows/test.yml

Lines 103 to 106 in 8857244

    
                 - name: Clear free space 
        
                   run: | 
        
                       sudo rm -rf /opt/ghc 
        
                       df -h

and https://github.com/flashlight/wav2letter/actions/runs/74797824/workflow#L14-L18)

multistage build for nvidia apex (see https://github.com/cisagov/crossfeed/blob/master/backend/Dockerfile.worker for an example)

.github/workflows/build_worker_images.yml

epicfaace · 2021-06-03T00:20:33Z

.github/workflows/build_worker_images.yml

+          sudo rm -rf /opt/ghc
+          sudo apt clean
+          docker rmi $(docker image ls -aq)
+          df -h
      - run: python3 codalab_service.py build ${SERVICE} $([ -z "${CODALAB_DOCKER_USERNAME}" ] || echo "--push")


Right now, it's pushing to a tag called HEAD, rather than the actual PR branch name. We should change this

Update the github action file, should be working now: https://hub.docker.com/layers/codalab/default-cpu/update_images/images/sha256-ee7b6df243b8e51db27fccc8f1f125dee21bddff29fa21da1377b11b75c4acd4?context=explore

epicfaace · 2021-06-03T00:21:18Z

Image size changes with this PR:
default-cpu (https://hub.docker.com/r/codalab/default-cpu/tags?page=1&ordering=last_updated): 5.1 GB -> 1.63 GB 🎉
default-gpu (https://hub.docker.com/r/codalab/default-gpu/tags?page=1&ordering=last_updated): 6.18 GB -> 7 GB

It might still be worth it to consider using multistage builds for default-gpu to save more space -- what do you think?

jzwang43 · 2021-06-03T02:12:20Z

Image size changes with this PR:
default-cpu (https://hub.docker.com/r/codalab/default-cpu/tags?page=1&ordering=last_updated): 5.1 GB -> 1.63 GB 🎉
default-gpu (https://hub.docker.com/r/codalab/default-gpu/tags?page=1&ordering=last_updated): 6.18 GB -> 7 GB

It might still be worth it to consider using multistage builds for default-gpu to save more space -- what do you think?

I think it might be fine for now, we can have a separate PR to do multistage.

epicfaace · 2021-06-03T19:11:52Z

Image size changes with this PR:
default-cpu (https://hub.docker.com/r/codalab/default-cpu/tags?page=1&ordering=last_updated): 5.1 GB -> 1.63 GB 🎉
default-gpu (https://hub.docker.com/r/codalab/default-gpu/tags?page=1&ordering=last_updated): 6.18 GB -> 7 GB

It might still be worth it to consider using multistage builds for default-gpu to save more space -- what do you think?

FYI @percyliang -- default-cpu size decreased because of optimizations / removing Python 2. default-gpu size increased because we upgraded ubuntu (which required us to upgrade cuda, which is much larger than the earlier version).

We could decrease the size of both images, especially default-gpu, by using docker multi-stage builds. However, we decided that this is beyond the scope of this PR and could do so in another PR.

epicfaace · 2021-06-03T19:21:17Z

docker_config/dockerfiles/Dockerfile.default-gpu

-# Newer pip dropped the support for python 2 so we need to specify the version.
-RUN python -m pip install --upgrade "pip < 21.0"
+
+## Python packages


Remove this line (duplicated from above)

Thanks, it has been removed.

jzwang43 added 2 commits May 26, 2021 22:58

update images

820b8f0

clean up

34a983e

jzwang43 changed the title ~~Update and make default-cpu / gpu dockerfiles smalle~~ Update and make default-cpu / gpu dockerfiles smaller May 27, 2021

jzwang43 marked this pull request as ready for review May 27, 2021 06:04

jzwang43 marked this pull request as draft May 27, 2021 06:04

Merge branch 'master' into update-images

0a80a7f

clean up install

f83cf88

jzwang43 marked this pull request as ready for review May 28, 2021 02:57

epicfaace changed the title ~~Update and make default-cpu / gpu dockerfiles smaller~~ Update and make default-cpu / gpu dockerfiles smaller, remove python 2 May 31, 2021

epicfaace reviewed May 31, 2021

View reviewed changes

jzwang43 added 2 commits May 31, 2021 15:22

remove python 2

7a381f4

fix images

44d3cda

epicfaace reviewed May 31, 2021

View reviewed changes

jzwang43 added 5 commits May 31, 2021 16:08

clean up images

7932a13

fix typo

d112b5b

add no cache

b8df1fc

fix gpu build

4c9d7f5

fix gpu

9d49a51

free disk space

fa0d357

epicfaace reviewed Jun 2, 2021

View reviewed changes

.github/workflows/build_worker_images.yml Show resolved Hide resolved

add more cleanup

1d18601

epicfaace reviewed Jun 3, 2021

View reviewed changes

epicfaace changed the title ~~Update and make default-cpu / gpu dockerfiles smaller, remove python 2~~ Optimize size of default-cpu / gpu dockerfiles, update to latest ubuntu and cuda, remove python 2 Jun 3, 2021

fix version

b4cc9d2

jzwang43 force-pushed the update-images branch from 0bc1849 to b4cc9d2 Compare June 3, 2021 18:51

epicfaace approved these changes Jun 3, 2021

View reviewed changes

jzwang43 added 2 commits June 3, 2021 21:41

remove extra comment

eed4025

Merge branch 'master' into update-images

2459527

jzwang43 added the ready-to-merge label Jun 4, 2021

Merge branch 'master' into update-images

129238f

epicfaace merged commit be644d3 into master Jun 4, 2021

epicfaace deleted the update-images branch June 4, 2021 14:42

jzwang43 mentioned this pull request Jun 8, 2021

Bump to 0.5.54 #3619

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize size of default-cpu / gpu dockerfiles, update to latest ubuntu and cuda, remove python 2 #3557

Optimize size of default-cpu / gpu dockerfiles, update to latest ubuntu and cuda, remove python 2 #3557

jzwang43 commented May 27, 2021 •

edited

Loading

epicfaace commented May 27, 2021

jzwang43 commented May 27, 2021 •

edited

Loading

epicfaace commented May 28, 2021

jzwang43 commented May 28, 2021 •

edited

Loading

epicfaace May 31, 2021

epicfaace May 31, 2021

epicfaace May 31, 2021

epicfaace May 31, 2021

epicfaace May 31, 2021

jzwang43 commented Jun 1, 2021 •

edited

Loading

epicfaace commented Jun 2, 2021

epicfaace Jun 3, 2021

jzwang43 Jun 3, 2021

epicfaace commented Jun 3, 2021 •

edited

Loading

jzwang43 commented Jun 3, 2021

epicfaace commented Jun 3, 2021 •

edited

Loading

epicfaace Jun 3, 2021

jzwang43 Jun 4, 2021

		@@ -60,26 +54,11 @@ RUN wget http://scala-lang.org/files/archive/scala-$SCALA_VERSION.deb && \
		apt-get clean && \
		apt-get autoremove && \

Optimize size of default-cpu / gpu dockerfiles, update to latest ubuntu and cuda, remove python 2 #3557

Optimize size of default-cpu / gpu dockerfiles, update to latest ubuntu and cuda, remove python 2 #3557

Conversation

jzwang43 commented May 27, 2021 • edited Loading

Reasons for making this change

Related issues

Screenshots

Checklist

epicfaace commented May 27, 2021

jzwang43 commented May 27, 2021 • edited Loading

epicfaace commented May 28, 2021

jzwang43 commented May 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jzwang43 commented Jun 1, 2021 • edited Loading

epicfaace commented Jun 2, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epicfaace commented Jun 3, 2021 • edited Loading

jzwang43 commented Jun 3, 2021

epicfaace commented Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jzwang43 commented May 27, 2021 •

edited

Loading

jzwang43 commented May 27, 2021 •

edited

Loading

jzwang43 commented May 28, 2021 •

edited

Loading

jzwang43 commented Jun 1, 2021 •

edited

Loading

epicfaace commented Jun 3, 2021 •

edited

Loading

epicfaace commented Jun 3, 2021 •

edited

Loading