Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update docker images to share scripts when possible #42201

Closed
3 of 35 tasks
malbarbo opened this issue May 24, 2017 · 6 comments · Fixed by #43129
Closed
3 of 35 tasks

Update docker images to share scripts when possible #42201

malbarbo opened this issue May 24, 2017 · 6 comments · Fixed by #43129
Labels
E-easy Call for participation: Easy difficulty. Experience needed to fix: Not much. Good first issue. E-help-wanted Call for participation: Help is requested to fix this issue. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Comments

@malbarbo
Copy link
Contributor

malbarbo commented May 24, 2017 •
edited by alexcrichton
Loading

#41958 added the possibility of sharing scripts used in docker images. This allows, for example, to use the same script to install sccache in all images, consequently updating sccache can be done updating only one file.

We should update all images to share scripts when possible.

To update an image, one can follow these sequence of steps:

  • Choose an unchecked image in the below list;
  • Replace long RUN lines in the dockerfile of the image with scripts (like sccache.sh and dum-init.sh in src/ci/docker/scripts/, see for example the updates of arm-android and dist-android). You can organize the dockerfile in sections to make it easy to follow. sccache should be installed last. This avoids the whole container rebuild when sscache is updated;
  • See if files in COPY lines are duplicated in other images (duplicated files can be found using the command fdupes -r src/ci/). If there are duplicated files, move then to src/ci/docker/scripts/ and update the docker files;
  • Test with the command src/ci/docker/run.sh image with the modified image still works (docker is required).

List of images (unchecked images need to be updated):

  • arm-android
  • armhf-gnu
  • cross
  • dist-aarch64-linux
  • dist-android
  • dist-armhf-linux
  • dist-arm-linux
  • dist-armv7-linux
  • dist-fuchsia
  • dist-i586-gnu-i686-musl
  • dist-i686-freebsd
  • dist-i686-linux
  • dist-mips64el-linux
  • dist-mips64-linux
  • dist-mipsel-linux
  • dist-mips-linux
  • dist-powerpc64le-linux
  • dist-powerpc64-linux
  • dist-powerpc-linux
  • dist-s390x-linux
  • dist-x86_64-freebsd
  • dist-x86_64-linux
  • dist-x86_64-musl
  • dist-x86_64-netbsd
  • emscripten
  • i686-gnu
  • i686-gnu-nopt
  • x86_64-gnu
  • x86_64-gnu-aux
  • x86_64-gnu-debug
  • x86_64-gnu-distcheck
  • x86_64-gnu-full-bootstrap
  • x86_64-gnu-incremental
  • x86_64-gnu-llvm-3.7
  • x86_64-gnu-nopt
@malbarbo
Copy link
Contributor Author

cc @alexcrichton

@alexcrichton alexcrichton added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) E-help-wanted Call for participation: Help is requested to fix this issue. labels May 24, 2017
@alexcrichton
Copy link
Member

Thanks for writing this up @malbarbo! If you're interested, want to write out some detailed steps for how to accomplish this? If we've got that listed we could likely list this as E-easy which would probably help get the work items here picked off very quickly!

@malbarbo
Copy link
Contributor Author

@alexcrichton I wrote the steps. What do you think? Is it enough?

@alexcrichton alexcrichton added the E-easy Call for participation: Easy difficulty. Experience needed to fix: Not much. Good first issue. label May 25, 2017
@alexcrichton
Copy link
Member

Looks great! If anyone's got any follow-up questions please feel free to ask here!

@malbarbo I think there were some examples of this in a previous PR of yours as well, mind linking those here?

@venkatagiri venkatagiri mentioned this issue May 25, 2017
Merged
@malbarbo
Copy link
Contributor Author

@alexcrichton I linked the android dockerfile updates.

@alexcrichton
Copy link
Member

Looks like @venkatagiri has taken care of the cross image!

Mark-Simulacrum added a commit to Mark-Simulacrum/rust that referenced this issue May 26, 2017
@Mark-Simulacrum
Rollup merge of rust-lang#42218 - venkatagiri:update_cross, r=alexcri…
ab6c565 
…chton

use shared scripts for init and sccache in cross image

cc rust-lang#42201
cc @malbarbo
Mark-Simulacrum added a commit to Mark-Simulacrum/rust that referenced this issue May 27, 2017
@Mark-Simulacrum
Rollup merge of rust-lang#42218 - venkatagiri:update_cross, r=alexchr…
0b4e7b4 
…ichton

use shared scripts for init and sccache in cross image

cc rust-lang#42201
cc @malbarbo
bors added a commit that referenced this issue May 28, 2017
@bors
Auto merge of #42218 - venkatagiri:update_cross, r=alexchrichton
920c479 
use shared scripts for init and sccache in cross image

cc #42201
cc @malbarbo
@Ophirr33 Ophirr33 mentioned this issue Jul 8, 2017
Merged
bors added a commit that referenced this issue Jul 13, 2017
@bors
Auto merge of #43129 - Ophirr33:master, r=alexcrichton
06ffdeb 
Updated docker images to share scripts

Attempts to resolve #42201. I managed to pull out five scripts (android-base-apt-get, ubuntu16-apt-get, make3, rustbuild-setup, and crosstool-ng). Let me know if there's more I can do or if I should change some names.
r? @malbarbo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
E-easy Call for participation: Easy difficulty. Experience needed to fix: Not much. Good first issue. E-help-wanted Call for participation: Help is requested to fix this issue. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Updated docker images to share scripts Ophirr33/rust
2 participants
@alexcrichton @malbarbo