-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Explicitly pin busybox and debian downloads #3701
Conversation
This will also protect against the inevitable breakage when I finally merge debuerreotype/docker-debian-artifacts#186 |
(This also pins to the older |
4c85afc
to
ef0bec2
Compare
I'd rather prefer to use skopeo or something so that we only need to care the single digest of the multi-platform index on an OCI registry |
I agree! 😅
|
Theoretically, I agree. Practically, we need to use two tools ( This was one of the reasons for #2741. Perhaps the best way is to create a program that will generate |
Something more like this? #!/usr/bin/env bash
set -Eeuo pipefail
images=(
# pinned to an older BusyBox (prior to 1.36 becoming "latest") because 1.36.0 has some unresolved bugs, especially around sha256sum
'https://github.com/docker-library/official-images/raw/eaed422a86b43c885a0f980d48f4bbf346086a4a/library/busybox:glibc'
# pinned to an older Debian Buster which has more architectures than the latest does (Buster transitioned from the Debian Security Team to the LTS Team which supports a smaller set)
'https://github.com/docker-library/official-images/raw/ce10f6b60289c0c0b5de6f785528b8725f225a58/library/debian:buster-slim'
)
bashbrew cat --format '
{{- "\n\n" -}}
{{- "case $goarch in\n" -}}
{{- range .TagEntry.Architectures -}}
{{- $repo := $.TagEntry.ArchGitRepo . | trimSuffixes ".git" -}}
{{- $branch := $.TagEntry.ArchGitFetch . | trimPrefixes "refs/heads/" -}}
{{- $commit := $.TagEntry.ArchGitCommit . -}}
{{- $dir := $.TagEntry.ArchDirectory . -}}
{{- $tarball := eq $.RepoName "debian" | ternary "rootfs.tar.xz" "busybox.tar.xz" -}}
{{ . | replace "arm64v8" "arm64" "arm32" "arm" "i386" "386" }} {{- ")\n" -}}
{{- "\t" -}}# {{ $repo }}/tree/{{ $branch }}{{- "\n" -}}
{{- "\t" -}}# {{ $repo }}/tree/{{ $commit }}/{{ $dir }}{{- "\n" -}}
{{- "\t" -}} {{- $.RepoName -}}_url="{{ $repo }}/raw/{{ $commit }}/{{ $dir }}/{{ $tarball }}"{{- "\n" -}}
{{- "\t" -}} ;; {{- "\n" -}}
{{- "\n" -}}
{{- end -}}
*) echo >&2 "error: unsupported architecture"; exit 1 ;;{{- "\n" -}}
{{- "esac\n" -}}
' "${images[@]}" Whose output looks like this: case $goarch in
amd64)
# https://github.com/docker-library/busybox/tree/dist-amd64
# https://github.com/docker-library/busybox/tree/31d342ad033e27c18723a516a2274ab39547be27/stable/glibc
busybox_url="https://github.com/docker-library/busybox/raw/31d342ad033e27c18723a516a2274ab39547be27/stable/glibc/busybox.tar.xz"
;;
armv5)
# https://github.com/docker-library/busybox/tree/dist-arm32v5
# https://github.com/docker-library/busybox/tree/96ea82ea25565f78b50bd032d5768d64985d6e11/stable/glibc
busybox_url="https://github.com/docker-library/busybox/raw/96ea82ea25565f78b50bd032d5768d64985d6e11/stable/glibc/busybox.tar.xz"
;;
armv7)
# https://github.com/docker-library/busybox/tree/dist-arm32v7
# https://github.com/docker-library/busybox/tree/5cb6c347469e86e4468e5e248de751b3598bb577/stable/glibc
busybox_url="https://github.com/docker-library/busybox/raw/5cb6c347469e86e4468e5e248de751b3598bb577/stable/glibc/busybox.tar.xz"
;;
arm64)
# https://github.com/docker-library/busybox/tree/dist-arm64v8
# https://github.com/docker-library/busybox/tree/94c664b5ca464546266bce54be0082874a44c7b2/stable/glibc
busybox_url="https://github.com/docker-library/busybox/raw/94c664b5ca464546266bce54be0082874a44c7b2/stable/glibc/busybox.tar.xz"
;;
386)
# https://github.com/docker-library/busybox/tree/dist-i386
# https://github.com/docker-library/busybox/tree/461a473aef31b7726ea99909a24551bf44565c05/stable/glibc
busybox_url="https://github.com/docker-library/busybox/raw/461a473aef31b7726ea99909a24551bf44565c05/stable/glibc/busybox.tar.xz"
;;
mips64le)
# https://github.com/docker-library/busybox/tree/dist-mips64le
# https://github.com/docker-library/busybox/tree/47f73f7c735dcd6760a976bfe0012d251b6ef0a9/stable/glibc
busybox_url="https://github.com/docker-library/busybox/raw/47f73f7c735dcd6760a976bfe0012d251b6ef0a9/stable/glibc/busybox.tar.xz"
;;
ppc64le)
# https://github.com/docker-library/busybox/tree/dist-ppc64le
# https://github.com/docker-library/busybox/tree/9ca13bc214717966383cf97e08606b444b7300e4/stable/glibc
busybox_url="https://github.com/docker-library/busybox/raw/9ca13bc214717966383cf97e08606b444b7300e4/stable/glibc/busybox.tar.xz"
;;
s390x)
# https://github.com/docker-library/busybox/tree/dist-s390x
# https://github.com/docker-library/busybox/tree/a03814d21bcf97767121bb9422a742ec237a09e2/stable/glibc
busybox_url="https://github.com/docker-library/busybox/raw/a03814d21bcf97767121bb9422a742ec237a09e2/stable/glibc/busybox.tar.xz"
;;
*) echo >&2 "error: unsupported architecture"; exit 1 ;;
esac
case $goarch in
amd64)
# https://github.com/debuerreotype/docker-debian-artifacts/tree/dist-amd64
# https://github.com/debuerreotype/docker-debian-artifacts/tree/686d9f6eaada08a754bc7abf6f6184c65c5b378f/buster/slim
debian_url="https://github.com/debuerreotype/docker-debian-artifacts/raw/686d9f6eaada08a754bc7abf6f6184c65c5b378f/buster/slim/rootfs.tar.xz"
;;
armv5)
# https://github.com/debuerreotype/docker-debian-artifacts/tree/dist-arm32v5
# https://github.com/debuerreotype/docker-debian-artifacts/tree/155640b6e2e249dfaeee8795d5de539ef3e49417/buster/slim
debian_url="https://github.com/debuerreotype/docker-debian-artifacts/raw/155640b6e2e249dfaeee8795d5de539ef3e49417/buster/slim/rootfs.tar.xz"
;;
armv7)
# https://github.com/debuerreotype/docker-debian-artifacts/tree/dist-arm32v7
# https://github.com/debuerreotype/docker-debian-artifacts/tree/60ff0c2c6ce9556e5d8a2758dd2b3f3731716a6f/buster/slim
debian_url="https://github.com/debuerreotype/docker-debian-artifacts/raw/60ff0c2c6ce9556e5d8a2758dd2b3f3731716a6f/buster/slim/rootfs.tar.xz"
;;
arm64)
# https://github.com/debuerreotype/docker-debian-artifacts/tree/dist-arm64v8
# https://github.com/debuerreotype/docker-debian-artifacts/tree/2f108af35e22064c848b8628a7cac56192246dba/buster/slim
debian_url="https://github.com/debuerreotype/docker-debian-artifacts/raw/2f108af35e22064c848b8628a7cac56192246dba/buster/slim/rootfs.tar.xz"
;;
386)
# https://github.com/debuerreotype/docker-debian-artifacts/tree/dist-i386
# https://github.com/debuerreotype/docker-debian-artifacts/tree/e4db8aa97f4366e6f27ddbdeaed0773fe0288d47/buster/slim
debian_url="https://github.com/debuerreotype/docker-debian-artifacts/raw/e4db8aa97f4366e6f27ddbdeaed0773fe0288d47/buster/slim/rootfs.tar.xz"
;;
mips64le)
# https://github.com/debuerreotype/docker-debian-artifacts/tree/dist-mips64le
# https://github.com/debuerreotype/docker-debian-artifacts/tree/e28cbd76dcfba10446b1722aebb5a996121e3d27/buster/slim
debian_url="https://github.com/debuerreotype/docker-debian-artifacts/raw/e28cbd76dcfba10446b1722aebb5a996121e3d27/buster/slim/rootfs.tar.xz"
;;
ppc64le)
# https://github.com/debuerreotype/docker-debian-artifacts/tree/dist-ppc64le
# https://github.com/debuerreotype/docker-debian-artifacts/tree/3ba08903ca3fd48fe59ba92b02744a2f5d4d9d6f/buster/slim
debian_url="https://github.com/debuerreotype/docker-debian-artifacts/raw/3ba08903ca3fd48fe59ba92b02744a2f5d4d9d6f/buster/slim/rootfs.tar.xz"
;;
s390x)
# https://github.com/debuerreotype/docker-debian-artifacts/tree/dist-s390x
# https://github.com/debuerreotype/docker-debian-artifacts/tree/2fddbf8fe632fc5865b140341b68a1358586fff2/buster/slim
debian_url="https://github.com/debuerreotype/docker-debian-artifacts/raw/2fddbf8fe632fc5865b140341b68a1358586fff2/buster/slim/rootfs.tar.xz"
;;
*) echo >&2 "error: unsupported architecture"; exit 1 ;;
esac Another alternative would be something like https://github.com/moby/moby/blob/ed8782fe0aa11dcc92f84cdcd1f9896d6d035328/contrib/download-frozen-image-v2.sh to download the rootfs layers from Hub directly, but that's a bit more fiddly (and really should probably be using something like |
@tianon using Can you finish this (I guess have something |
Unfortunately there's no checksum data available in I'll update this as soon as I can. 👍 |
ef0bec2
to
cf0d7ab
Compare
Ok, updated -- I tried to keep the general end result structured similarly to the existing |
81995f7
to
e73e984
Compare
Ok, I think I've finally satisfied all the linters and it appears to be working correctly. 👍 🎉 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎉 LGTM
1cb3369
to
af634b9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
still LGTM
|
||
cat <<'EOH' | ||
#!/bin/bash | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for being slow today, might make sense to add a note, something like
# DO NOT EDIT! Generated by $*.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No problem 😄
I opted to put the script name explicitly so we don't have to switch from <<'EOH'
back to <<EOH
and thus escape-all-the-things 😅
0813d93
to
8c8449f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@AkihiroSuda @thaJeztah PTAL
# that need to ensure the images are downloaded. Its output is suitable | ||
# for consumption by shell via eval (see helpers.bash). | ||
# | ||
# This script is generated via "bootstrap-get-images.sh" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: redundant
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated 👍
cat <<'EOH' | ||
#!/bin/bash | ||
|
||
# DO NOT EDIT! Generated by "bootstrap-get-images.sh" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe CI should validate that the file is not modified?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The whole point of having this bootstrap script is that we do not need extra dependencies in CI (bashbrew in the current implementation).
OTOH it can be a separate small job, let me take a look.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
awesome!
@AkihiroSuda test added, PTAL
8c8449f
to
86ff0b6
Compare
Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
This is to check that tests/integration/get-images.sh is in sync with tests/integration/bootstrap-get-images.sh. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
86ff0b6
to
3fbc5ba
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Closes #3699
Alternative to #3700
This pins to an older
debian:buster-slim
, back before it moved to LTS and lost many of the supported architectures.Gory details of how I generated these before munging by hand: