Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1774642: pkg/cli/admin/release/extract: Multi-arch extraction with --filter-by-os #172

Closed

Conversation

wking
Copy link
Member

@wking wking commented Nov 20, 2019

Avoid:

$ oc adm release extract --tools registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
error: image did not contain usr/share/openshift/mac/oc

by avoiding the hardlink paths and teaching extract about the new canonical paths from e949088 (#153).

@openshift-ci-robot
Copy link

@wking: This pull request references Bugzilla bug 1774642, which is invalid:

  • expected the bug to target the "4.3.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1774642: pkg/cli/admin/release/extract_tools: Extract oc from standard locations

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 20, 2019
@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: wking
To complete the pull request process, please assign smarterclayton
You can assign the PR to them by writing /assign @smarterclayton in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@wking
Copy link
Member Author

wking commented Nov 20, 2019

$ git describe
openshift-clients-4.3.0-201910250623-67-g9210429b0
$ go build -o ocx ./cmd/oc && ./ocx adm release extract --command=oc --command-os=mac registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
error: image did not contain /usr/share/openshift/darwin_amd64/oc
$ oc adm release info --image-for=cli-artifacts registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
registry.svc.ci.openshift.org/ocp/4.3-2019-11-20-121416@sha256:6497d5cb7102903baf98bbc0e07144f04827a21dcd800ff61360d25228a2d2ce
$ podman pull registry.svc.ci.openshift.org/ocp/4.3-2019-11-20-121416@sha256:6497d5cb7102903baf98bbc0e07144f04827a21dcd800ff61360d25228a2d2ce
$ podman save --format oci-dir --output cli-artifacts registry.svc.ci.openshift.org/ocp/4.3-2019-11-20-121416@sha256:6497d5cb7102903baf98bbc0e07144f04827a21dcd800ff61360d25228a2d2ce
$ jq -r '.layers[].digest' cli-artifacts/manifest.json
sha256:2cab4440f9070965c8454792eee4b122bce5939cc6d21e554469f01e71712a89
sha256:64f2bd9f473b0ae150f07dc65aafb90beeec7ebaa722625055cb8c1ddede0b09
sha256:bd48b9cc9f65a873bde992c97649ace2895af41307502429527f35f1bebda2c0
sha256:86e61694888a7971fcd711f84b1eff117844813adb4ed388d34bdb2d347fd33d
sha256:915b7fb689e0bc593393a11994653ab633fbb9e671276a9e75be478bea24857a
sha256:a18473fcedd3f809beb23c3f491bf4d26548637b1acd354e85023e9f7c6f9666
$ tar -tf cli-artifacts/a18473fcedd3f809beb23c3f491bf4d26548637b1acd354e85023e9f7c6f9666 
usr/
usr/share/
usr/share/openshift/
usr/share/openshift/.wh..wh..opq
usr/share/openshift/darwin_amd64/
usr/share/openshift/darwin_amd64/oc
usr/share/openshift/linux_amd64/
usr/share/openshift/linux_amd64/oc
usr/share/openshift/linux_arm64/
usr/share/openshift/linux_arm64/oc
usr/share/openshift/linux_ppc64le/
usr/share/openshift/linux_ppc64le/oc
usr/share/openshift/linux_s390x/
usr/share/openshift/linux_s390x/oc
usr/share/openshift/mac/
usr/share/openshift/mac/oc
usr/share/openshift/windows/
usr/share/openshift/windows/oc.exe
usr/share/openshift/windows_amd64/
usr/share/openshift/windows_amd64/oc.exe

Ah, maybe it's the leading slash...

@wking wking force-pushed the extract-from-consistent-locations branch from 9210429 to d74a65a Compare November 20, 2019 17:46
@wking
Copy link
Member Author

wking commented Nov 20, 2019

Pushed 9210429 -> d74a65a to drop the leading slash. oc adm release extract --command=oc --command-os=mac registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416 works now. Also set a Target Release on the bug:

/bugzilla refresh

@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Nov 20, 2019
@openshift-ci-robot
Copy link

@wking: This pull request references Bugzilla bug 1774642, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Pushed 9210429 -> d74a65a to drop the leading slash. oc adm release extract --command=oc --command-os=mac registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416 works now. Also set a Target Release on the bug:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@wking
Copy link
Member Author

wking commented Nov 20, 2019

$ git describe
openshift-clients-4.3.0-201910250623-67-gd74a65aad
$ go build -o ocx ./cmd/oc && ./ocx adm release extract --command=oc --command-os='*' registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
error: unable to iterate over layer sha256:13caf755813923e69e25ff1a28cc65766d6dcaa12676e5e274db9ec828e55d71 from registry.svc.ci.openshift.org/ocp/4.3-2019-11-20-121416@sha256:6497d5cb7102903baf98bbc0e07144f04827a21dcd800ff61360d25228a2d2ce: unable to find target with mapping name openshift-client-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz

Still trying to figure this one out...

wking added a commit to wking/oc that referenced this pull request Nov 20, 2019
Also teach oc to extract from standard locations e949088 (Enable all
Linux arches in cli-artifacts, 2019-11-07, openshift#153) to avoid choking on
hardlinks [1]:

  $ oc adm release extract --tools registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: image did not contain usr/share/openshift/mac/oc

Teaching extractTarget about architectures avoids conflicts with
multiple architectures trying to use the no-longer-specific-enough
oc-linux in targetsByName [2]:

  $ ./oc adm release extract --command=oc --command-os='*' registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: unable to iterate over layer sha256:13caf755813923e69e25ff1a28cc65766d6dcaa12676e5e274db9ec828e55d71 from registry.svc.ci.openshift.org/ocp/4.3-2019-11-20-121416@sha256:6497d5cb7102903baf98bbc0e07144f04827a21dcd800ff61360d25228a2d2ce: unable to find target with mapping name openshift-client-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz

Adding --command-arch on top of that allows us to set currentArch to
avoid extracting 'oc' for multiple architectures all into the same
file (making it unlikely that you get the architecture you want ;).

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774642
[2]: openshift#172 (comment)
@wking wking force-pushed the extract-from-consistent-locations branch from d74a65a to 5a3452f Compare November 20, 2019 19:06
@openshift-ci-robot openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 20, 2019
@wking
Copy link
Member Author

wking commented Nov 20, 2019

Still trying to figure this one out...

I needed arch filtering. Added that via --command-arch with d74a65a -> 5a3452f.

@wking wking changed the title Bug 1774642: pkg/cli/admin/release/extract_tools: Extract oc from standard locations Bug 1774642: pkg/cli/admin/release/extract: Multi-arch extraction with --command-arch Nov 20, 2019
wking added a commit to wking/oc that referenced this pull request Nov 20, 2019
Also teach oc to extract from standard locations e949088 (Enable all
Linux arches in cli-artifacts, 2019-11-07, openshift#153) to avoid choking on
hardlinks [1]:

  $ oc adm release extract --tools registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: image did not contain usr/share/openshift/mac/oc

Teaching extractTarget about architectures avoids conflicts with
multiple architectures trying to use the no-longer-specific-enough
oc-linux in targetsByName [2]:

  $ ./oc adm release extract --command=oc --command-os='*' registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: unable to iterate over layer sha256:13caf755813923e69e25ff1a28cc65766d6dcaa12676e5e274db9ec828e55d71 from registry.svc.ci.openshift.org/ocp/4.3-2019-11-20-121416@sha256:6497d5cb7102903baf98bbc0e07144f04827a21dcd800ff61360d25228a2d2ce: unable to find target with mapping name openshift-client-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz

Adding --command-arch on top of that allows us to set currentArch to
avoid extracting 'oc' for multiple architectures all into the same
file (making it unlikely that you get the architecture you want ;).

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774642
[2]: openshift#172 (comment)
@wking wking force-pushed the extract-from-consistent-locations branch from 5a3452f to f46e4c4 Compare November 20, 2019 19:15
@wking
Copy link
Member Author

wking commented Nov 20, 2019

$ git describe
openshift-clients-4.3.0-201910250623-67-gf46e4c422
$ go build -o ocx ./cmd/oc
$ ./ocx adm release extract --tools registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
$ cat sha256sum.txt 
3921c887913e19c0c0fe6336dc7e7fb234fb12fdf4e67114a8980ba010379326  openshift-client-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz
5d9d614915852828d959341c4c765b38d38fd1998a8dfff11b384767316f2372  openshift-install-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz
d115798f2647790b372efa9b33c26870aecc4bf2d311547233e1a099a8422e47  release.txt
$ ./ocx adm release extract --tools --command-arch=s390x registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
$ cat sha256sum.txt 
9918a2617b36fe6d09682c687f459bac3dc167a9a1c658baeb314d57b77943aa  openshift-client-linux-s390x-4.3.0-0.ci-2019-11-20-121416.tar.gz
d115798f2647790b372efa9b33c26870aecc4bf2d311547233e1a099a8422e47  release.txt
$ ./ocx adm release extract --tools --command-arch='*' registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
$ cat sha256sum.txt 
3921c887913e19c0c0fe6336dc7e7fb234fb12fdf4e67114a8980ba010379326  openshift-client-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz
124c365add2c0745e6d2cd6ea6b435db9d397e3b6d7e9f7aa8e7b9cd928b56a6  openshift-client-linux-arm64-4.3.0-0.ci-2019-11-20-121416.tar.gz
3b7641d90d814e54aa3617a5a3078ba63c004060b5a6a713f52f228c6c03cd32  openshift-client-linux-ppc64le-4.3.0-0.ci-2019-11-20-121416.tar.gz
9918a2617b36fe6d09682c687f459bac3dc167a9a1c658baeb314d57b77943aa  openshift-client-linux-s390x-4.3.0-0.ci-2019-11-20-121416.tar.gz
5d9d614915852828d959341c4c765b38d38fd1998a8dfff11b384767316f2372  openshift-install-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz
d115798f2647790b372efa9b33c26870aecc4bf2d311547233e1a099a8422e47  release.txt
$ ./ocx adm release extract --tools --command-os=windows --command-arch=amd64 registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416error: image did not contain usr/share/openshift/windows_amd64/oc.exe
error: image did not contain usr/share/openshift/windows_amd64/oc.exe

So close :p

@wking
Copy link
Member Author

wking commented Nov 20, 2019

$ tar -tvf cli-artifacts/a18473fcedd3f809beb23c3f491bf4d26548637b1acd354e85023e9f7c6f9666 
drwxr-xr-x 0/0               0 2019-10-08 05:41 usr/
drwxr-xr-x 0/0               0 2019-11-18 20:35 usr/share/
drwxr-xr-x 0/0               0 2019-11-18 20:35 usr/share/openshift/
-rwxr-xr-x 0/0               0 1969-12-31 16:00 usr/share/openshift/.wh..wh..opq
drwxr-xr-x 0/0               0 2019-11-18 20:31 usr/share/openshift/darwin_amd64/
-rwxr-xr-x 0/0        93382160 2019-11-18 20:31 usr/share/openshift/darwin_amd64/oc
drwxr-xr-x 0/0               0 2019-11-18 20:33 usr/share/openshift/linux_amd64/
-rwxr-xr-x 0/0        82720408 2019-11-18 20:33 usr/share/openshift/linux_amd64/oc
drwxr-xr-x 0/0               0 2019-11-18 20:34 usr/share/openshift/linux_arm64/
-rwxr-xr-x 0/0        77812608 2019-11-18 20:34 usr/share/openshift/linux_arm64/oc
drwxr-xr-x 0/0               0 2019-11-18 20:35 usr/share/openshift/linux_ppc64le/
-rwxr-xr-x 0/0        79059232 2019-11-18 20:35 usr/share/openshift/linux_ppc64le/oc
drwxr-xr-x 0/0               0 2019-11-18 20:35 usr/share/openshift/linux_s390x/
-rwxr-xr-x 0/0        81223040 2019-11-18 20:35 usr/share/openshift/linux_s390x/oc
drwxr-xr-x 0/0               0 2019-11-18 20:35 usr/share/openshift/mac/
hrwxr-xr-x 0/0               0 2019-11-18 20:31 usr/share/openshift/mac/oc link to usr/share/openshift/darwin_amd64/oc
drwxr-xr-x 0/0               0 2019-11-18 20:35 usr/share/openshift/windows/
-rwxr-xr-x 0/0        81619968 2019-11-18 20:32 usr/share/openshift/windows/oc.exe
drwxr-xr-x 0/0               0 2019-11-18 20:32 usr/share/openshift/windows_amd64/
hrwxr-xr-x 0/0               0 2019-11-18 20:32 usr/share/openshift/windows_amd64/oc.exe link to usr/share/openshift/windows/oc.exe

wking added a commit to wking/oc that referenced this pull request Nov 20, 2019
Also teach oc to extract from standard locations e949088 (Enable all
Linux arches in cli-artifacts, 2019-11-07, openshift#153) to avoid choking on
hardlinks [1]:

  $ oc adm release extract --tools registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: image did not contain usr/share/openshift/mac/oc

Teaching extractTarget about architectures avoids conflicts with
multiple architectures trying to use the no-longer-specific-enough
oc-linux in targetsByName [2]:

  $ ./oc adm release extract --command=oc --command-os='*' registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: unable to iterate over layer sha256:13caf755813923e69e25ff1a28cc65766d6dcaa12676e5e274db9ec828e55d71 from registry.svc.ci.openshift.org/ocp/4.3-2019-11-20-121416@sha256:6497d5cb7102903baf98bbc0e07144f04827a21dcd800ff61360d25228a2d2ce: unable to find target with mapping name openshift-client-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz

Adding --command-arch on top of that allows us to set currentArch to
avoid extracting 'oc' for multiple architectures all into the same
file (making it unlikely that you get the architecture you want ;).

Updated the completions with:

  $ make build
  $ hack/update-generated-completions.sh

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774642
[2]: openshift#172 (comment)
@wking wking force-pushed the extract-from-consistent-locations branch from f46e4c4 to c60223d Compare November 20, 2019 19:40
@wking
Copy link
Member Author

wking commented Nov 20, 2019

$ git describe
openshift-clients-4.3.0-201910250623-67-gc60223d63
$ go build -o ocx ./cmd/oc
$ ./ocx adm release extract --tools --command-os=windows --command-arch=amd64 registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
$ cat sha256sum.txt 
7e6b446bc63a640b53a95e0a6f0b5843f88f0f1fff5ff7446e94401208a3d681  openshift-client-windows-4.3.0-0.ci-2019-11-20-121416.tar.gz
d115798f2647790b372efa9b33c26870aecc4bf2d311547233e1a099a8422e47  release.txt
$ go build -o ocx ./cmd/oc && ./ocx adm release extract --tools --command-os='*' --command-arch='*' registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
$ cat sha256sum.txt 
3921c887913e19c0c0fe6336dc7e7fb234fb12fdf4e67114a8980ba010379326  openshift-client-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz
124c365add2c0745e6d2cd6ea6b435db9d397e3b6d7e9f7aa8e7b9cd928b56a6  openshift-client-linux-arm64-4.3.0-0.ci-2019-11-20-121416.tar.gz
3b7641d90d814e54aa3617a5a3078ba63c004060b5a6a713f52f228c6c03cd32  openshift-client-linux-ppc64le-4.3.0-0.ci-2019-11-20-121416.tar.gz
9918a2617b36fe6d09682c687f459bac3dc167a9a1c658baeb314d57b77943aa  openshift-client-linux-s390x-4.3.0-0.ci-2019-11-20-121416.tar.gz
0124abefcc68492d6cd61730d49c834c3eca2be8a1dee22520e74c8fc4ea4958  openshift-client-mac-4.3.0-0.ci-2019-11-20-121416.tar.gz
7e6b446bc63a640b53a95e0a6f0b5843f88f0f1fff5ff7446e94401208a3d681  openshift-client-windows-4.3.0-0.ci-2019-11-20-121416.tar.gz
5d9d614915852828d959341c4c765b38d38fd1998a8dfff11b384767316f2372  openshift-install-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz
57d9c5efaefa5730a2ab4dd2dc2d3615e1093b8332a34d7c816c9db008740f21  openshift-install-mac-4.3.0-0.ci-2019-11-20-121416.tar.gz
d115798f2647790b372efa9b33c26870aecc4bf2d311547233e1a099a8422e47  release.txt

So c60223d looks good to me.

wking added a commit to wking/oc that referenced this pull request Nov 20, 2019
Also teach oc to extract from standard locations e949088 (Enable all
Linux arches in cli-artifacts, 2019-11-07, openshift#153) to avoid choking on
hardlinks [1]:

  $ oc adm release extract --tools registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: image did not contain usr/share/openshift/mac/oc

Teaching extractTarget about architectures avoids conflicts with
multiple architectures trying to use the no-longer-specific-enough
oc-linux in targetsByName [2]:

  $ ./oc adm release extract --command=oc --command-os='*' registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: unable to iterate over layer sha256:13caf755813923e69e25ff1a28cc65766d6dcaa12676e5e274db9ec828e55d71 from registry.svc.ci.openshift.org/ocp/4.3-2019-11-20-121416@sha256:6497d5cb7102903baf98bbc0e07144f04827a21dcd800ff61360d25228a2d2ce: unable to find target with mapping name openshift-client-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz

Adding --command-arch on top of that allows us to set currentArch to
avoid extracting 'oc' for multiple architectures all into the same
file (making it unlikely that you get the architecture you want ;).

Updated the completions with:

  $ make build
  $ hack/update-generated-completions.sh

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774642
[2]: openshift#172 (comment)
@wking wking force-pushed the extract-from-consistent-locations branch from c60223d to 4820f6c Compare November 20, 2019 19:41
@wking
Copy link
Member Author

wking commented Nov 20, 2019

Hrm, I should probably reroll to deprecate --command-os and follow our existing --filter-by-os pattern.

/hold

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 20, 2019
wking added a commit to wking/oc that referenced this pull request Nov 21, 2019
Deprecating the previous --command-os to get consistency with other
image handling.  --filter-by-os originally landed in
openshift/origin@e04b16527b (cli: Mirror images across registries or
to S3, 2017-06-04, openshift/origin#14471).

The wrapping in:

  o.FilterOptions.FilterByOS = fmt.Sprintf("^%s/", o.CommandOperatingSystem)

guards against the unlikely case that a given --command-os value is a
valid prefix for a longer OS, or matches an arch or varient or some
such.

Also teach oc to extract from standard locations e949088 (Enable all
Linux arches in cli-artifacts, 2019-11-07, openshift#153) to avoid choking on
hardlinks [1]:

  $ oc adm release extract --tools registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: image did not contain usr/share/openshift/mac/oc

Teaching extractTarget about architectures avoids conflicts with
multiple architectures trying to use the no-longer-specific-enough
oc-linux in targetsByName [2]:

  $ ./oc adm release extract --command=oc --command-os='*' registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: unable to iterate over layer sha256:13caf755813923e69e25ff1a28cc65766d6dcaa12676e5e274db9ec828e55d71 from registry.svc.ci.openshift.org/ocp/4.3-2019-11-20-121416@sha256:6497d5cb7102903baf98bbc0e07144f04827a21dcd800ff61360d25228a2d2ce: unable to find target with mapping name openshift-client-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz

Adding --command-arch on top of that allows us to set currentArch to
avoid extracting 'oc' for multiple architectures all into the same
file (making it unlikely that you get the architecture you want ;).

Updated the completions with:

  $ make build
  $ hack/update-generated-completions.sh

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774642
[2]: openshift#172 (comment)
@wking wking force-pushed the extract-from-consistent-locations branch from 4820f6c to 88d9a93 Compare November 21, 2019 00:04
Deprecating the previous --command-os to get consistency with other
image handling.  --filter-by-os originally landed in
openshift/origin@e04b16527b (cli: Mirror images across registries or
to S3, 2017-06-04, openshift/origin#14471).

The wrapping in:

  o.FilterOptions.FilterByOS = fmt.Sprintf("^%s/", o.CommandOperatingSystem)

guards against the unlikely case that a given --command-os value is a
valid prefix for a longer OS, or matches an arch or varient or some
such.

Also teach oc to extract from standard locations e949088 (Enable all
Linux arches in cli-artifacts, 2019-11-07, openshift#153) to avoid choking on
hardlinks [1]:

  $ oc adm release extract --tools registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: image did not contain usr/share/openshift/mac/oc

Teaching extractTarget about architectures avoids conflicts with
multiple architectures trying to use the no-longer-specific-enough
oc-linux in targetsByName [2]:

  $ ./oc adm release extract --command=oc --command-os='*' registry.svc.ci.openshift.org/ocp/release:4.3.0-0.ci-2019-11-20-121416
  error: unable to iterate over layer sha256:13caf755813923e69e25ff1a28cc65766d6dcaa12676e5e274db9ec828e55d71 from registry.svc.ci.openshift.org/ocp/4.3-2019-11-20-121416@sha256:6497d5cb7102903baf98bbc0e07144f04827a21dcd800ff61360d25228a2d2ce: unable to find target with mapping name openshift-client-linux-4.3.0-0.ci-2019-11-20-121416.tar.gz

Adding --command-arch on top of that allows us to set currentArch to
avoid extracting 'oc' for multiple architectures all into the same
file (making it unlikely that you get the architecture you want ;).

Updated the completions with:

  $ make build
  $ hack/update-generated-completions.sh

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774642
[2]: openshift#172 (comment)
@wking wking force-pushed the extract-from-consistent-locations branch from 88d9a93 to 85df592 Compare November 21, 2019 00:05
@wking wking changed the title Bug 1774642: pkg/cli/admin/release/extract: Multi-arch extraction with --command-arch Bug 1774642: pkg/cli/admin/release/extract: Multi-arch extraction with --filter-by-os Nov 21, 2019
@wking
Copy link
Member Author

wking commented Nov 21, 2019

Rerolled to use --filter-by-os with 4820f6c -> 85df592.

/hold cancel

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 21, 2019
These are where the binaries lived before e949088 (Enable all Linux
arches in cli-artifacts, 2019-11-07, openshift#153), and since the extraction
chokes on hardlinks [1], we need to be looking at those locations to
extract from 4.2.z and older releases.

Also shuffles the Dockerfile around to move the darwin and Windows
builds to their old locations and symlink them from the standard
locations.  That way older 4.2.z releases will find regular files at
the locations they expect, but that we'll be able to serve files from
the standardized pattern in the downloads deployment [2].

I'm also removing the cli-artifacts-built linux/amd64 binary in favor
of the build we inherited from the parent cli image.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774642
[2]: openshift/console-operator#354
@wking
Copy link
Member Author

wking commented Nov 21, 2019

$ git describe
openshift-clients-4.3.0-201910250623-68-ge23baae6a
$ go build -o ocx ./cmd/oc
$ ./ocx adm release extract --tools --filter-by-os='.*' quay.io/openshift-release-dev/ocp-release:4.2.4
error: unable to find multiple files: usr/share/openshift/linux_arm64/oc, usr/share/openshift/linux_ppc64le/oc, usr/share/openshift/linux_s390x/oc

Grr, I need to reroll the Optional handling too.

/hold

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 21, 2019
With this command, we still error out if we didn't extract anything.
But if we got through the tarball without erroring on an extraction
and we got something out, just accept that missing files are because
the image being used didn't have some possible target, and that that's
ok.  Avoids:

  $ oc adm release extract --tools --filter-by-os='.*' quay.io/openshift-release-dev/ocp-release:4.2.4
  error: unable to find multiple files: usr/share/openshift/linux_arm64/oc, usr/share/openshift/linux_ppc64le/oc, usr/share/openshift/linux_s390x/oc

when we expect 4.2.4's cli-artifacts to be missing those new non-amd64
binaries.
@wking
Copy link
Member Author

wking commented Nov 21, 2019

Grr, I need to reroll the Optional handling too.

Optional (from 2449822, #57) isn't the greatest name; it's "should not be included in --tools" not "it's ok if extraction can't find this". I've pushed a90be83 to soften the "it's ok if extraction can't find this" checks.

@hongkailiu
Copy link
Member

/retest

@wking
Copy link
Member Author

wking commented Nov 21, 2019

/hold cancel

@openshift-ci-robot openshift-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Nov 21, 2019
@openshift-ci-robot
Copy link

@wking: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link

@wking: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-cmd a90be83 link /test e2e-cmd
ci/prow/verify-deps a90be83 link /test verify-deps

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-bot
Copy link
Contributor

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci-robot openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 27, 2020
@openshift-merge-robot
Copy link
Contributor

@wking: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-agnostic-cmd a90be83 link /test e2e-agnostic-cmd
ci/prow/e2e-aws-serial a90be83 link /test e2e-aws-serial

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-bot
Copy link
Contributor

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

@openshift-ci-robot openshift-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Dec 5, 2020
@openshift-bot
Copy link
Contributor

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

@openshift-ci-robot
Copy link

@openshift-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link

@wking: This pull request references Bugzilla bug 1774642. The bug has been updated to no longer refer to the pull request using the external bug tracker.

In response to this:

Bug 1774642: pkg/cli/admin/release/extract: Multi-arch extraction with --filter-by-os

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants