Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build and push disk images #30

Merged
merged 1 commit into from
Jul 22, 2024
Merged

Build and push disk images #30

merged 1 commit into from
Jul 22, 2024

Conversation

vraiti
Copy link
Collaborator

@vraiti vraiti commented Jul 9, 2024

No description provided.

@vraiti vraiti force-pushed the disk-img-pipeline branch 2 times, most recently from 9d3b286 to ac5cc52 Compare July 10, 2024 21:20
@vraiti vraiti changed the title WIP: Build and push disk images Build and push disk images Jul 10, 2024
@vraiti vraiti requested review from oglok and astoycos July 10, 2024 21:31

oras blob push "$URL" $ARTIFACT

SHA="sha256:"$(sha256sum $ARTIFACT | awk '{print $1}')
Copy link
Collaborator

@astoycos astoycos Jul 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm why are we having to calculate SHAs and sizes manually here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oras AFAIK has no built-in tools for creating image indexes (only for uploading them via oras manifest push), so I build the jsons by hand.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Urg that's annoying, I think we could do it all with buildah

Generate raw, qcow2
gzip raw, qcow2
buildah manifest create $IMAGE
buildah manifest add $IMAGE --artifact <gzipped raw> --arch arm64 --os linux --artifact-type application/vnd.diskimage+raw
buildah manifest add $IMAGE --artifact <gzipped qcow2> --arch arm64 --os linux --artifact-type application/vnd.diskimage+qcow2
buildah manifest annotate --index --annotation bootc-image=<source-image> --annotation bootc-digest=<source-digest> $IMAGE
buildah manifest push --all $IMAGE

I just don't think we need to / should be building the image index + child manifests manually :/

Copy link
Collaborator

@astoycos astoycos Jul 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or even crate the manifest with buildah and then add to it with ORAS, however being able to use one tool would be ideal

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going to say that buildah push always produces an invalid manifest for me and that's why I went this route, but I realized that I had not been including the --all flag haha... Just pushed a new script that uses buildah.

SHA="sha256:"$(sha256sum $ARTIFACT | awk '{print $1}')
SIZE=$(cat $ARTIFACT | wc -c)
cat .github/workflow-scripts/manifest.json.tmpl | envsubst > manifest.json
oras manifest push $URL manifest.json
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we just do

oras manifest push $URL
oras attach $URL --artifact-type application/vnd.diskimage+qcow2 --platform linux/arm64 output/qcow2/disk.qcow2  

without need to make/maintain the manifest ourselves?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Frank asked for the disk images to be findable by a single multiarch manifest, so I wanted to try to stay true to that. The index vs attach method could just be aesthetic, but I worry that because the Oras docs talk about using attachments as dependencies, it would force users to download all of the disk images for a given flavor in order to access one.

I'm also not sure if we can just attach to an empty manifest list? I tried running the commands you gave above and it gave me

vraiti@vraiti-thinkpadt14sgen2i:~/oras/oras1$ oras manifest push quay.io/vraiti/oras:attach index.json
Pushed [registry] quay.io/vraiti/oras:attach
Digest: sha256:806749360f1c4a2f8d7c94e4674fe5aeb5a494353d71fbc19fc90cae03251b08
vraiti@vraiti-thinkpadt14sgen2i:~/oras/oras1$ oras attach quay.io/vraiti/oras:attach --artifact-type application/vnd.diskimage+qcow2 --platform linux/arm64 amd.txt
Error response from registry: failed to resolve attach: sha256:806749360f1c4a2f8d7c94e4674fe5aeb5a494353d71fbc19fc90cae03251b08: not found: no matching manifest was found in the manifest list

fetch-artifact Outdated
@@ -0,0 +1,43 @@
#!/usr/bin/env python3
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no python in this repo yet I don't think, I would keep it that way unless it's really needed, if we're needing to wrap oras let's try and do it with a bash script

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed a new (much simplified) bash script to replace it. Turns out Fedora comes with a super old Oras version that was preventing me from using the --platform option.

@vraiti vraiti force-pushed the disk-img-pipeline branch 7 times, most recently from 8c53a95 to cb90b41 Compare July 18, 2024 15:43
@vraiti vraiti merged commit 81068e8 into main Jul 22, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants