Add S3 metadata in meta.json #2731

ravanelli · 2022-02-28T16:06:40Z

No description provided.

miabbott · 2022-02-28T16:54:24Z

Let's mark this as Draft, as I think there is some more design to be discussed here.

miabbott · 2022-02-28T16:57:05Z

I think the primary question is how do we want to encode information that is specific to the build pipeline into meta.json?

Perhaps a new function that we can call during the pipeline execution to write the specific values from the pipeline?

ravanelli · 2022-02-28T17:35:45Z

I think the primary question is how do we want to encode information that is specific to the build pipeline into meta.json?

Perhaps a new function that we can call during the pipeline execution to write the specific values from the pipeline?

We can add some arg for cmd-buildupload (s3 --meta) and use the function created in this commit update_meta_json to update it when we need.

miabbott · 2022-02-28T17:38:42Z

We can add some arg for cmd-buildupload (s3 --meta) and use the function created in this commit update_meta_json to update it when we need.

That sounds good to me! Let's keep this in Draft to see if other folks have input.

Could you also edit the first comment to give a bit more rationale for the change and why we want it?

jlebon · 2022-02-28T18:11:11Z

Could you also edit the first comment to give a bit more rationale for the change and why we want it?

👍 That would be welcome.

I'm assuming this is related to GC, correct? It sounds like you already have a design in mind. Would be good to write it up somewhere where we can discuss it (maybe a new issue on this repo?).

miabbott · 2022-02-28T18:19:32Z

I'm assuming this is related to GC, correct? It sounds like you already have a design in mind. Would be good to write it up somewhere where we can discuss it (maybe a new issue on this repo?).

It's more about providing information about the source of truth for RHCOS builds see - https://issues.redhat.com/browse/COS-1224

miabbott · 2022-03-03T21:35:53Z

Opened an issue for wider discussion - #2739

src/cmd-buildupload

schema/generate-schema.sh

schema/cosa/cosa_v1.go

src/cmd-koji-upload

miabbott

one small change requested, but otherwise i think this looks ready!

miabbott · 2022-03-24T14:58:44Z

is there an example meta.json with the new information recorded in it that we can see?

jlebon

This LGTM overall! Just wanted to make sure also that you saw #2731 (comment) and #2731 (comment).

ravanelli · 2022-03-29T19:57:51Z

This LGTM overall! Just wanted to make sure also that you saw #2731 (comment) and #2731 (comment).

Yeah, totally missed these. Thanks for all the review @jlebon

jlebon · 2022-03-30T13:47:19Z

src/cmd-koji-upload

+        help='Store key information in meta.json')
+
+    upload_cmd.add_argument(
+        '--s3-url', required=False,


Now, it's missing @miabbott's feedback in https://github.com/coreos/coreos-assembler/pull/2731/files#r834404028 :)

Should we change the parameter name to public-url too? I was thinking in let it as url. And only use the public-url structure .

I think it'd be good to be consistent, but personally fine with it. I'm not really concerned about cosa koji-upload becoming some API that people outside the CoreOS team starts using.

I'd prefer it to be consistent, but won't block merging on it for the same reasons Jonathan says. :)

src/v1.json

- Add S3 metadata in meta.json Signed-off-by: Renata Ravanelli <rravanel@redhat.com>

- Update generate-schema to reflex changes done in golang binaries and directory tree Signed-off-by: Renata Ravanelli <rravanel@redhat.com>

- This change is needed due brew/compliance improvements where we need to keep more information about where the artifacts are archived; - Add s3 parameters in order to update meta.json with bucket, prefix and url information used in S3 upload. Signed-off-by: Renata Ravanelli <rravanel@redhat.com>

jlebon · 2022-03-30T17:18:24Z

src/cmd-koji-upload

+        help='Store key information in meta.json')
+
+    upload_cmd.add_argument(
+        '--s3-url', required=False,


I think it'd be good to be consistent, but personally fine with it. I'm not really concerned about cosa koji-upload becoming some API that people outside the CoreOS team starts using.

ravanelli · 2022-03-31T17:00:45Z

/retest-required

ravanelli requested a review from miabbott February 28, 2022 16:07

ravanelli changed the title ~~Schema~~ Add S3 in metadata in meta.json Feb 28, 2022

ravanelli changed the title ~~Add S3 in metadata in meta.json~~ Add S3 metadata in meta.json Feb 28, 2022

ravanelli marked this pull request as draft February 28, 2022 17:30

openshift-ci bot added the do-not-merge/work-in-progress label Feb 28, 2022

miabbott mentioned this pull request Mar 3, 2022

Optionally record additional information about cosa buildupload s3 location + access to that location in meta.json #2739

Open

ravanelli force-pushed the schema branch 2 times, most recently from 2c3060f to 29accc9 Compare March 7, 2022 19:12

ravanelli marked this pull request as ready for review March 7, 2022 19:14

openshift-ci bot removed the do-not-merge/work-in-progress label Mar 7, 2022

ravanelli marked this pull request as draft March 7, 2022 19:17

openshift-ci bot added the do-not-merge/work-in-progress label Mar 7, 2022

ravanelli force-pushed the schema branch 2 times, most recently from f95592c to 2b48f39 Compare March 7, 2022 19:44

ravanelli marked this pull request as ready for review March 7, 2022 19:47

openshift-ci bot removed the do-not-merge/work-in-progress label Mar 7, 2022

ravanelli requested review from jlebon and cgwalters March 7, 2022 19:47

jlebon reviewed Mar 8, 2022

View reviewed changes

src/cmd-buildupload Outdated Show resolved Hide resolved

ravanelli force-pushed the schema branch 4 times, most recently from 250df18 to c69f5fe Compare March 11, 2022 19:08

jlebon reviewed Mar 14, 2022

View reviewed changes

schema/generate-schema.sh Outdated Show resolved Hide resolved

schema/generate-schema.sh Outdated Show resolved Hide resolved

schema/cosa/cosa_v1.go Outdated Show resolved Hide resolved

schema/cosa/cosa_v1.go Outdated Show resolved Hide resolved

ravanelli force-pushed the schema branch 2 times, most recently from 9e6e6f1 to 56de10b Compare March 16, 2022 15:05

ravanelli requested a review from jlebon March 16, 2022 15:06

miabbott reviewed Mar 24, 2022

View reviewed changes

src/cmd-koji-upload Show resolved Hide resolved

miabbott reviewed Mar 24, 2022

View reviewed changes

jlebon previously approved these changes Mar 24, 2022

View reviewed changes

ravanelli dismissed jlebon’s stale review via 24c25aa March 29, 2022 19:20

ravanelli force-pushed the schema branch 2 times, most recently from 24c25aa to 55baa72 Compare March 29, 2022 19:22

ravanelli requested review from miabbott and jlebon March 29, 2022 19:57

ravanelli force-pushed the schema branch from 55baa72 to fc11e1c Compare March 30, 2022 00:12

jlebon reviewed Mar 30, 2022

View reviewed changes

ravanelli force-pushed the schema branch from fc11e1c to 5484c2d Compare March 30, 2022 14:19

ravanelli added 2 commits March 30, 2022 11:26

Add S3 struct in schema

fc1e9f5

- Add S3 metadata in meta.json Signed-off-by: Renata Ravanelli <rravanel@redhat.com>

Update generate-schema

ec8b965

- Update generate-schema to reflex changes done in golang binaries and directory tree Signed-off-by: Renata Ravanelli <rravanel@redhat.com>

ravanelli force-pushed the schema branch 3 times, most recently from d044f20 to 4e6e43f Compare March 30, 2022 14:57

ravanelli force-pushed the schema branch from 4e6e43f to 33f6c78 Compare March 30, 2022 15:09

ravanelli requested a review from jlebon March 30, 2022 16:58

jlebon approved these changes Mar 30, 2022

View reviewed changes

miabbott approved these changes Mar 31, 2022

View reviewed changes

cgwalters merged commit 253f9bb into coreos:main Apr 1, 2022

ravanelli deleted the schema branch December 5, 2022 13:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add S3 metadata in meta.json #2731

Add S3 metadata in meta.json #2731

ravanelli commented Feb 28, 2022

miabbott commented Feb 28, 2022

miabbott commented Feb 28, 2022

ravanelli commented Feb 28, 2022 •

edited

Loading

miabbott commented Feb 28, 2022

jlebon commented Feb 28, 2022

miabbott commented Feb 28, 2022

miabbott commented Mar 3, 2022

miabbott left a comment

miabbott commented Mar 24, 2022

jlebon left a comment

ravanelli commented Mar 29, 2022

jlebon Mar 30, 2022

ravanelli Mar 30, 2022

jlebon Mar 30, 2022

miabbott Mar 31, 2022

jlebon Mar 30, 2022

ravanelli commented Mar 31, 2022

Add S3 metadata in meta.json #2731

Add S3 metadata in meta.json #2731

Conversation

ravanelli commented Feb 28, 2022

miabbott commented Feb 28, 2022

miabbott commented Feb 28, 2022

ravanelli commented Feb 28, 2022 • edited Loading

miabbott commented Feb 28, 2022

jlebon commented Feb 28, 2022

miabbott commented Feb 28, 2022

miabbott commented Mar 3, 2022

miabbott left a comment

Choose a reason for hiding this comment

miabbott commented Mar 24, 2022

jlebon left a comment

Choose a reason for hiding this comment

ravanelli commented Mar 29, 2022

jlebon Mar 30, 2022

Choose a reason for hiding this comment

ravanelli Mar 30, 2022

Choose a reason for hiding this comment

jlebon Mar 30, 2022

Choose a reason for hiding this comment

miabbott Mar 31, 2022

Choose a reason for hiding this comment

jlebon Mar 30, 2022

Choose a reason for hiding this comment

ravanelli commented Mar 31, 2022

ravanelli commented Feb 28, 2022 •

edited

Loading