-
Notifications
You must be signed in to change notification settings - Fork 165
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optionally record additional information about cosa buildupload s3
location + access to that location in meta.json
#2739
Comments
Nice write-up! The request makes sense to me. My only minor hesitation with the suggested approach is having the canonical Hmm, would it be sufficient to just put that information in a separate file when uploading to Brew? It doesn't solve the stale issue, but at least it makes it less likely that something else would use that information. But not strongly against shoving it in |
Worth keeping in mind that #2685 is pushing to change our "center of gravity" to be more container focused and less S3. |
- This change is needed due brew/compliance improvements where we need to keep more information about where the artifacts are archived; - Add --meta parameter in order to update meta.json with bucket, prefix and url information used in S3 upload. More information: coreos#2739 Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
I'm less concerned with the problem of stale information in this situation. Broadly speaking, if the build dir is being copied to a new location, the person/robot doing so should be aware of the implications and not be relying on the information about the S3 bucket after the copy operation. That being said, I wasn't terribly comfortable with encoding this data in Since @ravanelli is already going forward with the |
Feature Request
We wish to be able to upload the results of a successful RHCOS build into Brew. However, to avoid the huge penalty of uploading every (uncompressed!) artifact into Brew, we've reached an agreement with the Software Production folks that providing metadata about the build should suffice.
Since we treat S3 as our canonical source of truth (the bucket we use contains every production RHCOS build ever made), we want to be able to record this bucket in
meta.json
as a way for other processes that operate on Brew builds to know where the uploaded artifacts are located. Additionally, since the bucket is private and not accessible publicly, we want to provide information about the redirector service (https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/) so that it is possible for users to determine how to access the artifacts in the S3 bucket, by way of the redirector.Desired Feature
The original design that I'm proposing here consists of two parts: an update to the meta.json schema and an enhancement to the
cosa buildupload
command.The update to the meta.json schema would include a new optional
s3
dict that would allow us to record the bucket name, subdirectory/path, and URL for the redirector service.The enhancement to the
cosa buildupload
command would be some new option(s) that would allow the user to specify that information about the S3 bucket, path, and URL should be recorded in meta.json. Perhaps these options are only applicable tocosa buildupload s3
.Example Usage
Using the existing mechanism, this would instruct the command to split the bucket/path from the final argument and then record all the info to meta.json
$ cosa buildupload s3 --acl=private --record-s3-metadata --s3-url https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/art/storage/releases/rhcos-4.7-ppc64le art-rhcos-ci/releases/rhcos-4.7-ppc64le
Alternatively, change the invocation to support explicit options to specify bucket, path, url:
$ cosa buildupload s3 --acl=private --s3-url https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/art/storage/releases/rhcos-4.7-ppc64le --s3-bucket art-rhcos-ci --s3-path releases/rhcos-4.7-ppc64le
Other Information
This is in support of making RHCOS artifacts available to the various cloud marketplaces. See the AWS Marketplace Epic and the associated work stories attached to it https://issues.redhat.com/browse/COS-6
There is an outstanding PR that started on some of this request - #2731
The text was updated successfully, but these errors were encountered: