GH-40924: [MATLAB][Packaging] Add script for uploading Release Candidate (RC) MLTBX packages for the MATLAB bindings to the Apache Arrow GitHub Releases area #40956
Conversation
|
|
2. Update git tag 3. Fix target branch name
|
Ah, sorry for not sharing how to implement this in https://lists.apache.org/thread/hm7cbrkggomhx7nxdzr5xkrmkrztmmwo ... How about using We can detect whether this push is for tag or commit by If this push is for tag, we can download the built This is a workflow what arrow-adbc already uses.
|
Not yet but the option exists! We can sign via gha workflow (in apache/arrow), INFRA has to review and approve the workflow doing the signing and will add the key to be used as a repo secret. See https://www.apache.org/legal/release-policy.html#release-signing This change was added sometime last year but I never had the time to follow up but with the current discussions about making releases easier and this new addition it might be the right time to tackle it! (for both the monorepo and subprojects) |
|
#37350 is a related issue for automated release signing. |
|
Hi @kou and @assignUser, Thanks for sending us a lot of helpful resources on how to best cut a GitHub Release. @kevingurney and I will review them and re-work our implementation to use GitHub Actions. To start, we may use the manual signing approach @kou laid out just to get something working, but we would like to transition to an automated signing approach as @assignUser suggested. We'd be happy to investigate how we can get approval from the ASF to automate signing our release artifacts. Thanks again! |
|
I just wanted to say I really appreciate your work on this and your style of communication! Just to clarify the comment about the automated release signing was more a tangent and we would likely implement it for all things in the monorepo at once. So you can continue with the manual way for now with no problems! |
|
Thanks for clarifying @assignUser! We're excited to be working with you guys on this! |
|
As we were looking into this, @kevingurney and I realized we need to be able to connect the crossbow job ID that created the MLTBX file with the GitHub Actions workflow. We aren't aware of any existing mechanism that can provide inputs to a GitHub Actions Workflow that is trigged automatically. As an aside, it is possible to supply inputs to a GitHub Actions Workflow that is triggered manually (see this article). One workaround to enable the automatic approach would be to embed the crossbow job ID in the git tag's message and then extract the ID in the workflow. Using this ID and the We recognize that embedding the ID in the git tag message is not ideal, so please let us know if you have any other suggestions. |
2. Fix syntax errors
|
We can use |
dev/release/05-binary-upload.sh
Outdated
| release_notes="Release Candidate: ${version} RC${rc}" | ||
| title="Apache Arrow ${version} RC${rc}" | ||
| repository="https://github.com/apache/arrow" | ||
| gh release create \ |
There was a problem hiding this comment.
this means we requires the gh CLI installed locally, right?
Can we add a note on the requirements for the release tasks that we require gh CLI installed?
https://github.com/apache/arrow/blob/main/docs/source/developers/release.rst
There was a problem hiding this comment.
There was a problem hiding this comment.
Actually, I just realized you are right. We will need to add a note about requiring the GItHub CLI to be installed locally. While we are no longer using gh to create the release in 04-binary-upload.sh, we will need to use gh once the GitHub release is created to upload the signed MLTBX files.
I'll make sure to add a note about this requirement to the release documentation.
Sorry about the confusion!
…idate Number as Environment Variables
2. Fix archery install path
2. ./dev/release/04-binary-download.sh -> ./arrow/release/04-binary-download.sh 3. Fix directory path provided to find
2. Set environment variable GH_TOKEN in Create GitHub Release for Release Candidate step
| release_tag="apache-arrow-${version_with_rc}" | ||
| tag_message="Release candidate: ${version_with_rc}" | ||
| git tag -a ${release_tag} -m ${tag_message} | ||
| git push apache ${release_tag} |
There was a problem hiding this comment.
Could you do this in dev/release/01-prepare.sh something like the following?
diff --git a/dev/release/01-prepare.sh b/dev/release/01-prepare.sh
index 01fa2f3d80..f5ec0a8e26 100755
--- a/dev/release/01-prepare.sh
+++ b/dev/release/01-prepare.sh
@@ -33,7 +33,7 @@ next_version=$2
next_version_snapshot="${next_version}-SNAPSHOT"
rc_number=$3
-release_tag="apache-arrow-${version}"
+rc_tag="apache-arrow-${version}-rc${rc_number}"
release_branch="release-${version}"
release_candidate_branch="release-${version}-rc${rc_number}"
@@ -45,9 +45,9 @@ release_candidate_branch="release-${version}-rc${rc_number}"
: ${PREPARE_TAG:=${PREPARE_DEFAULT}}
if [ ${PREPARE_TAG} -gt 0 ]; then
- if [ $(git tag -l "${release_tag}") ]; then
- echo "Delete existing git tag $release_tag"
- git tag -d "${release_tag}"
+ if [ $(git tag -l "${rc_tag}") ]; then
+ echo "Delete existing git tag $rc_tag"
+ git tag -d "${rc_tag}"
fi
fi
@@ -91,7 +91,7 @@ if [ ${PREPARE_LINUX_PACKAGES} -gt 0 ]; then
fi
if [ ${PREPARE_VERSION_PRE_TAG} -gt 0 ]; then
- echo "Prepare release ${version} on tag ${release_tag} then reset to version ${next_version_snapshot}"
+ echo "Prepare release ${version} on tag ${rc_tag} then reset to version ${next_version_snapshot}"
update_versions "${version}" "${next_version}" "release"
git commit -m "MINOR: [Release] Update versions for ${version}"
@@ -100,5 +100,5 @@ fi
############################## Tag the Release ##############################
if [ ${PREPARE_TAG} -gt 0 ]; then
- git tag -a "${release_tag}" -m "[Release] Apache Arrow Release ${version}"
+ git tag -a "${rc_tag}" -m "[Release] Apache Arrow Release ${version}"
fiThere was a problem hiding this comment.
Hi @kou,
Thanks for sharing this diff! We just have one clarification question:
Based on the diff you shared, it looks like you want us to replace the release_tag with the rc_tag. Did you mean for us to actually add the rc_tag in addition to the release_tag? We asking because we assume we still want to create a release_tag.
There was a problem hiding this comment.
Did you mean for us to actually add the
rc_tagin addition to therelease_tag?
No. I mean for us to replace the release_tag with rc_tag.
We asking because we assume we still want to create a
release_tag.
I want to change the current workflow:
- Create a
apache-arrow-X.Y.Ztag for X.Y.Z RC0 - (Found a problem for X.Y.Z RC0)
- Remove the
apache-arrow-X.Y.Ztag for X.Y.Z RC0 - Create a
apache-arrow-X.Y.Ztag for X.Y.Z RC1 - ...
to the following:
- Create a
apache-arrow-X.Y.Z-rc0tag for X.Y.Z RC0 - (Found a problem for X.Y.Z RC0)
- Create a
apache-arrow-X.Y.Z-rc1tag for X.Y.Z RC1 - Vote
- Passed
- Create a
apache-arrow-X.Y.Ztag fromapache-arrow-X.Y.Z-rc1like apache/arrow-adbc and apache/arrow-flight-sql-postgresql do
Hmm. It's better that we handle the workflow change related task in a separated issue...
There was a problem hiding this comment.
Thanks for the clarification! I've created a issue #41102 to track this work. Once this PR is merged, I'll take ownership of that issue and make the required changes.
There was a problem hiding this comment.
github-actions
bot
added
awaiting changes
github-actions
bot
added
Component: Documentation
awaiting change review
|
Superseeded by #41131 |
…didates (e.g. apache-arrow-{MAJOR}.{MINOR}.{PATCH}-rc{RC_NUM}) (#41131)
### Rationale for this change
As per @ kou's [suggestion](#40956 (comment)) in #40956, we should create unique git tags (e.g. `apache-arrow-{MAJOR}.{MINOR}.{VERSION}-rc{RC_NUM}`) instead re-using the same git tag (`apache-arrow-{MAJOR}.{MINOR}.{VERSION}`) for each release candidate. The official release candidate tag (`apache-arrow-{MAJOR}.{MINOR}.{VERSION}`) should be created **only** after a release candidate is voted on and accepted. This "official" release tag should point to the same object in the git database as the accepted release candidate tag.
The new release workflow could look like the following:
> 1. Create a apache-arrow-X.Y.Z-rc0 tag for X.Y.Z RC0
> 2. (Found a problem for X.Y.Z RC0)
> 3. Create a apache-arrow-X.Y.Z-rc1 tag for X.Y.Z RC1
> 4. Vote
> 5. Passed
> 6. Create a apache-arrow-X.Y.Z tag from apache-arrow-X.Y.Z-rc1 ike apache/arrow-adbc and apache/arrow-flight-sql-postgresql do
See @ kou's [comment](#40956 (comment)) for more details.
### What changes are included in this PR?
1. Updated `dev/release/01-prepare.sh` to create release-candidate-specific git tags (e.g. `apache-arrow-{MAJOR}.{MINOR}.{PATCH}-rc{RC_NUM}`).
2. Updated scripts in `dev/release` to use the new git tag name.
3. Added GitHub Workflow file `publish_release_candidate.yml`. This workflow is triggered when a release candidate git tag is pushed and creates a Prerelease GitHub Release.
4. Added logic to `dev/release/02-post-binary.sh` to create and push the release git tag (i.e. `apache-arrow-{MAJOR}.{MINOR}.{PATCH}`).
5. Added GitHub Workflow `publish_release.yml`. This workflow is triggered when the release tag is pushed and creates a GitHub Release for the approved release (i.e. the voted upon release).
6. Added `dev/release/post-16-delete-release-candidates.sh` to delete the release candidate git tags and their associated GitHub Releases.
7. Updated `docs/developers/release.rst` with the new steps.
### Are these changes tested?
1. We were not able to verify the changes made to the scripts in `dev/release`. Any suggestions on how we can verify these scripts would be much appreciated :)
2. We did test the new GitHub Workflows (`publish_release_candidate.yml` and `publish_release.yml`) work as intended by pushing git tags to [`mathworks/arrow`](https://github.com/mathworks/arrow).
### Are there any user-facing changes?
No.
### Open Questions
1. We noticed that [apache/arrow-flight-sql-postgresql](https://github.com/apache/arrow-flight-sql-postgresql/releases) does **not** delete the release candidate Prereleases from their GitHub Releases area. Should we be doing the same? Or would it be preferable to just delete the the release candidates **without** deleting the release candidate tags.
2. We're not that familiar with ruby, so we're not sure if the changes we made to `dev/release/02-source-test.rb` make sense.
### Future Directions
1. Continue working on #40956
2. Add logic to auto-sign release artifacts in GitHub Actions Workflows.
* GitHub Issue: #41102
Lead-authored-by: Sarah Gilmore <sgilmore@mathworks.com>
Co-authored-by: Sarah Gilmore <74676073+sgilmore10@users.noreply.github.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sarah Gilmore <sgilmore@mathworks.com>
Rationale for this change
This is a follow up to #38660 and this mailing list discussion.
We would like to upload the MLTBX packages for the MATLAB bindings to the GitHub Releases area for the Apache Arrow GitHub project.
Using GitHub Releases will enable tight integration with the MathWorks File Exchange, which should make it easy for users to discover and install the MATLAB interface using MATLAB.
To make this possible, we need to update the release scripts under /dev/release to upload the MLTBX packages built by crossbow to the GitHub Releases area.
What changes are included in this PR?
dev/release/05-binary-upload.shto sign and upload release candidate MLTBX files to apache/arrow's GitHub Releases Area using thegh release createcommand.prereleaseto differentiate between release candidates and official releases.Are these changes tested?
I was not able to run
dev/release/05-binary-upload.shdirectly, but I did test thegh create release,gpg, andshasumcommands worked as expected by creating a "dummy" release for this repository:https://github.com/sgilmore10/TestRepo/releases/tag/1.0.0rc1Are there any user-facing changes?
No.
Future Directions
dev/release/post-02-upload.shto create a non-prerelease GitHub Release for the MATLAB bindings.05-binary-upload.shinto a separate function that can be reused bypost-02-upload.shto reduce code duplication.Open Questions
dev/release/05-binary-upload.sh?