Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS #11877

Merged
merged 21 commits into from
Jun 29, 2020

Conversation

TobKed
Copy link
Contributor

@TobKed TobKed commented Jun 1, 2020

Build python on GitHub actions for easier release process in the future.

GitHub Actions can be previewed on in fork: https://github.com/TobKed/beam/tree/github-actions-build-wheels-linux-macos

Files on GCS Bucket

Pattern

gs://[BUCKET_NAME]/[BRANCH_NAME]/[COMMIT-SHA1]-[WORKFLOW_RUN_ID]/

Files

  • *.tar.gz, *.zip - python source distribution
  • *.whl - wheels
  • event.json - github action event payload.
  • github_action_info - additional variables not available in event.json

Example output on GCS Bucket:

https://github.com/TobKed/beam/runs/793751247?check_suite_focus=true

gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache-beam-2.23.0.dev0.tar.gz
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache-beam-2.23.0.dev0.zip
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp27-cp27m-macosx_10_9_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp35-cp35m-macosx_10_9_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp36-cp36m-macosx_10_9_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp37-cp37m-macosx_10_9_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp38-cp38-macosx_10_9_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp38-cp38-manylinux1_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp38-cp38-manylinux1_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp38-cp38-manylinux2010_i686.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/apache_beam-2.23.0.dev0-cp38-cp38-manylinux2010_x86_64.whl
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/event.json
gs://***/master/8121a1172413214a774a7efabb32fde268133498-143078997-98/github_action_info

Before merging

Before merging it is required to setup related secrets:

  • GCP_PROJECT_ID - ID of the Google Cloud project e.g: apache-beam-testing
  • GCP_SA_EMAIL - Service account email address to use for authentication. Service account requires Storage Object Admin role. e.g.: beam-wheels-staging-sa@bapache-beam-testing.iam.gserviceaccount.com
  • GCP_SA_KEY - The service account key which will be used for authentication. Service account requires Storage Object Admin role. This key should be created, encoded as a Base64 string (eg. cat my-key.json | base64 on macOS)
  • Edit: since 992ee80 this is not required anymore. GCP_BUCKET - name of the GCS Bucket e.g: beam-wheels-staging

Staging Bucket settings

For practical reasons staging bucket shall have versioning and lifecycle management.

Object versioning

it will keep previous versions of uploaded files in case of rerunning workflow.
e.g. paths:

  • gs://bucket-name/branch-name/8121a1...-140818789/ - first build
  • gs://bucket-name/branch-name/8121a1...-140818789/ - rerun workflow, new files versions
  • gs://bucket-name/branch-name/2323b0...-140818794/ - new commit

Object versioning cen be enabled by command:

gsutil versioning set on gs://[BUCKET_NAME]

https://cloud.google.com/storage/docs/using-object-versioning

Lifecycle management

Lifecycle management is able to delete files which are older than some given age.
It will prevent accumulating old files.
I propose lifecycle settings as follows:

{
   "lifecycle": {
      "rule": [
         {
            "action": {
               "type": "Delete"
            },
            "condition": {
               "age": 365,
               "isLive": true
            }
         },
         {
            "action": {
               "type": "Delete"
            },
            "condition": {
               "age": 90,
               "isLive": false
            }
         }
      ]
   }
}

gsutil commnad:

gsutil lifecycle set lifecycle.json gs://[BUCKET_NAME]

https://cloud.google.com/storage/docs/managing-lifecycles

JIRA

Subtask of BEAM-9388 - Consider using github actions for building python wheels and more (aka. Transition from Travis)


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
--- Build Status
Build Status
Build Status
Build Status
Build Status
--- --- Build Status
XLang --- --- --- Build Status --- --- Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status
Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@TobKed TobKed marked this pull request as draft June 1, 2020 14:42
@TobKed
Copy link
Contributor Author

TobKed commented Jun 1, 2020

github-actions are currently running on PR in fork: TobKed#3

@TobKed TobKed changed the title Build python wheels on GItHub actions for Linux and MacOS [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS Jun 3, 2020
@TobKed TobKed force-pushed the github-actions-build-wheels-linux-macos branch from 391627b to 557209a Compare June 3, 2020 13:29
@TobKed TobKed marked this pull request as ready for review June 3, 2020 14:31
@TobKed
Copy link
Contributor Author

TobKed commented Jun 3, 2020

cc @kamilwu @damgad @aaltay @tysonjh

Copy link
Member

@aaltay aaltay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can I preview the action on the fork: TobKed#3 ?

.github/workflows/build_wheels.yml Outdated Show resolved Hide resolved
.github/workflows/build_wheels.yml Outdated Show resolved Hide resolved
.github/workflows/build_wheels.yml Outdated Show resolved Hide resolved
- name: Upload compressed sources
uses: actions/upload-artifact@v2
with:
name: source_gztar_zip
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this do? Is it re-compressing the source folder. I wonder if we can use the sdist output as it?

(Ideally the resulting GCS output look something close enoguh to a release output, e.g. https://dist.apache.org/repos/dist/release/beam/2.21.0/python/ )

Copy link
Contributor Author

@TobKed TobKed Jun 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is uploading sdist output (zip,tag.gz) files as artifacts where they are picked up by upload_source_to_gcs jobs. An additional advantage of the artifacts is fact that they can be downloaded from GitHub Actions workflow web view as well.

Example of GCS output for build_source -> upload_source_to_gcs jobs:

gs://**/apache-beam-2.23.0.dev0.tar.gz
gs://**/apache-beam-2.23.0.dev0.zip

here you can check GitHub Action which ran on my fork.

Wheel files (*.whl) are processed accordingly by build_wheels -> upload_wheels_to_gcs jobs.
Example of GCS output at the end of the workflow:

gs://**/apache-beam-2.23.0.dev0.tar.gz
gs://**/apache-beam-2.23.0.dev0.zip
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_x86_64.whl

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice.

2 follow up questions:

  1. Can we add a very last stage that can run: gsutil ls "gs://*/$GITHUB_REF##//" -> so that we can get the whole output of the gcs folder all at once.

(btw, do we have a mechanism to clean up these gcs locations?)

  1. What is the difference between "Build python wheels / Build wheels on ..." job executing "Upload wheels" step and "Upload wheels to GCS bucket ..." job?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Currently two steps: List sources on GCS bucket and Copy wheels to GCS bucket are listing files of specific types. Instead of this two separate steps I could create job which will list all files in specific gcs folder. I think it would be much cleaner and explicit. Did I understand correctly your idea?

About cleaning up these GCS locations I consider two options:

  • setting lifecycle management on the bucket which will delete files older than some arbitrary age, e.g. 365 days. I think advantage of this is that will be maintenance free.
  • creating another scheduled workflow on github actions which will delete gcs folders if corresponding branch does not exist anymore. Could be scheduled to run e.g. once pre week.

Which option has more sense for you?

  1. "Upload" steps perform file upload as artifacts so they could be passed between jobs and being available for download for 90 days (if not deleted earlier). These artifacts are picked up later by "Upload to GCS" jobs. What do you think about renaming these steps e.g.: "Upload wheels" -> "Upload wheels as artifacts" ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I missed this earlier.

I think it would be much cleaner and explicit. Did I understand correctly your idea?
Yes. Your understanding is correct. This sounds good to me.

Clean up:
Both options are good. Can we do both?

What do you think about renaming these steps e.g.: "Upload wheels" -> "Upload wheels as artifacts" ?
Sounds good.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be much cleaner and explicit. Did I understand correctly your idea?
Yes. Your understanding is correct. This sounds good to me.

I renamed steps. Please take a look is there no mistakes.

Clean up:
Both options are good. Can we do both?

Sure. We can do both.
Related to versioning and lifecycle management: I added information in the PR description.
For periodic cleaning tasks I created separate dependent draft PR: #12049

.github/workflows/build_wheels.yml Outdated Show resolved Hide resolved
@aaltay
Copy link
Member

aaltay commented Jun 3, 2020

Could we make it such that:

  • we have a cron job that builds nightly (like a snapshot release)
  • and a way to manually trigger this from the release branch, so that the release manager can build wheel files on demand.

python-version: 3.7
- name: Get build dependencies
working-directory: ./sdks/python
run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't need cython at this point, since it's installed and used in build_wheels job. Most probably the same applies to wheels.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point 👍
I've tested and for build_source job only python3 -m pip install -r build-requirements.txt is required, then python -m pip install cython is executed in build_wheels.

@TobKed
Copy link
Contributor Author

TobKed commented Jun 4, 2020

@aaltay
Copy link
Member

aaltay commented Jun 5, 2020

How can I preview the action on the fork: TobKed#3 ?

@aaltay https://github.com/TobKed/beam/actions?query=branch%3Agithub-actions-build-wheels-linux-macos

Super nice! I have not reviewed the steps, I think PR is better for doing the most initial review. We can do a final quick review of this once we settle on this PR.

@TobKed
Copy link
Contributor Author

TobKed commented Jun 5, 2020

cc @potiuk, @brucearctor

@TobKed TobKed force-pushed the github-actions-build-wheels-linux-macos branch from c7d9630 to 32cdf7a Compare June 8, 2020 23:06
@TobKed
Copy link
Contributor Author

TobKed commented Jun 9, 2020

After rethinking gh-actions I made some updates which changes CI behavior dependent on the triggering event as presented below:

  • on pull_request - operates on PR branch, builds wheels and uploads them only to gh-action artifacts (example-run)
  • on push - operates on master/release/release-candidate branches, build wheels and uploads them gh-action artifacts and GCS bucket (folder - branch name) (example-run)
  • on schedule - operates master branch, builds wheels and uploads them to gh-action artifacts and GCS bucket (folder - branch name), tags last commit on master as nightly-master (example-run)

Additionally I've added Cancel Previous Runs workflow (.gihub/workflows/cancel.yml) which cancels previous runs on the same branch.
It was inspired by Apache Airflow solution (link to PR)

@aaltay answering your questions:

Could we make it such that:

  • we have a cron job that builds nightly (like a snapshot release)
  • and a way to manually trigger this from the release branch, so that the release manager can build wheel files on demand.
  1. Nightly is addressed by on schedule trigger.
  2. I investigated triggering builds manually by using repository_dispatch and curl however build triggered by it is executed on master branch and I think there is no convenient way to run it on different branches. If your question is related to "Build Release Candidate" phase it may be solved here as well since build will be triggered during pushing to master/release/release-candidate branches (and related to it by commit hash).

WDYT?

@aaltay
Copy link
Member

aaltay commented Jun 9, 2020

This looks nice. I have a few clarifiying questions:

on pull_request: This is good. Would it trigger on every pull request. This may not be needed. I am not sure what GH resource we will have and what kind of queue there will be. We would not want ot add test load to all PRs.

Questions:

  • Can we trigger this with a phrase? And/Or limit it to python changes?
  • What is gh-action artifacts? How does one access this to get the artifacts?

on push:
Question:

  • What is release-candidate branches? It is release-* ? (If yes, this looks good.)

on schedule: this looks good.

Other questions:

  • How do we use cancel previous runs workflow? Does it work automatically? (If yes, this is great.)

Nightly is addressed by on schedule trigger.

+1.

I investigated triggering builds manually by using repository_dispatch and curl however build triggered by it is executed on master branch and I think there is no convenient way to run it on different branches. If your question is related to "Build Release Candidate" phase it may be solved here as well since build will be triggered during pushing to master/release/release-candidate branches (and related to it by commit hash).

Yes, I think this works. My question was how could a release manager build wheels. Release manager will push some commits but they will not do this using a PR. Would it still trigger builds? And release manager might want to build release at a specific commit in the release branch could they do this by opening a PR?

@TobKed
Copy link
Contributor Author

TobKed commented Jun 10, 2020

@aaltay thank you for review. Answers for your questions:

on pull_request:
Can we trigger this with a phrase? And/Or limit it to python changes?

I added paths for filtering so only meaningful changes will trigger build check.
https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#onpushpull_requestpaths.

What is gh-action artifacts? How does one access this to get the artifacts?

GitHub Action artifacts allow to persist data on storage space to share that data with another job.
They can be downloaded from GitHub UI for by API. GitHub stores artifacts for 90 days
You can test it on example run: https://github.com/TobKed/beam/actions/runs/129158855
GitHub Actions Artifacts documentation
GitHub API - Download an artifact

How do we use cancel previous runs workflow? Does it work automatically? (If yes, this is great.)

It is done automatically in .gihub/workflows/cancel.yml with usage of https://github.com/styfle/cancel-workflow-action.
This action is checking is there any other previously triggered and running workflow on given branch and cancel them.
It saves resources and prevents colliding actions on GCS bucket.

What is release-candidate branches? It is release-* ? (If yes, this looks good.)

I think I made mistake here. To be precise release-* branch is called RELEASE_BRANCH (based on build_release_candidate.sh#L57) and based on build_release_candidate.sh#L110 proper pattern for release-candidate branches should be: v* (previously this pattern was for tags only)
Edit: v* seems to be too general for branch, i change it to v**-RC*

Release manager will push some commits but they will not do this using a PR. Would it still trigger builds? And release manager might want to build release at a specific commit in the release branch could they do this by opening a PR?

Currently source files are build here build_release_candidate.sh#L174 based on RELEASE_BRANCH, the release branch is created few lines above build_release_candidate.sh#L109, this push will trigger build of python sources and wheels.
Build can be triggered in both ways: by pushing to branches matching patterns or creating PRs to these branches (however due to path filtering build on pull request may be not triggered).

I hope it answers your questions :)

@aaltay
Copy link
Member

aaltay commented Jun 10, 2020

I think your last reply mostly answer my questions.

Follow ups:

  • I do not understand why we added v-* or v**-RC* pattern. These look like tags:

Release branch names look like this: release-2.22.0
Release tags looks like: v2.22.0 OR v2.22.0-RC1

  • Related to the last part:

This action will create the source package and wheels right? So, it would not rely on the sh files to build a tar ball and push it?

@TobKed TobKed force-pushed the github-actions-build-wheels-linux-macos branch from 5581b66 to f675d00 Compare June 16, 2020 09:03
@TobKed
Copy link
Contributor Author

TobKed commented Jun 18, 2020

@aaltay in relation to triggering patters, I think I misunderstood naming conventions. I will make update. Thank you for noticing it.

This action will create the source package and wheels right? So, it would not rely on the sh files to build a tar ball and push it?

Yes, it will create both. Whole process of building from scratch can be done in the GItHub Actions workflow. I will create another PR dependent on it, which will use GitHub Actions Artifacts and/or GCS in release sh files.

@aaltay
Copy link
Member

aaltay commented Jun 19, 2020

Thank you @TobKed. I replied to one of your earlier comments. Besides that I think this looks good.

Feel free to ping me if I do not respond to your questions within a few days.

Tobiasz Kędzierski added 2 commits June 19, 2020 15:05
…rerun

Previously every build on given branch would ovewrite files:
	gs://bucket-name/branch-name/  # first build
	gs://bucket-name/branch-name/  # rerun worklow
	gs://bucket-name/branch-name/  # new commit

Now they will be differentiated:
	gs://bucket-name/branch-name/8121a1...-140818789/  # first build
	gs://bucket-name/branch-name/8121a1...-140818789/  # rerun worklow, new files versions
	gs://bucket-name/branch-name/2323b0...-140818794/  # new commit

Wersions of reruns files could be checked with objects versioning
  (if enabled on the bucket)

It will allow better builds tracking: e.g. night-builds
@TobKed TobKed force-pushed the github-actions-build-wheels-linux-macos branch from 9559e71 to 6237068 Compare June 19, 2020 13:53
@aaltay
Copy link
Member

aaltay commented Jun 19, 2020

@TobKed - please ping me if this is ready to be merged.

@TobKed
Copy link
Contributor Author

TobKed commented Jun 23, 2020

@TobKed - please ping me if this is ready to be merged.

@aaltay I think it is ready, however I have some questions before it can me merged:

  1. Who can can and should setup secrets for GitHub Repository? I described necessary secret variables in PR description (Before merging) (docs)
  2. Who can can and should setup versioning and lifecycle management for GCS Bucket?
  3. What do you think about not storing GCP_BUCKET as the secret but hardcoding it, as it is in beam-wheels repository?
    Staging bucket is publicly available and not using secret will allow to show full path to the files in the github actions logs.
    It will make easier to update/modify sh scripts for release process as well.
    I propose to use beam-wheels-staging bucket as it is in beam-wheels.

WDYT?

@aaltay
Copy link
Member

aaltay commented Jun 23, 2020

@TobKed - please ping me if this is ready to be merged.

@aaltay I think it is ready, however I have some questions before it can me merged:

  1. Who can can and should setup secrets for GitHub Repository? I described necessary secret variables in PR description (Before merging) (docs)

I do not have access to the "Settings" tab. Could you work with @tysonjh and infra on this one?

  1. Who can can and should setup versioning and lifecycle management for GCS Bucket?

I can give you access to apache-beam-testing project to update things. Or I can do it. Let me know.

  1. What do you think about not storing GCP_BUCKET as the secret but hardcoding it, as it is in beam-wheels repository?
    Staging bucket is publicly available and not using secret will allow to show full path to the files in the github actions logs.
    It will make easier to update/modify sh scripts for release process as well.
    I propose to use beam-wheels-staging bucket as it is in beam-wheels.

Sounds good to me.

WDYT?

@tysonjh could help.

@TobKed
Copy link
Contributor Author

TobKed commented Jun 25, 2020

HI @aaltay and @tysonjh
Big thanks for helping me with this.
According to JIRA secrets are set 🎉
https://issues.apache.org/jira/browse/INFRA-20459

Please take a final look on the code before merging

@potiuk
Copy link
Member

potiuk commented Jun 25, 2020

🎉

@aaltay aaltay merged commit 8a54c17 into apache:master Jun 29, 2020
@aaltay
Copy link
Member

aaltay commented Jun 29, 2020

Thank you! Could you check that new workflows work as expected?

@TobKed
Copy link
Contributor Author

TobKed commented Jun 29, 2020

Thanks! Sure, I will monitor is everything fine :)

@TobKed
Copy link
Contributor Author

TobKed commented Jun 29, 2020

Looks everything is fine 😀
https://github.com/apache/beam/actions/runs/151717014

@mik-laj
Copy link
Member

mik-laj commented Jun 29, 2020

🎉

@aaltay
Copy link
Member

aaltay commented Jun 29, 2020

Nice, thank you! 🎉

yirutang pushed a commit to yirutang/beam that referenced this pull request Jul 23, 2020
…pache#11877)

[BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS (apache#11877)

Individual commit notes:

* Add licence header

* Set environments labels to the 'latest'

* Fix typos

* Use default verbosity for cibuildwheel

* Add Python 3.8 version for wheels

* Simplify steps

* Refactor trigger events

* List files uploaded to GCS in seperate job

* Update step naming

* Add nightly master build with automatic tagging

* Cancel running builds on second push to PR

Based on Apache Airflow cancel workflow
https://github.com/apache/airflow/blob/master/.github/workflows/cancel.yml

* Upload files to GCS only for pushes to relese and release candidate branches

* Add pattern to match release candidate branch

* Add paths filter for pull requests

* fixup! Add paths filter for pull requests

* fixup! Cancel running builds on second push to PR

* Fix trigering patterns

* Upload additional files with information about buld

* Hange GCS path for uploading artifacts to prevent overwriting during rerun

Previously every build on given branch would ovewrite files:
	gs://bucket-name/branch-name/  # first build
	gs://bucket-name/branch-name/  # rerun worklow
	gs://bucket-name/branch-name/  # new commit

Now they will be differentiated:
	gs://bucket-name/branch-name/8121a1...-140818789/  # first build
	gs://bucket-name/branch-name/8121a1...-140818789/  # rerun worklow, new files versions
	gs://bucket-name/branch-name/2323b0...-140818794/  # new commit

Wersions of reruns files could be checked with objects versioning
  (if enabled on the bucket)

It will allow better builds tracking: e.g. night-builds

* Put bucket name directly into the workflow
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants