Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: try to speed up Github workflows #11944

Closed
wants to merge 5 commits into from

Conversation

ktmud
Copy link
Member

@ktmud ktmud commented Dec 5, 2020

SUMMARY

CI has been slow recently. This PR is the latest effort in trying to mitigate:

  1. Update the Cancel Previous Run job added in build: cancel previous github action runs #11940 to cancel future jobs as well.

  2. Skip dependabot branches when possible (e.g. don't run Python jobs for npm updates).

  3. Merge lighter actions (Prefer Typescript and License Check) into one workflow.

  4. Add a script to allow committers to manually cancel CI jobs in queue. This is useful on a busy day when the CI pipeline got clogged by continuous new commits to open PRs. Users have to configure GITHUB_TOKEN in their env variables in order to use this.

    (superset) ➜  incubator-superset git:(manage-github-workflow) ✗ ./scripts/cancel_github_workflows.py --help 
    Usage: cancel_github_workflows.py [OPTIONS] BRANCH_OR_PULL
    
      Cancel running or queued GitHub workflows by branch or pull request ID.
    
      By default jobs that are already running or are the latest of the branch
      or PR will not be cancelled.
    
    Options:
      --repo TEXT                     Default is apache/incubator-superset
      --event [pull_request|push|issue]
      --include-last / --no-include-last
                                      Whether to also cancel the lastest run.
      --include-running / --no-include-running
                                      Whether to also cancel running workflows
      --help                          Show this message and exit.
    

    Snip20201204_53

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TEST PLAN

Test locally and CI

ADDITIONAL INFORMATION

  • Has associated issue:
  • Changes UI
  • Requires DB Migration.
  • Confirm DB Migration upgrade and downgrade tested.
  • Introduces new feature or API
  • Removes existing feature or API

@ktmud ktmud force-pushed the manage-github-workflow branch from 190897c to f659c96 Compare December 5, 2020 04:38
@codecov-io
Copy link

codecov-io commented Dec 5, 2020

Codecov Report

Merging #11944 (e342ccb) into master (c306368) will decrease coverage by 2.58%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #11944      +/-   ##
==========================================
- Coverage   66.60%   64.02%   -2.59%     
==========================================
  Files         959      476     -483     
  Lines       47221    29386   -17835     
  Branches     4620        0    -4620     
==========================================
- Hits        31452    18813   -12639     
+ Misses      15642    10573    -5069     
+ Partials      127        0     -127     
Flag Coverage Δ
cypress ?
javascript ?
python 64.02% <ø> (-0.29%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/sql_validators/postgres.py 50.00% <0.00%> (-50.00%) ⬇️
superset/db_engine_specs/mysql.py 79.59% <0.00%> (-12.25%) ⬇️
superset/db_engine_specs/sqlite.py 65.62% <0.00%> (-9.38%) ⬇️
superset/databases/commands/create.py 83.67% <0.00%> (-8.17%) ⬇️
superset/databases/commands/update.py 85.71% <0.00%> (-8.17%) ⬇️
superset/utils/celery.py 96.42% <0.00%> (-3.58%) ⬇️
superset/models/core.py 85.90% <0.00%> (-2.99%) ⬇️
superset/views/core.py 72.97% <0.00%> (-2.48%) ⬇️
superset/db_engine_specs/base.py 85.75% <0.00%> (-1.85%) ⬇️
superset/datasets/api.py 89.44% <0.00%> (-1.84%) ⬇️
... and 489 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c306368...e342ccb. Read the comment docs.

@ktmud ktmud force-pushed the manage-github-workflow branch 6 times, most recently from 49d82bc to fef6218 Compare December 8, 2020 18:39
@graceguo-supercat
Copy link

graceguo-supercat commented Dec 8, 2020

who can run this scripts and what branches can be cancelled?
Thanks @ktmud

@ktmud
Copy link
Member Author

ktmud commented Dec 8, 2020

Only committers who have write access to this repo can cancel, but anyone can run this script to cancel actions on their own forks.

@ktmud ktmud force-pushed the manage-github-workflow branch from fef6218 to d0e7142 Compare December 16, 2020 23:44
@ktmud
Copy link
Member Author

ktmud commented Dec 17, 2020

This PR now also adopts another action library to cancel previous (AND future) scheduled runs. Previously we didn't see #11940 do much probably because it only cancel previous runs, but for the cancellation script to run, the future scheduled job needs to start, but however they are blocked in the queue by previous runs...

@ktmud ktmud changed the title chore: add a script to cancel Github workflows build: try to speed up Github workflows Dec 17, 2020
@ktmud ktmud closed this Dec 17, 2020
@ktmud ktmud reopened this Dec 17, 2020
@ktmud ktmud changed the title build: try to speed up Github workflows aaa-test: try to speed up Github workflows Dec 17, 2020
@ktmud ktmud changed the title aaa-test: try to speed up Github workflows test: try to speed up Github workflows Dec 17, 2020
@ktmud ktmud changed the title test: try to speed up Github workflows build: try to speed up Github workflows Dec 17, 2020
@ktmud ktmud closed this Dec 17, 2020
@ktmud ktmud reopened this Dec 17, 2020
@ktmud ktmud closed this Dec 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants