-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
airbyte-ci: run on github hosted runners #34316
airbyte-ci: run on github hosted runners #34316
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @alafanechere and the rest of your teammates on Graphite |
2ab4ff0
to
f533ff8
Compare
7fb9c53
to
83bdcd5
Compare
199f740
to
4f45659
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This all seems reasonable to me. 👍🏼
TAILSCALE_AUTH_KEY: ${{ inputs.tailscale_auth_key }} | ||
DOCKER_REGISTRY_MIRROR_URL: ${{ inputs.docker_registry_mirror_url }} | ||
PYTHON_REGISTRY_TOKEN: ${{ inputs.python_registry_token }} | ||
# give the Dagger Engine more time to push cache data to Dagger Cloud |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice - and thanks for explaining 👍🏻
sudo systemctl stop docker | ||
echo '{"registry-mirrors": ["https://${{inputs.docker_registry_mirror_url}}"]}' | sudo tee /etc/docker/daemon.json | ||
sudo systemctl start docker |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure how long this takes/if it is an issue at all.
But looks like you can force the docker daemon to reload its configuration without restarting:
Restart the Docker daemon. On Linux, you can avoid a restart (and avoid any downtime for your containers) by reloading the Docker daemon. If you use systemd, then use the command systemctl reload docker. Otherwise, send a SIGHUP signal to the dockerd process.
(from here)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the suggestion, I did not notice any related slowness, but if I do I'll consider it!
4f45659
to
cc09474
Compare
eeb7751
to
47039ff
Compare
6d43576
to
72c1211
Compare
72c1211
to
1b1709a
Compare
Suspect IssuesThis pull request was deployed and Sentry observed the following issues:
Did you find this useful? React with a 👍 or 👎 |
What
Closes #33914
We want to make our CI infrastructure running
airbyte-ci
commands run on GitHub hosted runners and not on self hosted runners.Tech spec
How
run-dagger-pipeline
action torun-airbyte-ci
and make this action:dockerd
to a registry mirror if provided as an input.airbyte-ci --ci-requirements
.Recommended reading order
Github actions changes
Refactor the
run-dagger-pipeline
action into smaller and reusable actions..github/actions/run-airbyte-ci/action.yml
.github/actions/install-airbyte-ci/action.yml
github/actions/get-dagger-engine-image/action.yml
Github workflows changes
All workflows using
airbyte-ci
in.github/*
get_ci_runner
jobs. We now callairbyte-ci --ci-requirements
to pull the correctdagger-engine
docker image according to the dagger SDK version. It's done within `.github/actions/install-airbyte-ci/action.yml``runs-on
to newly manually provisioned Github hosted runnersairbyte-ci
changes:TAILSCALE_AUTH_KEY
andINFRA_SUPPORTED_DAGGER_VERSION
constants--ci-requirements
at theairbyte-ci
root command group level--ci-requirements
output the Dagger engine image to use.🚨 Performance benchmark 🚨
Note on speed differences:
ubuntu-latest
) theformat
job is slighlty faster.