airbyte-ci: run on github hosted runners

[alafanechere]

What

Closes #33914
We want to make our CI infrastructure running airbyte-ci commands run on GitHub hosted runners and not on self hosted runners.
Tech spec

How

• Manually create new per use-case Github hosted runners.
• Make the CI workflows using airbyte-ci run on these new runners
• Rename run-dagger-pipeline action to run-airbyte-ci and make this action:
  • Optionally connect the runner's dockerd to a registry mirror if provided as an input.
  • Pull the dagger engine docker image according to the output of airbyte-ci --ci-requirements.
  • Cache the Dagger engine docker image to avoid pulling it on fresh runners.

Recommended reading order

Github actions changes

Refactor the run-dagger-pipeline action into smaller and reusable actions.

1. .github/actions/run-airbyte-ci/action.yml
2. .github/actions/install-airbyte-ci/action.yml
3. github/actions/get-dagger-engine-image/action.yml

Github workflows changes

All workflows using airbyte-ci in .github/*

• Remove the get_ci_runner jobs. We now call airbyte-ci --ci-requirements to pull the correct dagger-engine docker image according to the dagger SDK version. It's done within `.github/actions/install-airbyte-ci/action.yml``
• Set runs-on to newly manually provisioned Github hosted runners
• Use a new Dagger Cloud token secret

airbyte-ci changes:

• Remove the unused TAILSCALE_AUTH_KEY and INFRA_SUPPORTED_DAGGER_VERSION constants
• Expose the --ci-requirements at the airbyte-ci root command group level
• Make --ci-requirements output the Dagger engine image to use.

🚨 Performance benchmark 🚨

	Self hosted CPUs	Github Host CPUs	Self hosted duration	GitHub hosted duration	speed difference
format	8	2	3mn53	3mn38 s	-15s
test source-faker + source-postgres	16	16	9mn43	12mn32	+2mn49
publish source-pokeapi	16	16	2mn58s	4mn35	+1mn37s
Nightly build	32	32	4h31	4h54	+23mn

Note on speed differences:

• Interestingly, on a smaller runner with only 2 CPUs (ubuntu-latest) the format job is slighlty faster.
• We can note that on similar instance type (16CPUs) test and publish are slower. The main reason is that on Github hosted runners we have a final step gracefully stopping the dagger engine. The engine shut down can take up to 5mn to upload the dagger cache to Dagger Cloud. This increase in run duration does not impact DX because commits statuses and slack messages are reported before the engine shut down. So we can safely say that the speed difference is not impeeding DX.
• I'm not alarmed by the speed difference for nightly builds: I sequentially run both self hosted and github hosted workflow. Nightly build consumes the API rate limits so the second run (Github hosted) can easily be slower. Moreover the Github hosted runners did not have their dagger cache seeded.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jan 29, 2024 9:24am

[alafanechere]

• master
  • airbyte-ci: run on github hosted runners #34316 👈
    • airbyte-ci: target custom docker registry index #34551
    • [DNM] useLocalCdk = true for a handful of connectors #34518
      • [DNM] add noise 1 #34519
        • [DNM] add noise 2 #34520
          • [DNM] add noise 3 #34521
            • [DNM] add noise 4 #34522
              • [DNM] add noise 5 #34523
                • [DNM] add noise 6 #34524
                  • [DNM] add noise 7 #34525

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @alafanechere and the rest of your teammates on Graphite

[perangel]

This all seems reasonable to me. 👍🏼

[erohmensing]

Nice - and thanks for explaining 👍🏻

[erohmensing]

Not sure how long this takes/if it is an issue at all.

But looks like you can force the docker daemon to reload its configuration without restarting:

Restart the Docker daemon. On Linux, you can avoid a restart (and avoid any downtime for your containers) by reloading the Docker daemon. If you use systemd, then use the command systemctl reload docker. Otherwise, send a SIGHUP signal to the dockerd process.

(from here)

[alafanechere]

Thanks for the suggestion, I did not notice any related slowness, but if I do I'll consider it!

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ QueryError: failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://production.cloudf... dagger.client._core in execute View Issue

_{Did you find this useful? React with a 👍 or 👎}