-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
airbyte-ci: CLI exposes CI requirements #34218
airbyte-ci: CLI exposes CI requirements #34218
Conversation
In this stack:
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @alafanechere and the rest of your teammates on Graphite |
fb831ed
to
91b9c91
Compare
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
5163fd2
to
18552bb
Compare
18552bb
to
9087f08
Compare
@@ -121,3 +126,27 @@ def decorated_function(*args: Any, **kwargs: Any) -> Any: # noqa: ANN401 | |||
return f(*args, **kwargs) | |||
|
|||
return decorated_function | |||
|
|||
|
|||
def click_ci_requirements_option() -> Callable[[FC], FC]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can eventually add arguments to this decorator to customize requirements according to commands.
|
||
|
||
@dataclass | ||
class CIRequirements: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I used a dataclass here in case we want to make the CI requirements payload a bit more complex. It does not have custom attributes at the moment but we can easily add some if needed.
9087f08
to
211409e
Compare
211409e
to
88c7b9e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@@ -83,6 +88,9 @@ def check_local_docker_configuration() -> None: | |||
|
|||
|
|||
def is_dagger_run_enabled_by_default() -> bool: | |||
if CI_REQUIREMENTS_OPTION_NAME in sys.argv: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💅 Note for future: We do need to invert this at some point where commands specifiy if they need dagger run or not.
/approve-and-merge reason="format is known to be broken on airbyte-ci changes, #34220 will solve it" |
What
Relates to #34076
We currently have to define the dagger version to use at two places:
dagger-io
python package versionOn the infra side we have pre-populated runners using different dagger version: 0.6.4 and 0.9.5 .
The targeted runner for a CI job is currently in the
runs-on
field of our GHA workflows.We can face CI failures when we have a mismatch between the client and infra side dagger version.
Mismatch can happen when:
Fixing these mismatch scenario will make the Dagger upgrade / airbyte-ci version pinning a lot easier.
How
airbyte-ci
define its CI requirements, (only the dagger version ATM): we expose a new--ci-requirements
option to the commands in the CI. It outputs a JSON payload, currently{"dagger_version": "0.9.5"}
runs-on
from--ci-requirements
#34220 34220: In our GHA worfklows using airbyte-ci: Run an initialget-ci-runner
job on a GHA runner which will call--ci-requirements
for the specific command that will be run in a next job. The output ofget-ci-runner
is used as theruns-on
field value of the main job.We'll be able to introduce more complex scenario, like picking a runner size dynamically, by changing the output of
--ci-requirements
. We'll have to adapt workflows accordingly.