airbyte-ci: productionize up-to-date 🧹 + 🚀 #39600

alafanechere · 2024-06-19T17:21:17Z

What

Closes https://github.com/airbytehq/airbyte-internal-issues/issues/8319
Closes https://github.com/airbytehq/airbyte-internal-issues/issues/7418

👀 : A loom is worth a thousand words.

We originally had multiple command we used manually and locally to ship mass connector updates.
These commands had optional bumping/PR creation logic with a lot of shared but duplicate logic.

This PR cleans things up and productionizes the up-to-date command so that it can run automatically in a cronjob.

How

Adopt the "unix philosophy" for migration commands made to be use locally. Developers are expected to run the migration commands (like migrate-to-poetry) and then execute bump-version and pull-request commands to update the connector versions and open a PR. This allows keeping "simple" and isolated steps with less potential side effects.
As the up-to-date command is meant to be automated in GHA this command is not "unix style". It:
- Updates the connector base image to the latest base image version
- Run poetry update to update Python dependencies
- Bump the connector version
- Create a up-to-date/<connector-name> branch and force pushes the modified files to it. (CI disabled)
- Open/update a PR .
- Update the connector changelog with the created PR numbers
- Force pushes all the modified files on the connector (with CI enabled + auto-merge label if the --auto-merge option is passed).

Review guide

* The up-to-date pipeline logic is the most important part of this PR. Everything else is basically code cleaning and slight re-architecture.

The workflow which will automatically create the PRs is there. It's meant to run on a daily basis. We have ~300 python connectors that we want to auto update. A single connector update flow takes ~2mn. So I expect this workflow to run in ~10 hours. This is why I plan to schedule it daily. But there might be some scheduling adjustment to make so that it coordinates well with the auto-merge daily run.

Demo

I manually ran the workflow from this branch. (It's not done yet and I'll monitor if it can correctly open PRs for our 300 connectors in a reasonable amount of time).
It created these PRs.
I disabled the auto-merge label for testing.

User Impact

Not much. Some command names (using kebab case) and options changed but the documentation has been updated.

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

vercel · 2024-06-19T17:21:20Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 21, 2024 9:07am

alafanechere · 2024-06-19T17:21:31Z

airbyte-ci: productionize up-to-date 🧹 + 🚀 #39600 👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @alafanechere and the rest of your teammates on Graphite

alafanechere · 2024-06-19T21:05:59Z

airbyte-ci/connectors/connector_ops/connector_ops/utils.py

@@ -58,11 +58,11 @@ def download_catalog(catalog_url):


 OSS_CATALOG = download_catalog(OSS_CATALOG_URL)
-METADATA_FILE_NAME = "metadata.yaml"


It was declared twice

alafanechere · 2024-06-19T21:06:59Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/bump_version/commands.py

@@ -58,7 +56,6 @@ async def bump_version(
        ctx.obj["execute_timeout"],
        bump_type,
        changelog_entry,
-        pull_request_number,


Going "unix style" here: the bump command just bumps version / changelog, it does not open PRs; the pull requests command can be used sequentially following a version bump.

alafanechere · 2024-06-19T21:07:29Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/bump_version/pipeline.py

 from pipelines.models.steps import Step, StepResult, StepStatus

 if TYPE_CHECKING:
    from anyio import Semaphore


-def get_bumped_version(version: str | None, bump_type: str) -> str:


Moved this to the BumpConnectorVersion step

alafanechere · 2024-06-19T21:07:59Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/bump_version/pipeline.py

@@ -79,202 +60,11 @@ async def _cleanup(self) -> StepResult:
        return StepResult(step=self, status=StepStatus.SUCCESS)


-class AddChangelogEntry(Step):


Moved this to pipelines.airbyte_ci.steps.changelog as it's a step reused between multiple commands.

alafanechere · 2024-06-19T21:08:25Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/bump_version/pipeline.py

-        )
-
-
-class SetConnectorVersion(Step):


Moved to pipelines.airbyte_ci.steps.bump_version

alafanechere · 2024-06-19T21:09:19Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/commands.py

Adopting consistent kebab casing for connectors subcommands

alafanechere · 2024-06-19T21:10:14Z

...te-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/migrate_to_base_image/commands.py

-        pull_request_number,
-        changelog,
-        bump,


Going unix style: the bump and pull request pr can be used after this command if needed.

alafanechere · 2024-06-19T21:10:32Z

...te-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/migrate_to_base_image/pipeline.py


    from anyio import Semaphore


-class UpgradeBaseImageMetadata(Step):


Moved to pipelines.airbyte_ci.steps.base_image

alafanechere · 2024-06-20T08:32:22Z

airbyte-ci/connectors/pipelines/tests/test_changelog.py

-    changelog.add_entry(semver.VersionInfo.parse("3.4.0"), datetime.date.fromisoformat("2024-03-01"), 123456, "test1")
-    changelog.add_entry(semver.VersionInfo.parse("3.4.0"), datetime.date.fromisoformat("2024-03-01"), 123457, "test2")
-    check_result(changelog, "dupicate_version_date_" + filename)
+# This test is disabled because the current implementation allows inserting non number PR numbers.


cc @stephane-airbyte

alafanechere · 2024-06-20T08:33:58Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/commands.py

I made the command name casing consistent: use kebab-case everywhere instead of a mix of kebab and snake.

alafanechere · 2024-06-20T08:34:32Z

...i/connectors/pipelines/pipelines/airbyte_ci/connectors/migrate_to_logging_logger/pipeline.py

Import cleaning following poe lint --fix

alafanechere · 2024-06-20T08:35:13Z

...te-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/migrate_to_base_image/pipeline.py

Simplified this command: it does not bump the connector version. Use bump-version command for that

alafanechere · 2024-06-20T08:36:34Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/common/regression_test.py

This was a set of checks performed on the connector following updates. Now that updates are automated I believe it's CI role to validate the updates are valid, it should not be done by the up-to-date pipeline itself

alafanechere · 2024-06-20T08:48:05Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/upgrade_base_image/__init__.py

Deleted as upgrade_base_image is now part of the up-to-date command

alafanechere · 2024-06-20T08:48:08Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/upgrade_base_image/commands.py

Deleted as upgrade_base_image is now part of the up-to-date command

alafanechere · 2024-06-20T08:48:12Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/upgrade_base_image/pipeline.py

Deleted as upgrade_base_image is now part of the up-to-date command

bleonard

exciting. I'm a bit curious to see it to:

make a PR
master changes in lots of ways
update the PR

I expected to see something like a force push or using the master branch as the base_ref in the PR code. I had some crazy-long comment about that before that I can't find.

Anyway, be on the lookout for that.

alafanechere · 2024-06-21T08:49:58Z

I had some crazy-long comment about that before that I can't find.

@bleonard I removed it because I "resolved" it:

When the worfklow runs it checkouts the current master head.
If an up-to-date branch / PR exists for the connector it will force push modification to it and update titles.

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Jun 19, 2024

vercel bot deployed to Preview June 19, 2024 17:25 View deployment

alafanechere force-pushed the augustin/06-17-airbyte-ci_productionize_up_to_date branch from c7631da to b8aba10 Compare June 19, 2024 17:26

octavia-squidington-iii removed the area/documentation Improvements or additions to documentation label Jun 19, 2024

alafanechere force-pushed the augustin/06-17-airbyte-ci_productionize_up_to_date branch 2 times, most recently from 9c874f5 to eb5b545 Compare June 19, 2024 21:04

alafanechere commented Jun 19, 2024

View reviewed changes

alafanechere force-pushed the augustin/06-17-airbyte-ci_productionize_up_to_date branch 3 times, most recently from 6af37db to f648f51 Compare June 20, 2024 07:14

alafanechere mentioned this pull request Jun 20, 2024

up-to-date: create workflow (not tested) #39607

Merged

alafanechere force-pushed the augustin/06-17-airbyte-ci_productionize_up_to_date branch 7 times, most recently from 51a7203 to e431f7b Compare June 20, 2024 08:18

alafanechere commented Jun 20, 2024

View reviewed changes

alafanechere changed the title ~~airbyte-ci: productionize up to date~~ airbyte-ci: productionize up-to-date 🧹 + 🚀 Jun 20, 2024

alafanechere commented Jun 20, 2024

View reviewed changes

alafanechere force-pushed the augustin/06-17-airbyte-ci_productionize_up_to_date branch from e431f7b to edc9da8 Compare June 20, 2024 08:59

alafanechere marked this pull request as ready for review June 20, 2024 09:00

alafanechere requested a review from a team as a code owner June 20, 2024 09:00

alafanechere force-pushed the augustin/06-17-airbyte-ci_productionize_up_to_date branch 3 times, most recently from a0f72b1 to e781c80 Compare June 20, 2024 10:45

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Jun 20, 2024

vercel bot deployed to Preview June 20, 2024 11:26 View deployment

alafanechere requested review from bleonard, clnoll and natikgadzhi June 20, 2024 16:31

bleonard approved these changes Jun 21, 2024

View reviewed changes

airbyte-ci: productionize up to date

dc398e9

alafanechere force-pushed the augustin/06-17-airbyte-ci_productionize_up_to_date branch from e781c80 to dc398e9 Compare June 21, 2024 09:03

vercel bot deployed to Preview June 21, 2024 09:07 View deployment

alafanechere enabled auto-merge (squash) June 21, 2024 09:15

alafanechere merged commit 9eb248f into master Jun 21, 2024
30 checks passed

alafanechere deleted the augustin/06-17-airbyte-ci_productionize_up_to_date branch June 21, 2024 09:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

airbyte-ci: productionize up-to-date 🧹 + 🚀 #39600

airbyte-ci: productionize up-to-date 🧹 + 🚀 #39600

alafanechere commented Jun 19, 2024 •

edited

Loading

vercel bot commented Jun 19, 2024 •

edited

Loading

alafanechere commented Jun 19, 2024

alafanechere Jun 19, 2024

alafanechere Jun 19, 2024

alafanechere Jun 19, 2024

alafanechere Jun 19, 2024

alafanechere Jun 19, 2024

alafanechere Jun 19, 2024

alafanechere Jun 19, 2024

alafanechere Jun 19, 2024

alafanechere Jun 20, 2024

alafanechere Jun 20, 2024

alafanechere Jun 20, 2024

alafanechere Jun 20, 2024

alafanechere Jun 20, 2024

alafanechere Jun 20, 2024

alafanechere Jun 20, 2024

alafanechere Jun 20, 2024

bleonard left a comment

alafanechere commented Jun 21, 2024

		@@ -58,11 +58,11 @@ def download_catalog(catalog_url):


		OSS_CATALOG = download_catalog(OSS_CATALOG_URL)
		METADATA_FILE_NAME = "metadata.yaml"

		@@ -79,202 +60,11 @@ async def _cleanup(self) -> StepResult:
		return StepResult(step=self, status=StepStatus.SUCCESS)


		class AddChangelogEntry(Step):


		from anyio import Semaphore


		class UpgradeBaseImageMetadata(Step):

airbyte-ci: productionize up-to-date 🧹 + 🚀 #39600

airbyte-ci: productionize up-to-date 🧹 + 🚀 #39600

Conversation

alafanechere commented Jun 19, 2024 • edited Loading

What

How

Review guide

Demo

User Impact

Can this PR be safely reverted and rolled back?

vercel bot commented Jun 19, 2024 • edited Loading

alafanechere commented Jun 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bleonard left a comment

Choose a reason for hiding this comment

alafanechere commented Jun 21, 2024

alafanechere commented Jun 19, 2024 •

edited

Loading

vercel bot commented Jun 19, 2024 •

edited

Loading