feat(pipelines) Add manifest only build #39906

bnchrch · 2024-06-20T17:02:07Z

TL;DR

The following changes were made to the Airbyte CI to add support for manifest-only connectors.

What changed?

New Connector Language: Added MANIFEST_ONLY as a new connector language in ConnectorLanguage enum.
Path Handling: Introduced _manifest_only_path method in connector_ops/utils.py to handle paths for manifest-only connectors.
Manifest Path Resolution: Updated manifest_path property in connector_ops/utils.py to resolve manifest paths based on file existence.
Build Pipeline: Created manifest_only_connectors.py in build_image/steps to define build steps for manifest-only connectors, and added support for it in pipeline.py and __init__.py.
Testing: Added manifest_only_connectors.py in test/steps to define test steps and updated pipeline.py and __init__.py accordingly.
Tests: Introduced test_manifest_only_connectors.py in tests/test_build_image for testing the new functionality.
Version Bump: Updated the versions of connector_ops and pipelines packages.

How to test?

Run CI pipeline to ensure all tests pass.

vercel · 2024-06-20T17:02:11Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jul 11, 2024 2:10pm

bnchrch · 2024-06-20T17:02:28Z

demo(source-declarative-manifest): Migrate source-pokeapi #39907
feat(pipelines) Add manifest only build #39906 👈
feat(source-declarative-manifest): add local yaml support #39905
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @bnchrch and the rest of your teammates on Graphite

sentry-io · 2024-06-20T17:02:33Z

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: airbyte-ci/connectors/connector_ops/connector_ops/utils.py

Function	Unhandled Issue
`metadata`	KeyError: 'data' connector_ops.utils in metadata `Event Count:` 2

_{Did you find this useful? React with a 👍 or 👎}

alafanechere

Approving as there's nothing utterly blocking on my end but please:

Consider sharing as much logic as possible between the "python" connector types. It'll simplify refactor if we have to modify the testing flow or the build flow.
Can you update all logics in airbyte-ci, qa_checks, connectors_insights which are based on the ConnectorLanguage? So far I've mainly used if connector.language in [ConnectorLanguage.PYTHON, ConnectorLanguage.LOW_CODE] to filter "python" connectors.

airbyte-ci/connectors/connector_ops/connector_ops/utils.py

alafanechere · 2024-07-01T16:10:04Z

airbyte-ci/connectors/pipelines/README.md

@@ -761,6 +761,7 @@ E.G.: running Poe tasks on the modified internal packages of the current branch:

 | Version | PR                                                         | Description                                                                                                                  |
 | ------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
+| 4.21.0  | [#TODO](https://github.com/airbytehq/airbyte/pull/TODO)    | Add manifest only build pipeline                                                          |


alafanechere · 2024-07-01T16:15:56Z

...tors/pipelines/pipelines/airbyte_ci/connectors/build_image/steps/manifest_only_connectors.py

+        self.logger.info(f"Building connector from base image in metadata for {platform}")
+        base = self._get_base_container(platform)
+
+        customized_base = await build_customization.pre_install_hooks(self.context.connector, base, self.logger)


I think there's duplication between _build_from_base_image in this class vs the python one. Can we keep a common code path for setting entrypoints, labels, calling pre/post hooks etc?

Your could might "just be" considered a pre-install hook?

@alafanechere I was thinking on this.

Im not sure this is the place to DRY up the code completely and have a common build for python and manifest only connectors.

The reasons for that are

The similarities between the manifest and python _build_from_base_image are that they use pre and post commit hooks. After that the python build does a lot more inbetween with two stage build

We are likely going to deviate more from python as we look at how we add support for custom components, potentially in other languages.

I'm almost more inclined to

Remove the hooks for manifest only

Add a helper function for adding the docker labels that the two builds can share

alafanechere · 2024-07-01T16:17:58Z

.../connectors/pipelines/pipelines/airbyte_ci/connectors/test/steps/manifest_only_connectors.py

+from pipelines.helpers.execution.run_steps import STEP_TREE, StepToRun
+
+
+def get_test_steps(context: ConnectorTestContext) -> STEP_TREE:


💅 I'm wondering if we could find a way to have a common tree for low_code/manifest/python connectors and just add unit tests / integration tests for low_code and python connectors.

🤔

Looking ahead I see

Low-code being replaced by manifest only

Manifest only deviating from python (perhaps we allow js custom components? or we don't allow any third party dependencies? or if we do we need to reconcile them

Then I think its best to keep them separate and not share to much logic or test coverage.

WDYT?

yes, but these changes are in the rather distant future, aren't they? I imagine we'll be reworking the step tree long before then. Anyway, it's not blocking for me, so I'll let you do what you think is wise :)

Theyre not too far away!

Also the way were likely to do "upload for pyairbyte" changes.

So the types of tests we run are even different

https://airbytehq-team.slack.com/archives/C02TL38U5L7/p1719448571131109

Thanks for the push here btw.

alafanechere · 2024-07-01T16:25:07Z

...tors/pipelines/pipelines/airbyte_ci/connectors/build_image/steps/manifest_only_connectors.py

+        customized_base = await build_customization.pre_install_hooks(self.context.connector, base, self.logger)
+        entrypoint = build_customization.get_entrypoint(self.context.connector)
+
+        manifest_file_name = "manifest.yaml"


💅 This can be stored in the const module

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/publish/pipeline.py

alafanechere · 2024-07-01T16:29:01Z

airbyte-ci/connectors/pipelines/poetry.lock

not sure why an update of the poetry lock was required.

bnchrch · 2024-07-09T14:36:21Z

airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/build_image/steps/common.py

@@ -35,9 +43,10 @@ async def _run(self, *args: Any) -> StepResult:
        build_results_per_platform = {}
        for platform in self.build_platforms:
            try:
-                connector = await self._build_connector(platform, *args)
+                connector_container = await self._build_connector(platform, *args)
+                connector_container = apply_airbyte_docker_labels(connector_container, self.context.connector)


@alafanechere I had an idea to move the application of the docker labels to a generic place for all connectors.

WDYT?

coderabbitai · 2024-07-10T13:44:00Z

Important

Review skipped

Auto reviews are limited to specific labels.

Labels to auto review (1)

coderabbit-ai-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

bnchrch mentioned this pull request Jun 20, 2024

feat(source-declarative-manifest): add local yaml support #39905

Merged

bnchrch mentioned this pull request Jun 20, 2024

demo(source-declarative-manifest): Migrate source-pokeapi #39907

Closed

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch from 4312a30 to 8b9cff3 Compare June 20, 2024 19:38

bnchrch force-pushed the 06-20-feat_source-declarative-manifest_add_local_yaml_support branch from 32b1f15 to 5057405 Compare June 26, 2024 00:52

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch 2 times, most recently from 5cbae6b to cbb57f7 Compare June 26, 2024 18:54

octavia-squidington-iii added area/connectors Connector related issues connectors/source/declarative-manifest labels Jun 26, 2024

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch from ac1f121 to 5360c5e Compare June 27, 2024 19:00

bnchrch marked this pull request as ready for review June 27, 2024 21:32

bnchrch requested a review from a team as a code owner June 27, 2024 21:32

alafanechere approved these changes Jul 1, 2024

View reviewed changes

natikgadzhi mentioned this pull request Jul 2, 2024

⭐ New Source: Goldcast #38786

Merged

2 tasks

bnchrch force-pushed the 06-20-feat_source-declarative-manifest_add_local_yaml_support branch from 5057405 to a01f5a2 Compare July 8, 2024 15:41

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch from 5360c5e to 9c15fef Compare July 8, 2024 15:42

bnchrch force-pushed the 06-20-feat_source-declarative-manifest_add_local_yaml_support branch from a01f5a2 to c68951f Compare July 9, 2024 14:14

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch from 9c15fef to 2aef347 Compare July 9, 2024 14:15

octavia-squidington-iii removed the area/connectors Connector related issues label Jul 9, 2024

bnchrch force-pushed the 06-20-feat_source-declarative-manifest_add_local_yaml_support branch from c68951f to 5082709 Compare July 9, 2024 14:23

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch from 2aef347 to 6665351 Compare July 9, 2024 14:24

bnchrch force-pushed the 06-20-feat_source-declarative-manifest_add_local_yaml_support branch from 5082709 to 2715f39 Compare July 9, 2024 14:28

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch from 6665351 to ca3daf0 Compare July 9, 2024 14:28

bnchrch commented Jul 9, 2024

View reviewed changes

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch from ca3daf0 to d22ea61 Compare July 9, 2024 15:46

bnchrch force-pushed the 06-20-feat_source-declarative-manifest_add_local_yaml_support branch from 2715f39 to 4b329f0 Compare July 9, 2024 19:53

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch from d22ea61 to 247685c Compare July 9, 2024 19:54

Base automatically changed from 06-20-feat_source-declarative-manifest_add_local_yaml_support to master July 9, 2024 23:07

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch from 247685c to fb8df7d Compare July 10, 2024 13:43

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch 3 times, most recently from 1f472fa to bfd0d0f Compare July 11, 2024 13:32

bnchrch enabled auto-merge (squash) July 11, 2024 13:51

bnchrch added 4 commits July 11, 2024 07:08

feat(pipelines) Add manifest only build

4ea2b43

Add manifest only test pipeline

d943b73

Add tests

fdad35d

fix tests

1719f64

bnchrch force-pushed the 06-20-feat_pipelines_add_manifest_only_build branch from bfd0d0f to 1719f64 Compare July 11, 2024 14:09

bnchrch merged commit 614acdb into master Jul 11, 2024
34 checks passed

bnchrch deleted the 06-20-feat_pipelines_add_manifest_only_build branch July 11, 2024 14:27

xiaohansong pushed a commit that referenced this pull request Jul 12, 2024

feat(pipelines) Add manifest only build (#39906)

298503d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pipelines) Add manifest only build #39906

feat(pipelines) Add manifest only build #39906

bnchrch commented Jun 20, 2024 •

edited

Loading

vercel bot commented Jun 20, 2024 •

edited

Loading

bnchrch commented Jun 20, 2024 •

edited

Loading

sentry-io bot commented Jun 20, 2024

alafanechere left a comment

alafanechere Jul 1, 2024

alafanechere Jul 1, 2024

bnchrch Jul 9, 2024

alafanechere Jul 1, 2024

bnchrch Jul 9, 2024

alafanechere Jul 9, 2024

bnchrch Jul 9, 2024

alafanechere Jul 1, 2024

alafanechere Jul 1, 2024

bnchrch Jul 9, 2024

coderabbitai bot commented Jul 10, 2024

Review skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

	\| 4.21.0 \| [#TODO](https://github.com/airbytehq/airbyte/pull/TODO) \| Add manifest only build pipeline \|
	\| 4.21.0 \| [#39906](https://github.com/airbytehq/airbyte/pull/39906) \| Add manifest only build pipeline \|

		from pipelines.helpers.execution.run_steps import STEP_TREE, StepToRun


		def get_test_steps(context: ConnectorTestContext) -> STEP_TREE:

feat(pipelines) Add manifest only build #39906

feat(pipelines) Add manifest only build #39906

Conversation

bnchrch commented Jun 20, 2024 • edited Loading

TL;DR

What changed?

How to test?

vercel bot commented Jun 20, 2024 • edited Loading

bnchrch commented Jun 20, 2024 • edited Loading

sentry-io bot commented Jun 20, 2024

🔍 Existing Issues For Review

alafanechere left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coderabbitai bot commented Jul 10, 2024

Review skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

bnchrch commented Jun 20, 2024 •

edited

Loading

vercel bot commented Jun 20, 2024 •

edited

Loading

bnchrch commented Jun 20, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)