Add provinance to the registry entry #37804

bnchrch · 2024-05-03T18:41:09Z

What

Adds the generation time and root metadata file info to the registry entry

Spun out of #32715 as a stack

closes https://github.com/airbytehq/airbyte-internal-issues/issues/7466

vercel · 2024-05-03T18:41:12Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		May 9, 2024 2:29pm

bnchrch · 2024-05-03T18:41:26Z

feat(registry): add cdk version #37809
Add provinance to the registry entry #37804 👈
Notify commit owner via slack message #37803
Upload git info with metadata file (#1) #37802
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @bnchrch and the rest of your teammates on Graphite

natikgadzhi

Haven't tested, but the code looks great. Thank you for extensive comments, and for poetry locking so we get a few vulns fixed while we're at it!

natikgadzhi · 2024-05-05T14:15:50Z

...data_service/lib/metadata_service/models/generated/ConnectorRegistryDestinationDefinition.py

+
+    commit_sha: Optional[str] = Field(
+        None,
+        description="The git commit sha of the last commit that modified this file. DO NOT DEFINE THIS FIELD IN YOUR SPEC. It will be overwritten by the CI.",


Nit: should we lift the autogeneration notice up one/two levels and just say that the whole source_file_info probably should not be defined in a spec?

natikgadzhi · 2024-05-05T14:17:41Z

airbyte-ci/connectors/metadata_service/orchestrator/orchestrator/utils/object_helpers.py

+
+
+def default_none_to_dict(value, key, obj):
+    """Set the value of a key in a dictionary to an empty dictionary if the value is None.


TY for a nice doc! <3

natikgadzhi · 2024-05-05T14:18:06Z

airbyte-ci/connectors/metadata_service/orchestrator/orchestrator/ops/slack.py

@@ -29,7 +29,7 @@ def send_slack_message(context: OpExecutionContext, channel: str, message: str,
        channel (str): The channel to send the message to.
        message (str): The message to send.
    """
-    if os.getenv("SLACK_TOKEN"):
+    if os.getenv("SLACK_TOKEN") == "" and os.getenv("SLACK_NOTIFICATIONS_DISABLED") != "true":


Is the new env var documented somewhere?

alafanechere · 2024-05-06T09:43:04Z

.../metadata_service/lib/metadata_service/models/generated/ConnectorRegistrySourceDefinition.py

+    metadata_etag: Optional[str] = None
+    metadata_file_path: Optional[str] = None
+    metadata_bucket_name: Optional[str] = None
+    metadata_last_modified: Optional[str] = None


Nice addition for debugging

alafanechere · 2024-05-06T09:43:32Z

.../metadata_service/lib/metadata_service/models/generated/ConnectorRegistrySourceDefinition.py

+
+class SourceFileInfo(BaseModel):
+    metadata_etag: Optional[str] = None
+    metadata_file_path: Optional[str] = None


It's a blob/key path isn't?

alafanechere · 2024-05-06T09:44:33Z

.../metadata_service/lib/metadata_service/models/generated/ConnectorRegistrySourceDefinition.py

@@ -87,6 +117,11 @@ class BreakingChangeScope(BaseModel):
    __root__: StreamBreakingChangeScope = Field(..., description="A scope that can be used to limit the impact of a breaking change.")


+class GeneratedFields(BaseModel):


As I said in #37802 I think all generated fields should stay in the registry, not be metadata fields (which is what you're doing here 👍 )

alafanechere · 2024-05-06T09:45:15Z

...-ci/connectors/metadata_service/lib/metadata_service/models/generated/ConnectorRegistryV0.py

+    commit_sha: Optional[str] = Field(
+        None,
+        description="The git commit sha of the last commit that modified this file. DO NOT DEFINE THIS FIELD IN YOUR SPEC. It will be overwritten by the CI.",
+    )
+    commit_timestamp: Optional[datetime] = Field(
+        None,
+        description="The git commit timestamp of the last commit that modified this file. DO NOT DEFINE THIS FIELD IN YOUR SPEC. It will be overwritten by the CI.",
+    )
+    commit_author: Optional[str] = Field(
+        None,
+        description="The git commit author of the last commit that modified this file. DO NOT DEFINE THIS FIELD IN YOUR SPEC. It will be overwritten by the CI.",
+    )
+    commit_author_email: Optional[str] = Field(
+        None,
+        description="The git commit author email of the last commit that modified this file. DO NOT DEFINE THIS FIELD IN YOUR SPEC. It will be overwritten by the CI.",
+    )


This is already declared in #37802
You might need to gt restack

bnchrch · 2024-05-09T14:18:22Z

Merge activity

May 9, 10:18 AM EDT: @bnchrch started a stack merge that includes this pull request via Graphite.
May 9, 10:29 AM EDT: Graphite rebased this pull request as part of a merge.
May 9, 10:45 AM EDT: @bnchrch merged this pull request with Graphite.

This was referenced May 3, 2024

Upload git info with metadata file (#1) #37802

Merged

Notify commit owner via slack message #37803

Merged

bnchrch marked this pull request as ready for review May 3, 2024 18:43

bnchrch requested review from a team and alafanechere May 3, 2024 18:43

bnchrch mentioned this pull request May 3, 2024

CI: Add attention messages and commit links to slack lifecycle #32715

Closed

bnchrch force-pushed the bnchrch/registry/add-provience-metadata branch from 64a89fa to 9d7c6c2 Compare May 3, 2024 20:57

bnchrch mentioned this pull request May 3, 2024

feat(registry): add cdk version #37809

Merged

natikgadzhi approved these changes May 5, 2024

View reviewed changes

octavia-squidington-iv requested a review from a team May 5, 2024 14:19

alafanechere reviewed May 6, 2024

View reviewed changes

alafanechere approved these changes May 6, 2024

View reviewed changes

bnchrch force-pushed the bnchrch/registry/add-slack-tag-notifiation branch from 2b03faf to d5ce17d Compare May 7, 2024 19:21

bnchrch force-pushed the bnchrch/registry/add-provience-metadata branch from 9d7c6c2 to 2c8909d Compare May 7, 2024 19:22

bnchrch force-pushed the bnchrch/registry/add-slack-tag-notifiation branch from d5ce17d to 78f8640 Compare May 8, 2024 21:34

bnchrch force-pushed the bnchrch/registry/add-provience-metadata branch from 2c8909d to 654f28d Compare May 8, 2024 21:34

bnchrch force-pushed the bnchrch/registry/add-slack-tag-notifiation branch from 78f8640 to 9c54656 Compare May 9, 2024 14:19

Base automatically changed from bnchrch/registry/add-slack-tag-notifiation to master May 9, 2024 14:28

bnchrch added 2 commits May 9, 2024 14:29

Add provinance to the registry entry

e4c725b

feat: add last modified time

926ee78

bnchrch force-pushed the bnchrch/registry/add-provience-metadata branch from 654f28d to 926ee78 Compare May 9, 2024 14:29

bnchrch merged commit f029414 into master May 9, 2024
32 checks passed

bnchrch deleted the bnchrch/registry/add-provience-metadata branch May 9, 2024 14:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add provinance to the registry entry #37804

Add provinance to the registry entry #37804

bnchrch commented May 3, 2024 •

edited

Loading

vercel bot commented May 3, 2024 •

edited

Loading

bnchrch commented May 3, 2024 •

edited

Loading

natikgadzhi left a comment

natikgadzhi May 5, 2024

natikgadzhi May 5, 2024

natikgadzhi May 5, 2024

alafanechere May 6, 2024

alafanechere May 6, 2024

alafanechere May 6, 2024

alafanechere May 6, 2024

bnchrch commented May 9, 2024 •

edited

Loading



		def default_none_to_dict(value, key, obj):
		"""Set the value of a key in a dictionary to an empty dictionary if the value is None.

		@@ -87,6 +117,11 @@ class BreakingChangeScope(BaseModel):
		__root__: StreamBreakingChangeScope = Field(..., description="A scope that can be used to limit the impact of a breaking change.")


		class GeneratedFields(BaseModel):

Add provinance to the registry entry #37804

Add provinance to the registry entry #37804

Conversation

bnchrch commented May 3, 2024 • edited Loading

What

vercel bot commented May 3, 2024 • edited Loading

bnchrch commented May 3, 2024 • edited Loading

natikgadzhi left a comment

Choose a reason for hiding this comment

natikgadzhi May 5, 2024

Choose a reason for hiding this comment

natikgadzhi May 5, 2024

Choose a reason for hiding this comment

natikgadzhi May 5, 2024

Choose a reason for hiding this comment

alafanechere May 6, 2024

Choose a reason for hiding this comment

alafanechere May 6, 2024

Choose a reason for hiding this comment

alafanechere May 6, 2024

Choose a reason for hiding this comment

alafanechere May 6, 2024

Choose a reason for hiding this comment

bnchrch commented May 9, 2024 • edited Loading

Merge activity

bnchrch commented May 3, 2024 •

edited

Loading

vercel bot commented May 3, 2024 •

edited

Loading

bnchrch commented May 3, 2024 •

edited

Loading

bnchrch commented May 9, 2024 •

edited

Loading