-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable low-code CDK users to specify schema in the manifest #20375
Conversation
8d86dc7
to
d4bb23c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great to me! Anything in particular you'd like feedback on?
Thanks @sherifnada - is there documentation that I should update as part of this PR? |
f3d0a00
to
71782a8
Compare
@clnoll updating the documentation is a good idea! We should update the doc that describes how to define the schema and the tutorial |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm! Approved pending bumping the CDK version in setup.py an updating the changelog
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One thought to discuss about the name of the component, but the code itself looks good!
I think another interesting discussion might be how we want to organize the schemas now that we can define them in the manifest itself. I saw in an earlier draft that it was in definitions
which technically works because we can reference any part of the YAML. But it starts to make the manifest pretty unreadable because schemas will eventually take up a bulk of the file and developers will have to either do searches or just scroll around a ton. I think we might want to add in another top level field in the manifest for schemas
at the bottom of manifest and we can encourage the convention to only add them there. And then all the various parts don't get split up by a massive blob of schema yaml.
It goes a little bit outside the scope of the work you're doing, but just something I thought of while reviewing this.
|
||
|
||
@dataclass | ||
class StaticSchemaLoader(SchemaLoader): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Something I'm thinking about is whether StaticSchemaLoader
is the right name for this component. Static feels like too vague a term because technically all our connector's that reference some underlying example_schema.json
are also static, just not living within the manifest file itself.
InlineSchemaLoader(SchemaLoader)
might be a good alternative since what we want to highlight from the component is that its defined within the manifest, not that its static.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great suggestions, @brianjlai - I like InlineSchemaLoader
and will update the name in the relevant places.
I'll also update the example inline schemas in #20405 to use a top-level schemas
key. I believe that will require a small change to this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pushed the suggested changes to this PR and #20405.
5bbfe37
to
625d612
Compare
/publish-cdk dry-run=true
|
/publish-cdk dry-run=false
|
What
Allows users of the low-code CDK to define stream schemas inline in the manifest.
How
Adds a new
StaticSchemaLoader
to the list of schema loader options. When a user specifies that theStaticSchemaLoader
should be used with a stream, they must also supply theschema
property.Recommended reading order
airbyte-cdk/python/airbyte_cdk/sources/declarative/schema/static_schema_loader.py
airbyte-cdk/python/airbyte_cdk/sources/declarative/config_component_schema.json
airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/class_types_registry.py
airbyte-cdk/python/airbyte_cdk/sources/declarative/schema/__init__.py
airbyte-cdk/python/unit_tests/sources/declarative/schema/test_static_schema_loader.py
🚨 User Impact 🚨
No breaking changes; the output of
discovery
is exactly the same with inline schema versus on-disk schema.Pre-merge Checklist
Tests
Unit
Put your unit tests output here.
Integration
Put your integration tests output here.
Acceptance
Put your acceptance tests output here.