-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[#14361] Adding new generator for configuration based source template #14887
Conversation
…es that aren't relevant to config based connectors
from setuptools import find_packages, setup | ||
|
||
MAIN_REQUIREMENTS = [ | ||
"airbyte-cdk~=0.1.64", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there a way to always get the latest version when running the generator?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we always want the latest bumps in major versions too? The ~=
should continue to get the latest patch. I can see extending this out to the minor version, but we may still want to be restrictive against getting the latest when theres a major version bump.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's fine to freeze the requirements, but i would expect the generator to create a connector using the latest version of the cdk
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looking into it a bit, it is possible since we could parse the cdk's setup.py
file to find the latest, but I don't see a great way of doing that. maybe some derivative of using the setuptools
package, but that normally gets the current working version. Not super blocking so I'm gonna move forward with merging it in.
What
Add a new template to
./generator.sh
that allows developers to instantiate a new configuration based source connector. This automatically generates the source yaml parser and a limited set of components to get a basic connector working.After generating the boilerplate, the remaining requirements for the implementer are as follows:
spec.yaml
/schemas
config.json
url_base
,token
,path
How
Adapts the existing http-source generator, but the key areas it diverges is in how
source.py
is implemented in order to parse a YAML file into a fully functioning source that adheres to the CDK interface. It also defines the baseline YAML config.Recommended reading order
plopfile.js
source.py.hbs
{{snakeCase name}}.yaml.hbs
README.md.hbs
schemas/TODO.md.hbs