-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🎉 Source Square: migrate to Beta (YAML) #19369
Conversation
/test connector=connectors/source-square
Build PassedTest summary info:
|
6348742
to
280f228
Compare
/test connector=connectors/source-square
Build FailedTest summary info:
|
/test connector=connectors/source-square
Build FailedTest summary info:
Build PassedTest summary info:
|
airbyte-integrations/connectors/source-square/source_square/components.py
Outdated
Show resolved
Hide resolved
|
||
|
||
@dataclass | ||
class SquareSubstreamSlicer(SquareSlicer): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what missing features is this covering for? Would love to find a way to reuse existing components or make this generalizable enough that it can be reused
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class is similar to SubstreamSlicer
, and even can be used here, but it utilised 1:1 relation (parent:child). In my case I need to get all parent records (locations
stream), get id
from them and iterate in batch (10 ids in request)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we want to batch 10 the slices in batches of 10?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you create an issue advocating for adding this functionality into the lowcode CDK as described here?
If I'm understanding correctly, the additional features needed from the substream slicer are:
- parsing a subfield of the parent record/slice
- batching multiple parent streams in the output
same thing as mentioned above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this looks pretty good. can you document the custom components so we know the intent and can find ways to generalize them in the future?
I'll approve afterwards
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only light comments, but also echoing Alex's feedback to document what specific functionality these custom component solve for in the context of the integration.
airbyte-integrations/connectors/source-square/source_square/components.py
Outdated
Show resolved
Hide resolved
/test connector=connectors/source-square
Build FailedTest summary info:
|
/test connector=connectors/source-square
Build PassedTest summary info:
|
/publish connector=connectors/source-square auto-bump-version=false
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/source-square
if you have connectors that successfully published but failed definition generation, follow step 4 here |
What
migrate to YAML
How
Describe the solution
Recommended reading order
x.java
y.python
🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereUpdating a connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changesTests
Unit
Put your unit tests output here.
Integration
Put your integration tests output here.
Acceptance
Put your acceptance tests output here.