-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix cdk bug to send legacy format if connector overrides read() #16566
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!!
# send state in the old format for these connectors, but once all have been upgraded, this block can be removed | ||
# vars(self.__class__) checks if the current class directly overrides the read() function | ||
if "read" in vars(self.__class__): | ||
return state_obj |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we should log something warning the connector developer to do… something soon
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agreed, i think i'll jot this down to follow up and add this to the next checkin of per-stream. not sure the best way of surfacing it, maybe a logger message, but not really something we should be surfacing to customers viewing logs
/publish-cdk dry-run=false
|
…ytehq#16566) * fix cdk bug to send legacy format if connector overrides read() * fix comment * update changelog and setup.py
…ytehq#16566) * fix cdk bug to send legacy format if connector overrides read() * fix comment * update changelog and setup.py
What
For connectors like
source-faker
that override theread()
function, they might not be able to support the new list based format that state arrives in. To guard against this, we should continue to send the old format for these types of connectors.How
While reading in state from
source.py
we can determine if the connector directly overrides theread
function usingvars(self.__class__)
. In that case, we continue to send the flat state object instead of state in the new formatTested against
source-faker
which implements its own read() which receives the old state format