Support creating destinations in the Python CDK

[sherifnada]

Tell us about the problem you're trying to solve

I want to write a destination in Python

Describe the solution you’d like

• support creating destinations in the python CDK #4442
• Create a connector template for writing a Python destination #4443
• Write docs for how to write a destination in Python #4444
• Create an example Python Destination #4697
• Enable Destination Acceptance Tests cross-language support #4698

Describe the alternative you’ve considered or used

Additional context

Are you willing to submit a PR?

[AetherUnbound]

Hello there! I'm interested in trying to build a destination connector with Python. It looks like the places where this might need to be added are to the Destination integration and the base-python entrypoint (a new bash entrypoint similar to javabase.sh might also be needed to ensure stdin is redirected appropriately). Is that correct? Are there any other major pieces I'm missing?

[sherifnada]

@AetherUnbound you are correct, however the base-python package is deprecated and we're now using the Python CDK. The equivalent class would be the Destination class

I'm currently working on adding these classes, aiming to close #4442 and #4443 this week, hopefully #4444 as well. What's your timeline for writing a destination?

[AetherUnbound]

Oh excellent! Thanks for the pointer on base-python's deprecation.

I'm personally hoping to start building the destination later this week as well, though I don't say that as a desire to alter or speed up your timeline. I'm happy to help out where I can if there's anything I can jump on/assist with! As someone relatively new to Airbyte, I wouldn't mind testing out the destination creation process and possibly providing feedback 😄

[sherifnada]

that's great to hear! Would love your thoughts on it -- you would be the first non-Airbyter to use it :) Will ping you again as soon as we have something presentable.

[AetherUnbound]

Hey again @sherifnada, thanks for tackling all of these tickets! I'm trying this out locally now, and it seems that a newer version of airbyte-cdk which has the destination spec hasn't been released to PyPI yet. Do you know if that would be possible or what the timeline on that would be?

[sherifnada]

@AetherUnbound fixed -- there should be a 0.1.6 available on Pypi now. Apologies, it had been released to test Pypi for testing but not to the official one. Can you try again?

[AetherUnbound]

I was able to install just fine now, thank you!

[olivermeyer]

Happy to read that the CDK now supports destinations - @sherifnada is there any kind of documentation I could read? I'm looking at different reverse ETL tools and Airbyte with custom destinations is a candidate, especially since we're using it already.

Found https://github.com/airbytehq/airbyte/pull/4800/files feel free to ignore :-)

[sherifnada]

@olivermeyer writing destinations is fully functional but doesn't yet have acceptance tests. It just means you'll need to be extra cautious and write more tests, but it should work :)