Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python sources refactoring #592

Merged
merged 16 commits into from
Oct 16, 2020
Merged

Python sources refactoring #592

merged 16 commits into from
Oct 16, 2020

Conversation

michel-tricot
Copy link
Contributor

@michel-tricot michel-tricot commented Oct 16, 2020 •
edited
Loading

What

Few refactorings:

  • Install dependencies as editable to play better with the local environment and to allow developers to use the regular python tools
  • split up airbyte protocol in submodules
  • make entrypoint for python part of the protocol module and expose it as a binary
  • use a default implementation for spec since it generally always need the same file
  • create a template for python sources
  • create a template for singer sources (we can do better at simplifying the implementation)
  • start "grouping" the source specific inputs in dockerfiles so it is less error prone
  • we were using a global variable in the entrypoint instead of the source attribute
  • add the logger to spec
  • always copy spec.json in /airbyte
  • very barebone documentation in template/python-source
  • create test main to debug in IntelliJ
  • bump up com.diffplug.gradle.spotless to fix a out of bound exception for no good reasons

there should be no behavior changes

Checklist

  • Play better with gradle (future PR)
  • Document how to develop in IJ (future PR)
  • generate egg-info files in build (future PR)
  • Simplify singer implementation even more (idea)
  • Use more convention over configurations to simplify implementation (idea)
  • Add structure for python tests (idea)

Recommended reading order

  1. base-python
  2. base-singer
  3. template/python-source

@michel-tricot michel-tricot added type/enhancement New feature or request area/connectors Connector related issues type/refactoring Focused on refactoring. area/developer Everything related to build, developer UX labels Oct 16, 2020
@michel-tricot michel-tricot added this to the v0.2.0 milestone Oct 16, 2020
@michel-tricot michel-tricot requested a review from jrhizor October 16, 2020 05:57
@michel-tricot michel-tricot self-assigned this Oct 16, 2020
jrhizor
jrhizor approved these changes Oct 16, 2020
View reviewed changes
Copy link
Contributor

@jrhizor jrhizor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this!

airbyte-integrations/base-python/.dockerignore
!Dockerfile
!base.py
!airbyte_protocol
build
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use the exclusion pattern for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question :) we want to have a small context sent to docker for our core piece since we don't want to pull all the compiled and dist files but for connectors, having it it as an exclude list is very confusing to someone who tries to create a new connector since it becomes yet another place that you need to update.

Copy link
Contributor

@jrhizor jrhizor Oct 16, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright sounds fine. We will have to be careful once these have python tests though, potentially with secrets.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not a problem. they won't end up in the image just in the context used to build the image

airbyte-integrations/base-python/Dockerfile Show resolved Hide resolved
airbyte-integrations/singer/base-singer/.dockerignore
!base_singer/__init__.py
!base_singer/singer_helpers.py
!setup.py
build
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

airbyte-integrations/singer/stripe_abprotocol/source/.dockerignore Show resolved Hide resolved
airbyte-integrations/template/python-source/.dockerignore Show resolved Hide resolved
airbyte-integrations/template/singer-source/spec.json Outdated
"type": "object",
"required": ["start_date", "base"],
"additionalProperties": false,
"properties": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the example properties here shouldn't belong to the exchangerate api

build.gradle Outdated Show resolved Hide resolved
tools/gradle/commons/integrations/python.gradle Outdated Show resolved Hide resolved
airbyte-integrations/template/singer-source/template_singer_source/source.py
from base_singer import SingerHelper


class TemplateSingerSource(Source):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also feels like it shouldn't be based on the exchange rate api. Either we should actually use the exhangerates api as the template we point devs to or we should use something that won't diverge.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed. Let's take that conversation to the other PR with the singer shim

airbyte-integrations/base-python/airbyte_protocol/entrypoint.py Outdated Show resolved Hide resolved
@michel-tricot michel-tricot merged commit bc56f02 into master Oct 16, 2020
@michel-tricot michel-tricot deleted the mt-python-palyt branch October 16, 2020 22:58
sherifnada pushed a commit that referenced this pull request Oct 23, 2020
@michel-tricot @sherifnada
Python sources refactoring (#592)
ee45b04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jrhizor jrhizor jrhizor approved these changes

Assignees

@michel-tricot michel-tricot

Labels
area/connectors Connector related issues area/developer Everything related to build, developer UX type/enhancement New feature or request type/refactoring Focused on refactoring.
Projects
None yet
Milestone
v0.2.0
Development

Successfully merging this pull request may close these issues.
2 participants
@michel-tricot @jrhizor