Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix connector base versioning #2726

Merged
merged 5 commits into from
Apr 3, 2021
Merged

fix connector base versioning #2726

merged 5 commits into from
Apr 3, 2021

Conversation

jrhizor
Copy link
Contributor

@jrhizor jrhizor commented Apr 2, 2021

Our Python connectors have not picked up changes from the bases. This leads to problems like JSON decoding errors if a configured catalog has something like primary key configuration which wasn't present in an older base version.

This PR changes everything to use versions and will require testing/publishing all Python-based connectors.

@auto-assign auto-assign bot requested review from davinchia and michel-tricot April 2, 2021 23:26
@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 2, 2021 •
edited by github-actions bot
Loading

/publish connector=bases/airbyte-protocol

🕑 bases/airbyte-protocol https://github.com/airbytehq/airbyte/actions/runs/713054251
❌ bases/airbyte-protocol https://github.com/airbytehq/airbyte/actions/runs/713054251

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 2, 2021 •
edited
Loading

My plan is to serially run the following slash commands:

  • /publish connector=bases/airbyte-protocol
  • /publish connector=bases/base-python
  • /publish connector=bases/base-singer
  • /publish connector=bases/base-python-test
  • /publish connector=bases/base-normalization
  • /publish connector=connectors/source-exchangeratesapi-singer
  • bump and test demo.airbyte.io version to check for PK configuration
  • publish all. of. the. rest.

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 2, 2021 •
edited by github-actions bot
Loading

/publish connector=bases/airbyte-protocol

🕑 bases/airbyte-protocol https://github.com/airbytehq/airbyte/actions/runs/713094822
✅ bases/airbyte-protocol https://github.com/airbytehq/airbyte/actions/runs/713094822

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=bases/base-python

🕑 bases/base-python https://github.com/airbytehq/airbyte/actions/runs/713109350
✅ bases/base-python https://github.com/airbytehq/airbyte/actions/runs/713109350

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=bases/base-singer

🕑 bases/base-singer https://github.com/airbytehq/airbyte/actions/runs/713122064
✅ bases/base-singer https://github.com/airbytehq/airbyte/actions/runs/713122064

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=bases/base-python-test

🕑 bases/base-python-test https://github.com/airbytehq/airbyte/actions/runs/713132404
✅ bases/base-python-test https://github.com/airbytehq/airbyte/actions/runs/713132404

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/713132849
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/713132849

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-exchangeratesapi-singer

🕑 connectors/source-exchangeratesapi-singer https://github.com/airbytehq/airbyte/actions/runs/713133165
✅ connectors/source-exchangeratesapi-singer https://github.com/airbytehq/airbyte/actions/runs/713133165

cgardens
cgardens approved these changes Apr 3, 2021
View reviewed changes
Copy link
Contributor

@cgardens cgardens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are a hero.

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021

I manually deleted the images I pushed up to this point so they can pick up new versions from 9ede7f6, which is the piece that is truly responsible for this problem.

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=bases/airbyte-protocol

🕑 bases/airbyte-protocol https://github.com/airbytehq/airbyte/actions/runs/713198448
✅ bases/airbyte-protocol https://github.com/airbytehq/airbyte/actions/runs/713198448

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=bases/base-python

🕑 bases/base-python https://github.com/airbytehq/airbyte/actions/runs/713213988
✅ bases/base-python https://github.com/airbytehq/airbyte/actions/runs/713213988

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=bases/base-singer

🕑 bases/base-singer https://github.com/airbytehq/airbyte/actions/runs/713225669
✅ bases/base-singer https://github.com/airbytehq/airbyte/actions/runs/713225669

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=bases/base-python-test

🕑 bases/base-python-test https://github.com/airbytehq/airbyte/actions/runs/713240169
✅ bases/base-python-test https://github.com/airbytehq/airbyte/actions/runs/713240169

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/713240482
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/713240482

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-exchangeratesapi-singer

🕑 connectors/source-exchangeratesapi-singer https://github.com/airbytehq/airbyte/actions/runs/713240891
✅ connectors/source-exchangeratesapi-singer https://github.com/airbytehq/airbyte/actions/runs/713240891

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021

Yay! It fixed the problem with exchangeratesapi.

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-tempo

🕑 connectors/source-tempo https://github.com/airbytehq/airbyte/actions/runs/713269682
✅ connectors/source-tempo https://github.com/airbytehq/airbyte/actions/runs/713269682

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-google-workspace-admin-reports

🕑 connectors/source-google-workspace-admin-reports https://github.com/airbytehq/airbyte/actions/runs/713270035
✅ connectors/source-google-workspace-admin-reports https://github.com/airbytehq/airbyte/actions/runs/713270035

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-gitlab-singer

🕑 connectors/source-gitlab-singer https://github.com/airbytehq/airbyte/actions/runs/713270270
✅ connectors/source-gitlab-singer https://github.com/airbytehq/airbyte/actions/runs/713270270

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-instagram

🕑 connectors/source-instagram https://github.com/airbytehq/airbyte/actions/runs/713270398
✅ connectors/source-instagram https://github.com/airbytehq/airbyte/actions/runs/713270398

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-google-directory

🕑 connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/713270422
✅ connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/713270422

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-appstore-singer

🕑 connectors/source-appstore-singer https://github.com/airbytehq/airbyte/actions/runs/713270489
✅ connectors/source-appstore-singer https://github.com/airbytehq/airbyte/actions/runs/713270489

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-looker

🕑 connectors/source-looker https://github.com/airbytehq/airbyte/actions/runs/713270595
✅ connectors/source-looker https://github.com/airbytehq/airbyte/actions/runs/713270595

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-greenhouse

🕑 connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/713271877
❌ connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/713271877

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-slack-singer

🕑 connectors/source-slack-singer https://github.com/airbytehq/airbyte/actions/runs/713272013
✅ connectors/source-slack-singer https://github.com/airbytehq/airbyte/actions/runs/713272013

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-braintree-singer

🕑 connectors/source-braintree-singer https://github.com/airbytehq/airbyte/actions/runs/713272061
✅ connectors/source-braintree-singer https://github.com/airbytehq/airbyte/actions/runs/713272061

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-freshdesk

🕑 connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/713272113
✅ connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/713272113

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-twilio-singer

🕑 connectors/source-twilio-singer https://github.com/airbytehq/airbyte/actions/runs/713272141
✅ connectors/source-twilio-singer https://github.com/airbytehq/airbyte/actions/runs/713272141

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-http-request

🕑 connectors/source-http-request https://github.com/airbytehq/airbyte/actions/runs/713272755
✅ connectors/source-http-request https://github.com/airbytehq/airbyte/actions/runs/713272755

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-shopify-singer

🕑 connectors/source-shopify-singer https://github.com/airbytehq/airbyte/actions/runs/713272702
✅ connectors/source-shopify-singer https://github.com/airbytehq/airbyte/actions/runs/713272702

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/713272787
❌ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/713272787

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-facebook-marketing

🕑 connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/713272851
❌ connectors/source-facebook-marketing https://github.com/airbytehq/airbyte/actions/runs/713272851

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-googleanalytics-singer

🕑 connectors/source-googleanalytics-singer https://github.com/airbytehq/airbyte/actions/runs/713272856
❌ connectors/source-googleanalytics-singer https://github.com/airbytehq/airbyte/actions/runs/713272856

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-mailchimp

🕑 connectors/source-mailchimp https://github.com/airbytehq/airbyte/actions/runs/713272998
✅ connectors/source-mailchimp https://github.com/airbytehq/airbyte/actions/runs/713272998

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-stripe-singer

🕑 connectors/source-stripe-singer https://github.com/airbytehq/airbyte/actions/runs/713273296
✅ connectors/source-stripe-singer https://github.com/airbytehq/airbyte/actions/runs/713273296

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-salesforce-singer

🕑 connectors/source-salesforce-singer https://github.com/airbytehq/airbyte/actions/runs/713273357
❌ connectors/source-salesforce-singer https://github.com/airbytehq/airbyte/actions/runs/713273357

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-google-sheets

🕑 connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/713273635
✅ connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/713273635

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-marketo-singer

🕑 connectors/source-marketo-singer https://github.com/airbytehq/airbyte/actions/runs/713273726
✅ connectors/source-marketo-singer https://github.com/airbytehq/airbyte/actions/runs/713273726

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-sendgrid

🕑 connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/713273734
✅ connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/713273734

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-recurly

🕑 connectors/source-recurly https://github.com/airbytehq/airbyte/actions/runs/713273900
✅ connectors/source-recurly https://github.com/airbytehq/airbyte/actions/runs/713273900

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-github-singer

🕑 connectors/source-github-singer https://github.com/airbytehq/airbyte/actions/runs/713274078
✅ connectors/source-github-singer https://github.com/airbytehq/airbyte/actions/runs/713274078

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-google-adwords-singer

🕑 connectors/source-google-adwords-singer https://github.com/airbytehq/airbyte/actions/runs/713274247
❌ connectors/source-google-adwords-singer https://github.com/airbytehq/airbyte/actions/runs/713274247

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/713274585
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/713274585

@jrhizor
Copy link
Contributor Author

jrhizor commented Apr 3, 2021 •
edited
Loading

Looks like all pushed except for:

  • source-mixpanel-singer (standard test failed) (failing since Jan 20)
  • source-zendesk-support-singer (standard test failed) (usually has been passing)
  • source-greenhouse (standard test failed) (last 10 runs failed)
  • source-facebook-marketing (python integration tests failed) (last 10 runs failed, but it looks more recent)
  • source-googleanalytics-singer (taking 1.5hr so I killed it) (looks spotty testing in general)
  • source-salesforce-singer (has no integration test) (had tests Jan 21 but not since)
  • source-google-adwords-singer (standard test failed) (last 10 failed, fairly recent)

I'm pushing these all without integration tests from my laptop.

@jrhizor jrhizor merged commit 0609011 into master Apr 3, 2021
@jrhizor jrhizor deleted the jrhizor/fix-connector-base-versioning branch April 3, 2021 05:46
davinchia pushed a commit that referenced this pull request Apr 6, 2021
@jrhizor
fix connector base versioning (#2726)
a7cbdfd 
* fix connector base versioning

* only run integration tests in non-bases directories

* upgrade datamodel-code-generator which coincidentally has already fixed this issue with generating List of Lists

* bump forgotten hubspot version

* fix build
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cgardens cgardens cgardens approved these changes

@michel-tricot michel-tricot michel-tricot approved these changes

@davinchia davinchia Awaiting requested review from davinchia

Assignees
No one assigned
Labels
connectors/source/appstore-singer connectors/source/facebook-marketing connectors/sources-api
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jrhizor @michel-tricot @cgardens @karinakuz @igrankova