🎉 New Destination: Google Firestore #7231

ad-m · 2021-10-21T05:56:10Z

What

That PR add connectors to write to Google Firestore. Connector have been implemented in Python.

That PR is part of Airbyte hacktoberfest contest

How

Describe the solution

I have implemented connectors in Python. It support application default credentials (including Google metadata service when Airbyte is running on Google-backed insfructure eg. virtual machines) and JSON credentials.

I have been inspired by existing kvdb destination to keep code coherent.

Considering the existing cooperation between Google and Airbyte, I am not sure how to proceed with the access data to the demo environment, and also to prove the operation.

$ python -m pytest integration_tests --full-trace
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.9.4, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /home/adas/Devel/airbyte/airbyte-integrations/connectors/destination-firestore/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/adas/Devel/airbyte, configfile: pytest.ini
collected 4 items                                                                                                                                                                                                 

integration_tests/integration_test.py::test_check_valid_config PASSED
integration_tests/integration_test.py::test_check_invalid_config PASSED
integration_tests/integration_test.py::test_write_append PASSED
integration_tests/integration_test.py::test_write_overwrite PASSED

================================================================================================ 4 passed in 5.80s ================================================================================================

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

CLAassistant · 2021-10-21T05:56:14Z

All committers have signed the CLA.

harshithmullapudi · 2021-10-21T08:57:44Z

Hey @ad-m thanks for your contribution. Can you share you credentials over slack (DM) so that I can test this source.

ad-m · 2021-10-21T22:24:21Z

@harshithmullapudi I believe I already shared credentials via Slack.

marcosmarxm · 2021-10-25T19:39:39Z

thanks for this contribution @ad-m I requested the review to our team.

mik-laj · 2021-10-25T23:41:34Z

@marcosmarxm;what does gl label mean?

antixar

Hi @ad-m,
Thank you for this PR.
Could you correct the following points?

add Python annotation to your writer.py
does this library support the native oauth2 auth (not system account)? I mean some refresh_token with client_id/client_secret. I ask because The Aitbyte project is planning to add support of Oauth2 user authorization for connectors which can use it.

antixar · 2021-10-28T08:19:51Z

airbyte-integrations/connectors/destination-firestore/destination_firestore/writer.py

+    def __init__(self, project_id=None, credentials_json=None, collection=None):
+        connection = {}
+
+        if project_id:


is the project_id optional or not? I see to it is required into your spec file

You are right, it's requried. Fixed.

airbyte-integrations/connectors/destination-firestore/destination_firestore/writer.py

antixar · 2021-10-28T09:06:07Z

airbyte-integrations/connectors/destination-firestore/destination_firestore/writer.py

+from google.oauth2 import service_account
+


this library is not set as dependence obviously into the setup file. I see that it is installed with the google-cloud-firestore lib but it isn't evident. Could you add all used libraries into the setup file?

All used libraries added into the setup file.

ad-m · 2021-10-29T15:32:48Z

add Python annotation to your writer.py

OK.

does this library support the native oauth2 auth (not system account)? I mean some refresh_token with client_id/client_secret. I ask because The Aitbyte project is planning to add support of Oauth2 user authorization for connectors which can use it.

Yes, that library might be used for OAuth 2.0 flow: https://google-auth.readthedocs.io/en/master/user-guide.html#user-credentials

ad-m · 2021-11-04T18:07:41Z

@marcosmarxm is there anything how I can push forward that PR?

harshithmullapudi · 2021-11-05T06:48:32Z

@antixar is it a +1 for you on this PR? can I process to publish this ?

marcosmarxm · 2021-11-05T15:31:10Z

@ad-m please share the acceptance tests screenshot showing those tests are passing too.

ad-m · 2021-11-05T21:16:37Z

@marcosmarxm is there any documentation regarding how to use acceptance tests for destination connector in Python?

I can see that there are open issue to enable Destination Acceptance Tests cross-language support ( #4698 ) and improve documentation for DAT ( airbytehq/airbyte-internal-issues#268 ). However, these issues do not imply that such a standard exists for destinations connector already.

Tests pass despite of small tweaks since PR have been created:

ad-m · 2021-11-08T23:41:46Z

@marcosmarxm , according to @sherifnada :

Currently the only available option is to write acceptance tests in java

As it stands, should I try to create a mixed connector that would run Java acceptance tests for the Python connector (I haven't analyzed whether it makes sense to accept acceptance tests due to the level of abstraction)? I do not have a full picture of the management and standards of the project yet, hence this question. I have the impression that mixing languages in one connector will be problematic in the long run.

antixar

Looks good! Sorry for delay of this review

marcosmarxm · 2021-11-15T16:19:12Z

Hello! Sorry to not finished your contribution before the date stipulated in the contest. All contributions made before 15-November are eligible to receive the award. We're trying to review your contribution as soon as possible.

marcosmarxm · 2021-11-15T16:19:28Z

@harshithmullapudi can you ship this one?

ad-m · 2021-11-15T19:16:39Z

Hello! Sorry to not finished your contribution before the date stipulated in the contest. All contributions made before 15-November are eligible to receive the award. We're trying to review your contribution as soon as possible.

Is there anything how I can help to handle that PR?

harshithmullapudi · 2021-11-16T11:35:39Z

Yeah @marcosmarxm just trying to understand is there anything more we need to do other than publishing the connector ?

harshithmullapudi · 2021-11-21T13:05:01Z

@ad-m hey were you able to build docker image of this connector ?

* destination-firestore: add connector * destination-firestore: Add connector to indexes in docs * destination-firestore: fix sync mode spec * destination-firestore: fix typo after connector rename * destination-firestore: fix required field in spec * destination firestore: fix code formatting * destionation-firestore: add python adnotation for writer * destination firestore: add all used libraries into the setup file * destionation-firestore: fix python formatting

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 21, 2021

octavia-squidington-iii added the community label Oct 21, 2021

ad-m changed the title ~~🎉 Destination Firestore: add initial implementation of new connection~~ 🎉 New Destionation: Google Firestore Oct 21, 2021

ad-m changed the title ~~🎉 New Destionation: Google Firestore~~ 🎉 New Destination: Google Firestore Oct 21, 2021

ad-m mentioned this pull request Oct 21, 2021

New Source: Firebase Realtime Database airbytehq/connector-contest#20

Closed

harshithmullapudi added the hacktober label Oct 21, 2021

harshithmullapudi mentioned this pull request Oct 21, 2021

New Destination Firestore: create dev account and sandbox credentials #7236

Closed

harshithmullapudi self-assigned this Oct 21, 2021

ad-m mentioned this pull request Oct 21, 2021

Firestore source request #3323

Closed

ad-m force-pushed the firestore branch from f381355 to 12d480a Compare October 22, 2021 02:24

ad-m added 6 commits October 22, 2021 07:33

destination-firestore: add connector

cceb90d

destination-firestore: Add connector to indexes in docs

ebbedd9

destination-firestore: fix sync mode spec

817e3d5

destination-firestore: fix typo after connector rename

a571b6e

destination-firestore: fix required field in spec

b77ceea

destination firestore: fix code formatting

e2c1859

ad-m force-pushed the firestore branch from 12d480a to e2c1859 Compare October 22, 2021 05:34

marcosmarxm added the gl label Oct 25, 2021

antixar suggested changes Oct 28, 2021

View reviewed changes

destionation-firestore: add python adnotation for writer

03f9efb

ad-m added 2 commits October 29, 2021 17:33

destination firestore: add all used libraries into the setup file

72ed2bb

destionation-firestore: fix python formatting

34715cf

marcosmarxm assigned antixar Nov 1, 2021

ad-m requested a review from antixar November 4, 2021 18:06

ad-m mentioned this pull request Nov 8, 2021

Enable Destination Acceptance Tests cross-language support #4698

Open

9 tasks

antixar approved these changes Nov 12, 2021

View reviewed changes

Merge branch 'master' into firestore

eced407

harshithmullapudi mentioned this pull request Nov 21, 2021

Publish Destination Firestore #8158

Merged

harshithmullapudi merged commit 519ea2a into airbytehq:master Nov 21, 2021

jrhizor mentioned this pull request Nov 23, 2021

Bump Airbyte version from 0.32.5-alpha to 0.32.6-alpha #8192

Merged

karinakuz added connectors/destinations-database connectors/destination/google_firestore labels Jan 14, 2022

marcosmarxm removed the hacktober label Oct 20, 2022

octavia-squidington-iv added bounty connectors/destination/firestore labels Oct 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Destination: Google Firestore #7231

🎉 New Destination: Google Firestore #7231

ad-m commented Oct 21, 2021 •

edited by harshithmullapudi

Loading

CLAassistant commented Oct 21, 2021 •

edited

Loading

harshithmullapudi commented Oct 21, 2021

ad-m commented Oct 21, 2021

marcosmarxm commented Oct 25, 2021

mik-laj commented Oct 25, 2021

antixar left a comment

antixar Oct 28, 2021

ad-m Oct 29, 2021

antixar Oct 28, 2021

ad-m Oct 29, 2021

ad-m commented Oct 29, 2021 •

edited

Loading

ad-m commented Nov 4, 2021

harshithmullapudi commented Nov 5, 2021

marcosmarxm commented Nov 5, 2021

ad-m commented Nov 5, 2021

ad-m commented Nov 8, 2021

antixar left a comment

marcosmarxm commented Nov 15, 2021

marcosmarxm commented Nov 15, 2021

ad-m commented Nov 15, 2021

harshithmullapudi commented Nov 16, 2021

harshithmullapudi commented Nov 21, 2021

🎉 New Destination: Google Firestore #7231

🎉 New Destination: Google Firestore #7231

Conversation

ad-m commented Oct 21, 2021 • edited by harshithmullapudi Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

CLAassistant commented Oct 21, 2021 • edited Loading

harshithmullapudi commented Oct 21, 2021

ad-m commented Oct 21, 2021

marcosmarxm commented Oct 25, 2021

mik-laj commented Oct 25, 2021

antixar left a comment

Choose a reason for hiding this comment

antixar Oct 28, 2021

Choose a reason for hiding this comment

ad-m Oct 29, 2021

Choose a reason for hiding this comment

antixar Oct 28, 2021

Choose a reason for hiding this comment

ad-m Oct 29, 2021

Choose a reason for hiding this comment

ad-m commented Oct 29, 2021 • edited Loading

ad-m commented Nov 4, 2021

harshithmullapudi commented Nov 5, 2021

marcosmarxm commented Nov 5, 2021

ad-m commented Nov 5, 2021

ad-m commented Nov 8, 2021

antixar left a comment

Choose a reason for hiding this comment

marcosmarxm commented Nov 15, 2021

marcosmarxm commented Nov 15, 2021

ad-m commented Nov 15, 2021

harshithmullapudi commented Nov 16, 2021

harshithmullapudi commented Nov 21, 2021

ad-m commented Oct 21, 2021 •

edited by harshithmullapudi

Loading

CLAassistant commented Oct 21, 2021 •

edited

Loading

ad-m commented Oct 29, 2021 •

edited

Loading