🎉 New Source: Retently #6966

subhash-kloudio · 2021-10-12T08:56:17Z

What

Source connector for Retently
Syncs customers, companies and reports from the SaaS application

How

HTTP API based source connector

Pre-merge Checklist

CLAassistant · 2021-10-12T08:56:25Z

All committers have signed the CLA.

subhash-kloudio · 2021-10-12T08:58:15Z

Fixes airbytehq/connector-contest#66

marcosmarxm · 2021-10-14T03:02:45Z

Thanks for your contribution! We're going to review this tomorrow max by Friday. Can you share some screenshots showing the integratinos tests are passing?

subhash-kloudio · 2021-10-14T10:33:21Z

Sure!

Unit tests:

(.venv) (base) subhash@Subhashs-MacBook-Pro source-retently % python -m pytest unit_tests
Test session starts (platform: darwin, Python 3.8.5, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/subhash/kloudio/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ... 
 airbyte-integrations/connectors/source-retently/unit_tests/test_source.py::test_check_connection ✓    8% ▉         
 airbyte-integrations/connectors/source-retently/unit_tests/test_source.py::test_streams ✓            17% █▋        
 airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py::test_request_params ✓    25% ██▌       
 airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py::test_next_page_token ✓   33% ███▍      
 airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py::test_parse_response ✓    42% ████▎     
 airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py::test_request_headers ✓   50% █████     
 airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py::test_http_method ✓       58% █████▉    
 airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py::test_should_retry[HTTPStatus.OK-False] ✓67% ██████▋   
 airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py::test_should_retry[HTTPStatus.BAD_REQUEST-False] ✓75% ███████▌  
 airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py::test_should_retry[HTTPStatus.TOO_MANY_REQUESTS-True] ✓83% ████████▍ 
 airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py::test_should_retry[HTTPStatus.INTERNAL_SERVER_ERROR-True] ✓92% █████████▎
 airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py::test_backoff_time ✓     100% ██████████
================================================= warnings summary =================================================
airbyte-integrations/connectors/source-retently/unit_tests/test_source.py::test_streams
  /Users/subhash/kloudio/airbyte/airbyte-integrations/connectors/source-retently/source_retently/source.py:26: DeprecationWarning: Call to deprecated class TokenAuthenticator. (Use airbyte_cdk.sources.streams.http.requests_native_auth.TokenAuthenticator instead) -- Deprecated since version 0.1.20.
    auth = TokenAuthenticator(token, auth_method=auth_method)

airbyte-integrations/connectors/source-retently/unit_tests/test_source.py: 4 warnings
airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py: 10 warnings
  /Users/subhash/kloudio/elt/airbyte/connectors/source-retently/.venv/lib/python3.8/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

airbyte-integrations/connectors/source-retently/unit_tests/test_source.py: 3 warnings
airbyte-integrations/connectors/source-retently/unit_tests/test_streams.py: 10 warnings
  /Users/subhash/kloudio/elt/airbyte/connectors/source-retently/.venv/lib/python3.8/site-packages/airbyte_cdk/sources/streams/http/http.py:37: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator = NoAuth()

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.42s):
      12 passed

Integration tests:

(.venv) (base) subhash@Subhashs-MacBook-Pro source-retently % python -m pytest integration_tests -p integration_tests.acceptance
Test session starts (platform: darwin, Python 3.8.5, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/subhash/kloudio/airbyte, configfile: pytest.ini
plugins: sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ... 
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓9% ▉         
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓18% █▊        
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓27% ██▊       
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓36% ███▋      
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓45% ████▋     
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓55% █████▌    
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓64% ██████▍   
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓73% ███████▍  
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓82% ████████▎ 
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓91% █████████▏
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓100% ██████████
============================================= short test summary info ==============================================
SKIPPED [1] ../../bases/source-acceptance-test/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../../bases/source-acceptance-test/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config

Results (10.18s):
      11 passed

marcosmarxm · 2021-10-19T17:50:46Z

Hi @subhash-kloudio we add your contribution to our sprint. Sorry to not review it yet, there are so many connectors from Hacktober.

marcosmarxm · 2021-10-22T04:41:52Z

airbyte-integrations/connectors/source-retently/source_retently/source.py

+        auth = TokenAuthenticator(token, auth_method=auth_method)
+        return [ Customers(auth), Companies(auth), Reports(auth) ]
+
+class Stream(HttpStream):


Suggested change

class Stream(HttpStream):

class RetentlyStream(HttpStream):

Is a common pattern, can you change it?

marcosmarxm · 2021-10-22T04:43:22Z

airbyte-integrations/connectors/source-retently/source_retently/source.py

+        api_key = config["api_key"]
+        if not api_key:
+            return False, f"API key is missing"
+        else:
+            return True, None


It's better call a try/except block.
See

airbyte/airbyte-integrations/connectors/source-surveymonkey/source_surveymonkey/source.py

Lines 18 to 27 in c83cafc

def check_connection(self, logger: AirbyteLogger, config: Mapping[str, Any]) -> Tuple[bool, Any]:

try:

authenticator = TokenAuthenticator(token=config["access_token"])

start_date = pendulum.parse(config["start_date"])

stream = Surveys(authenticator=authenticator, start_date=start_date)

records = stream.read_records(sync_mode=SyncMode.full_refresh)

next(records)

return True, None

except Exception as e:

return False, repr(e)

and api_key is a required field dont need to check if is null.

marcosmarxm

sorry for the delay in review! some small changes. I'll try to run tests locally to validate the code. Thanks @subhash-kloudio

subhash-kloudio · 2021-10-22T09:36:01Z

sorry for the delay in review! some small changes. I'll try to run tests locally to validate the code. Thanks @subhash-kloudio

No problem. Comments have been incorporated, thanks

marcosmarxm · 2021-10-25T21:31:17Z

airbyte-integrations/connectors/source-retently/acceptance-test-config.yml

@@ -0,0 +1,15 @@
+connector_image: airbyte/source-retently


Suggested change

connector_image: airbyte/source-retently

connector_image: airbyte/source-retently:dev

marcosmarxm · 2021-10-25T21:31:44Z

airbyte-integrations/connectors/source-retently/sample_files/invalid_config.json

+    "username": "foo",
+    "password": "bar"


the config only use api_key right?

marcosmarxm · 2021-10-25T21:32:50Z

airbyte-integrations/connectors/source-retently/Dockerfile

+ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
+
+LABEL io.airbyte.version=0.1.0
+LABEL io.airbyte.name=airbyte/source_retently


Suggested change

LABEL io.airbyte.name=airbyte/source_retently

LABEL io.airbyte.name=airbyte/source-retently

marcosmarxm · 2021-10-25T21:33:23Z

airbyte-integrations/connectors/source-retently/README.md

+## [Quickrun](https://docs.airbyte.io/connector-development/tutorials/cdk-speedrun)
+
+1. `python -m venv .venv`
+2. `source .venv/bin/activate`
+3. `pip install -r requirements.txt` (Alter path in requirements.txt)
+4. `python main.py check --config sample_files/config.json` should report SUCCESS
+5. `python main.py check --config sample_files/invalid_config.json` should report FAILED
+6. `python main.py discover --config sample_files/config.json` presents schema
+7. `python main.py read --config sample_files/config.json --catalog sample_files/configured_catalog.json` runs source
+8. `docker build . -t <repo>/source-retently:<n>` builds docker image
+10. `docker push <repo>/source-retently:<n>` .. and push 
+11. Add connector from http://localhost:8000/settings/source
+12. `pip install .[tests]`
+13. `python -m pytest unit_tests`
+14. `python -m pytest integration_tests`
+15. `python -m pytest integration_tests -p integration_tests.acceptance`


Please keep the README as same from other connectors.

marcosmarxm · 2021-10-25T21:45:41Z

airbyte-integrations/connectors/source-retently/integration_tests/sample_config.json

@@ -0,0 +1,3 @@
+{
+  "api_key": "b89fe9a7-d33c-43ab-a215-373c27593a69"


you dont need to let your api key here, change to only an example.

marcosmarxm · 2021-10-25T21:47:21Z

airbyte-integrations/connectors/source-retently/source_retently/schemas/companies.json

+    "id": { "type": "string" }, 
+    "createdDate": { "type": "string" }, 
+    "tags": { "type": "array", "items": { "type": "string" } }, 
+    "cxMetrics": { "type": "object" },


cxMetrics is not possible to create the object or is dynamic?

marcosmarxm · 2021-10-25T21:48:24Z

airbyte-integrations/connectors/source-retently/source_retently/schemas/reports.json

+  "type": "object",
+  "properties": {
+    "last": { "type": "object" }, 
+    "trend": { "type": "array" }, 


is not possible to add the nested objecT?
see API example

day" : "2020-01-12", "promoters" : 292, "passives" : 194, "detractors" : 25, "total" : 511, "score" : 52

marcosmarxm

Some comments, integration tests are working. After solving the comments we can merge this. Please make sure to resolve all problems when running ./gradlew format there is some libs imported and not used.

marcosmarxm · 2021-10-25T21:49:05Z

airbyte-integrations/connectors/source-retently/source_retently/source.py

+            token = ""
+            auth = TokenAuthenticator(token, auth_method=auth_method)


Suggested change

token = ""

auth = TokenAuthenticator(token, auth_method=auth_method)

auth = TokenAuthenticator(token="", auth_method=auth_method)

marcosmarxm · 2021-10-25T21:49:31Z

airbyte-integrations/connectors/source-retently/source_retently/source.py

+    @property
+    @abstractmethod
+    def json_path(self):
+        pass


Why this function?

It is overriden in the specific stream classes to provide the path within the response that holds the stream data. Eg. subscribers for Customers

marcosmarxm · 2021-10-25T21:52:45Z

airbyte-integrations/connectors/source-retently/unit_tests/test_source.py

+def test_check_connection(mocker):
+    source = SourceRetently()
+    logger_mock, config_mock = MagicMock(), MagicMock()
+    assert source.check_connection(logger_mock, config_mock) == (True, None)


This is not working.

> assert source.check_connection(logger_mock, config_mock) == (True, None) E assert (False,\n "HTTPError('401 Client Error: Unauthorized for url: "\n "https://app.retently.com/api/v2/companies')") == (True, None) E At index 0 diff: False != True E Full diff: E ( E - True, E - None, E + False, E + "HTTPError('401 Client Error: Unauthorized for url: " E + "https://app.retently.com/api/v2/companies')", E )

subhash-kloudio · 2021-10-26T06:31:59Z

Some comments, integration tests are working. After solving the comments we can merge this. Please make sure to resolve all problems when running ./gradlew format there is some libs imported and not used.

Thanks.

All comments have been addressed
Tests are working
./gradlew format - This comment is pending. I am not sure how to go about this. Please see error below. I am hoping you can guide me to get this to work

% gradle wrapper format --warning-mode all
Building all of Airbyte.
/Users/subhash/kloudio/airbyte/airbyte-integrations/connectors

> Configure project :buildSrc
The RepositoryHandler.jcenter() method has been deprecated. This is scheduled to be removed in Gradle 8.0. JFrog announced JCenter's sunset in February 2021. Use mavenCentral() instead. Consult the upgrading guide for further information: https://docs.gradle.org/7.2/userguide/upgrading_version_6.html#jcenter_deprecation
        at build_ag0ycgcg86xuhdfhg2sqeexnc$_run_closure1.doCall(/Users/subhash/kloudio/airbyte/buildSrc/build.gradle:6)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)

> Task :buildSrc:jar FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':buildSrc:jar'.
> Entry AirbytePythonPlugin.class is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/7.2/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 612ms
8 actionable tasks: 1 executed, 7 up-to-date

marcosmarxm · 2021-10-26T16:49:54Z

Some comments, integration tests are working. After solving the comments we can merge this. Please make sure to resolve all problems when running ./gradlew format there is some libs imported and not used.

Thanks.

All comments have been addressed
Tests are working
./gradlew format - This comment is pending. I am not sure how to go about this. Please see error below. I am hoping you can guide me to get this to work

% gradle wrapper format --warning-mode all
Building all of Airbyte.
/Users/subhash/kloudio/airbyte/airbyte-integrations/connectors

> Configure project :buildSrc
The RepositoryHandler.jcenter() method has been deprecated. This is scheduled to be removed in Gradle 8.0. JFrog announced JCenter's sunset in February 2021. Use mavenCentral() instead. Consult the upgrading guide for further information: https://docs.gradle.org/7.2/userguide/upgrading_version_6.html#jcenter_deprecation
        at build_ag0ycgcg86xuhdfhg2sqeexnc$_run_closure1.doCall(/Users/subhash/kloudio/airbyte/buildSrc/build.gradle:6)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)

> Task :buildSrc:jar FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':buildSrc:jar'.
> Entry AirbytePythonPlugin.class is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/7.2/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 612ms
8 actionable tasks: 1 executed, 7 up-to-date

You need to run ./gradlew format in the root folder

subhash-kloudio · 2021-10-28T03:52:10Z

You need to run ./gradlew format in the root folder

Done! And reported problems fixed!

Please let me know if there's anything else @marcosmarxm

marcosmarxm

LGTM! Great work @subhash-kloudio

* retently * PR comments * PR comments * mask api keys * README * keep gradle happy

retently

9e16af7

github-actions bot added the area/connectors Connector related issues label Oct 12, 2021

subhash-kloudio mentioned this pull request Oct 12, 2021

Source: Retently airbytehq/connector-contest#66

Closed

avaidyanatha added the community_pr label Oct 12, 2021

avaidyanatha changed the title ~~Source Retently: new connector~~ 🎉 New Source: Retently Oct 12, 2021

marcosmarxm added community hacktober and removed community_pr labels Oct 13, 2021

marcosmarxm mentioned this pull request Oct 14, 2021

CI Sandbox for Retently #7030

Closed

sherifnada added this to the Connectors 2021-10-29 milestone Oct 14, 2021

marcosmarxm added the zzm label Oct 19, 2021

Zirochkaa self-assigned this Oct 21, 2021

marcosmarxm reviewed Oct 22, 2021

View reviewed changes

marcosmarxm requested changes Oct 22, 2021

View reviewed changes

PR comments

7e488ca

marcosmarxm reviewed Oct 25, 2021

View reviewed changes

marcosmarxm requested changes Oct 25, 2021

View reviewed changes

subhash-kloudio added 3 commits October 26, 2021 11:35

PR comments

9c918f5

mask api keys

0eb9467

README

94f8909

keep gradle happy

d298c4d

sherifnada unassigned Zirochkaa Oct 29, 2021

sherifnada removed this from the Connectors 2021-10-29 milestone Oct 29, 2021

marcosmarxm approved these changes Nov 2, 2021

View reviewed changes

marcosmarxm merged commit 63d6301 into airbytehq:master Nov 2, 2021

jrhizor mentioned this pull request Nov 3, 2021

Bump Airbyte version from 0.30.25-alpha to 0.30.26-alpha #7599

Merged

lmossman pushed a commit that referenced this pull request Nov 3, 2021

🎉 New Source: Retently (#6966)

3f3adea

* retently * PR comments * PR comments * mask api keys * README * keep gradle happy

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

🎉 New Source: Retently (airbytehq#6966)

46e5fea

* retently * PR comments * PR comments * mask api keys * README * keep gradle happy

marcosmarxm removed the hacktober label Oct 20, 2022

octavia-squidington-iv added bounty connectors/source/retently labels Oct 20, 2022

sajarin added internal and removed bounty labels Oct 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Retently #6966

🎉 New Source: Retently #6966

subhash-kloudio commented Oct 12, 2021 •

edited by Zirochkaa

Loading

CLAassistant commented Oct 12, 2021 •

edited

Loading

subhash-kloudio commented Oct 12, 2021

marcosmarxm commented Oct 14, 2021

subhash-kloudio commented Oct 14, 2021

marcosmarxm commented Oct 19, 2021

marcosmarxm Oct 22, 2021

marcosmarxm Oct 22, 2021

marcosmarxm Oct 22, 2021

marcosmarxm left a comment

subhash-kloudio commented Oct 22, 2021

marcosmarxm Oct 25, 2021

marcosmarxm Oct 25, 2021

marcosmarxm Oct 25, 2021

marcosmarxm Oct 25, 2021

marcosmarxm Oct 25, 2021

marcosmarxm Oct 25, 2021

marcosmarxm Oct 25, 2021

marcosmarxm left a comment

marcosmarxm Oct 25, 2021

marcosmarxm Oct 25, 2021

subhash-kloudio Oct 26, 2021

marcosmarxm Oct 25, 2021

subhash-kloudio Oct 26, 2021

subhash-kloudio Oct 26, 2021

subhash-kloudio commented Oct 26, 2021

marcosmarxm commented Oct 26, 2021

subhash-kloudio commented Oct 28, 2021

marcosmarxm left a comment

	def check_connection(self, logger: AirbyteLogger, config: Mapping[str, Any]) -> Tuple[bool, Any]:
	try:
	authenticator = TokenAuthenticator(token=config["access_token"])
	start_date = pendulum.parse(config["start_date"])
	stream = Surveys(authenticator=authenticator, start_date=start_date)
	records = stream.read_records(sync_mode=SyncMode.full_refresh)
	next(records)
	return True, None
	except Exception as e:
	return False, repr(e)

	connector_image: airbyte/source-retently
	connector_image: airbyte/source-retently:dev

	LABEL io.airbyte.name=airbyte/source_retently
	LABEL io.airbyte.name=airbyte/source-retently

		@@ -0,0 +1,3 @@
		{
		"api_key": "b89fe9a7-d33c-43ab-a215-373c27593a69"

		token = ""
		auth = TokenAuthenticator(token, auth_method=auth_method)

	token = ""
	auth = TokenAuthenticator(token, auth_method=auth_method)
	auth = TokenAuthenticator(token="", auth_method=auth_method)

🎉 New Source: Retently #6966

🎉 New Source: Retently #6966

Conversation

subhash-kloudio commented Oct 12, 2021 • edited by Zirochkaa Loading

What

How

Recommended reading order

Pre-merge Checklist

CLAassistant commented Oct 12, 2021 • edited Loading

subhash-kloudio commented Oct 12, 2021

marcosmarxm commented Oct 14, 2021

subhash-kloudio commented Oct 14, 2021

marcosmarxm commented Oct 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcosmarxm left a comment

Choose a reason for hiding this comment

subhash-kloudio commented Oct 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcosmarxm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

subhash-kloudio commented Oct 26, 2021

marcosmarxm commented Oct 26, 2021

subhash-kloudio commented Oct 28, 2021

marcosmarxm left a comment

Choose a reason for hiding this comment

subhash-kloudio commented Oct 12, 2021 •

edited by Zirochkaa

Loading

CLAassistant commented Oct 12, 2021 •

edited

Loading