🎉 source Salesloft: added several new streams #11460

yannibenoit · 2022-03-28T07:24:51Z

What

Issue: #11198
Community Slack Message: https://airbytehq.slack.com/archives/C01VDDEGL7M/p1647435938137639

New sources added for Salesloft integration ->

How

For each new stream :

Added new schema *.json for this new endpoint
Added new IncrementalSalesloftStream or SalesloftStream class
Edited streams method in SourceSalesloft class to add new class in return
Edited test_source.py to allow one more stream when unit testing

🚨 User Impact 🚨

New streams will be available in connector settings

New tables will be available in destination

**

**

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…r non-nullable fields

marcosmarxm · 2022-04-07T16:31:20Z

@yannibenoit ping me when this is ready to review.

yannibenoit · 2022-04-11T13:19:21Z

Warning: there are some API Limits errors but sync went well :)

2022-04-11 12:09:52 source > Caught retryable error 'Request URL: https://api.salesloft.com/v2/crm_activities?per_page=100&page=947, Response Code: 429, Response Text: {"error":"API rate limit exceeded"}

yannibenoit · 2022-04-11T13:19:54Z

Needed to edit next_page_token method to check if paging key was available in response metadata because that cause the sync stream error below

2022-04-11 12:38:19 source > Encountered an exception while reading stream groups
Traceback (most recent call last):
  File "/airbyte/integration_code/source_salesloft/source.py", line 33, in next_page_token
    next_page = response.json()["metadata"]["paging"].get("next_page")
KeyError: 'paging'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 119, in read
    internal_config=internal_config,
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 159, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 248, in _read_full_refresh
    for record in records:
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/streams/http/http.py", line 376, in read_records
    next_page_token = self.next_page_token(response)
  File "/airbyte/integration_code/source_salesloft/source.py", line 36, in next_page_token
    raise KeyError(f"error parsing next_page token: {e}")
KeyError: "error parsing next_page token: 'paging'"

…ments, groups

yannibenoit · 2022-04-11T13:33:22Z

@yannibenoit ping me when this is ready to review.

@marcosmarxm it's good for review now

alafanechere · 2022-04-15T07:23:37Z

/test connector=connectors/source-salesloft

🕑 connectors/source-salesloft https://github.com/airbytehq/airbyte/actions/runs/2171457991
✅ connectors/source-salesloft https://github.com/airbytehq/airbyte/actions/runs/2171457991
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  886    259    71%
Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
base_python/cdk/utils/casing.py                                                                                                     4      0   100%
base_python/__init__.py                                                                                                            13      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
base_python/cdk/utils/event_timing.py                                                                                              47      3    94%
base_python/cdk/streams/auth/core.py                                                                                                8      1    88%
base_python/cdk/streams/exceptions.py                                                                                              10      2    80%
base_python/cdk/streams/auth/token.py                                                                                               9      4    56%
base_python/logger.py                                                                                                              33     15    55%
base_python/cdk/streams/rate_limiting.py                                                                                           30     14    53%
base_python/integration.py                                                                                                         52     25    52%
base_python/cdk/streams/http.py                                                                                                    67     33    51%
base_python/cdk/streams/core.py                                                                                                    63     32    49%
base_python/client.py                                                                                                              56     33    41%
base_python/catalog_helpers.py                                                                                                     10      6    40%
base_python/source.py                                                                                                              51     34    33%
base_python/cdk/streams/auth/oauth.py                                                                                              37     26    30%
base_python/cdk/abstract_source.py                                                                                                 89     64    28%
base_python/schema_helpers.py                                                                                                      56     41    27%
base_python/entrypoint.py                                                                                                          70     56    20%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                             832    389    53%
Name                           Stmts   Miss  Cover
--------------------------------------------------
source_salesloft/__init__.py       2      0   100%
source_salesloft/source.py       161     36    78%
--------------------------------------------------
TOTAL                            163     36    78%

alafanechere · 2022-04-19T13:14:08Z

/publish connector=connectors/source-salesloft auto-bump-version=false

🕑 connectors/source-salesloft https://github.com/airbytehq/airbyte/actions/runs/2189823045
🚀 Successfully published connectors/source-salesloft
✅ connectors/source-salesloft https://github.com/airbytehq/airbyte/actions/runs/2189823045

alafanechere · 2022-04-19T16:07:59Z

Thank you @yannibenoit for this big improvement!

yannibenoit added 2 commits March 24, 2022 16:30

changed schema for users, cadences, cadence_memberships and people fo…

6827820

…r non-nullable fields

updated schemas for some endpoint + bumped connector version

5bb0a83

github-actions bot added the area/connectors Connector related issues label Mar 28, 2022

yannibenoit added 2 commits March 28, 2022 09:32

updated docs for salesloft-integration

12708b2

updated docs for salesloft-integration

b8d9e40

github-actions bot added the area/documentation Improvements or additions to documentation label Mar 28, 2022

yannibenoit marked this pull request as draft April 6, 2022 11:31

marcosmarxm added the community label Apr 6, 2022

added new streams successes, crm users & activities, templates attach…

9d0550e

…ments, groups

yannibenoit marked this pull request as ready for review April 11, 2022 13:32

alafanechere self-assigned this Apr 12, 2022

alafanechere added 2 commits April 15, 2022 09:22

format

2919af5

Merge branch 'master' into yanni-source-salesloft-issue-11198-v2

6f80112

update source definitions / source specs

4245bb7

alafanechere merged commit 1fe7222 into airbytehq:master Apr 19, 2022

octavia-squidington-iii mentioned this pull request Apr 19, 2022

Bump Airbyte version from 0.36.0-alpha to 0.36.1-alpha #12148

Merged

suhomud pushed a commit that referenced this pull request May 23, 2022

🎉 source Salesloft: added several new streams (#11460)

6afea1a

topefolorunso mentioned this pull request Oct 3, 2023

Source Salesloft: add new streams #11198

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 source Salesloft: added several new streams #11460

🎉 source Salesloft: added several new streams #11460

yannibenoit commented Mar 28, 2022 •

edited

Loading

marcosmarxm commented Apr 7, 2022

yannibenoit commented Apr 11, 2022

yannibenoit commented Apr 11, 2022

yannibenoit commented Apr 11, 2022

alafanechere commented Apr 15, 2022 •

edited by github-actions bot

Loading

alafanechere commented Apr 19, 2022 •

edited by github-actions bot

Loading

alafanechere commented Apr 19, 2022

🎉 source Salesloft: added several new streams #11460

🎉 source Salesloft: added several new streams #11460

Conversation

yannibenoit commented Mar 28, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

marcosmarxm commented Apr 7, 2022

yannibenoit commented Apr 11, 2022

yannibenoit commented Apr 11, 2022

yannibenoit commented Apr 11, 2022

alafanechere commented Apr 15, 2022 • edited by github-actions bot Loading

alafanechere commented Apr 19, 2022 • edited by github-actions bot Loading

alafanechere commented Apr 19, 2022

yannibenoit commented Mar 28, 2022 •

edited

Loading

alafanechere commented Apr 15, 2022 •

edited by github-actions bot

Loading

alafanechere commented Apr 19, 2022 •

edited by github-actions bot

Loading