🎉 New Source: Pipedrive

[Tana8M]

Tell us about the new integration you’d like to have

• Which source and which destination?
  In the Airbyte not have pipedrive Source.
  API or DB?
  API

Describe the context around this new integration

• Which team in your company wants this integration, what for? This helps us understand the use case.
  In the company, Sale team use Pipedrive for check activity working and not have history record for implement next step.
• How often do you want to run syncs?
  1 time per day
• If this is an API source connector, which entities/endpoints do you need supported?
  pipedrive have API.

Describe the alternative you are considering or using

What are you considering doing if you don’t have this integration through Airbyte?
I want to collect all activity for checking and create report to stakeholder

┆Issue is synchronized with this Asana task by Unito

[marcosmarxm]

thanks @Tana8M I added the request to our roadmap, we're releasing the CDK a tool to simplify the building of new connectors you can read more here or enter the channel #airbyte_cdk.

[marcosmarxm]

Info from #3322

Tell us about the new integration you’d like to have

• Which source and which destination? Pipedrive source for datawarehouse destination.
• Do you need a specific version of the underlying data source e.g: you specifically need support for an older version of the API or DB? No

Describe the context around this new integration

• Which team in your company wants this integration, what for? This helps us understand the use case.
• How often do you want to run syncs? As a Data Analyst I want to sync our CRM data with our warehouse so that I can cross this source with others like app db, other 3rd parties applications...Today Pipedrive is not supported by other ETL tools unlike Hubspot or Salesforce.
• If this is an API source connector, which entities/endpoints do you need supported? All those which allow to retrieve the data about deals, leads, contact, company, task, emails, notes and so on.

Describe the alternative you are considering or using

What are you considering doing if you don’t have this integration through Airbyte? Today Pipedrive is only supported by singer (possible with Stitch, incoming feature for Fivetran) but many disadvantages here.

[marcosmarxm]

@mcouto-sossego you can follow implementation by subscribing to this thread. You pointed a singer tap for this connector, right? but for new connectors, we are using our CDK to improve maintenance and reliability.

[htrueman]

Integration Vetting

Webhook-based?
Does support both Webhooks and API calls

Available authentication modes
Oauth 2.0, API token auth

Creating an account
Account created, credentials in last pass.

How to populate the account with data?
Can it be done entirely through the API? Almost entirely, except search Results, permission Sets, mailbox, item Search and global Messages, activityFields, currencies , leadSources, noteFields , recents, userConnections, userSettings .
Will it require some work through the UI? If so, what is required? Only initial API key retrieval
Will it require performing some paid action like running an Ad campaign? If it can be done with under $30, do it and make sure to send the bill to invoices@airbyte.io. If it costs more than $30 please tag @ johnlafleur No, $15 sub is enough in our case.
Do you need any actions from Airbyte to populate the account with data? If so, please tag @ sherifnada for support No

Available streams for sync

Activities
ActivityFields
ActivityTypes
CallLogs
Currencies
Deals
DealFields
Files
Filters
GlobalMessages
Goals
ItemSearch
Leads
LeadLabels
LeadSources
Mailbox
Notes
NoteFields
Organizations
OrganizationFields
OrganizationRelationships
PermissionSets
Persons
PersonFields
Pipelines
Products
ProductFields
Recents
Roles
SearchResults
Stages
Subscriptions
Teams
Users
UserConnections
UserSettings

Integration supports incremental sync?
We can use filter functionality in our requests

List APIs such as the APIs to retrieve all activities return bulk data. In the query string of an API request, you can use a combination of the following parameters to filter and limit the response data

ATTRIBUTE NAME	DATA TYPE	DESCRIPTION
start	integer	For pagination, the position that represents the first result for the page
limit	integer	For pagination, the limit of entries to be returned. If not provided, 100 items will be returned.
start_date	string	Use the Activity due date where you wish to begin fetching Activities from. Insert due date in YYYY-MM-DD format.

Other information/blockers
The permissible number of API calls in a minute is based on the Essential plan opted for an organization. The rate limit for each plan is fixed:

Rate limiting when authenticating with api_token

PLAN API	RATE LIMIT
Essential	20 requests per 2 seconds per api_token
Advanced	40 requests per 2 seconds per api_token
Professional	80 requests per 2 seconds per api_token
Enterprise	120 requests per 2 seconds per api_token

Rate limiting when authenticating with access_token

PLAN API	RATE LIMIT
Essential	80 requests per 2 seconds per access_token
Advanced	60 requests per 2 seconds per access_token
Professional	320 requests per 2 seconds per access_token
Enterprise	480 requests per 2 seconds per access_token

The rate limit is applicable to an account, irrespective of the IP address from which calls originate.

In addition to rate limiting, public API has a daily API fair usage limit for all POST/PUT endpoints. Fair usage is considered to be a maximum of 10 000 POST/PUT requests daily per user per 24 hours. The daily limit will be reset at midnight in UTC.

See for more rate limit info https://pipedrive.readme.io/docs/core-api-concepts-rate-limiting

[htrueman]

Product Requirements & features

1. What is the “ideal end state” for the features offered by this connector? At a minimum mention:
   a. Output schema: How will tables/entities/etc.. map to Airbyte streams and fields?
   MVP streams schemas (autogenerated at https://www.jsonschema.net, so may be updated): https://drive.google.com/drive/folders/14SC3phypV-2tR7iuDJ-0yRyWKN53lCDw?usp=sharing
   b. Authentication
   The API is stateless – all requests are validated against an API token. The API token can be obtained manually from the Pipedrive app.
   c. Include more topics as you see fit depending on the connector (etc…)
   This connector's API seems to be very pure and well documented, so couldn't find any possible issues for now
2. What is the release roadmap for this connector?
   a. Which features are absolutely necessary for the MVP release?
   I think of Deals, Leads, Activities, Persons, Pipelines and Stages, Users (required to fully catch records from other streams) streams
   b. Which features can be released as follow ups?
   These may be ActivityFields, ActivityTypes, DealFields, Files, GlobalMessages, LeadLabels, LeadSources, Organizations, OrganizationFields, OrganizationRelationships, Teams, Mailbox streams
   c. What steps will we take as we build the connector to ensure backwards compatibility from MVP to the ideal end state?
   We need to implement at least a single stream of each type, so they may be referenced in future
3. Check if and how Fivetran or Stitch Data support this connector
   Fivetran supports ACTIVITY, ACTIVITY_TYPE, DEAL, NOTE, ORGANIZATION, PERSON, PIPELINE, STAGE, and USER tables sync
   Stitch Data supports sync of activities activity_types, currencies, deals, email_messages, email_threads, files, filters, goals, notes, organizations, persons, pipelines, products, stages, users streams
   a. Do they do something better than our proposed implementation?
   I don’t think any of those integrations are somehow better than the proposed one. The list of available streams is not obviously good, and our opportunities for connector expansion are great
   b. Is there something we can/should do better?
   Yes, we may select the better list of initial streams, also we allow to expand the connector easily.

Implementation Scoping

• Describe the pagination strategy used by the API
  It's limit/offset pagination available for all the streams with large datasets. This pagination method is consistent throughout the whole Pipedrive API
• Describe the authentication strategy used by the API. If Oauth, go into detail about how the token can be obtained and if there are any special gotchas.
  Oauth. API token is easy to obtain, see https://pipedrive.readme.io/docs/how-to-find-the-api-token
• Describe how we will populate the sandbox account with data
  We'll do it manually, but if you'd like it's CRUD API methods available for almost all the streams
• How will rate limiting be handled?
  Pipedrive source connector now gracefully handles rate limiting. Limits described in source vetting. So our default backoff approach should perfectly handle it.
• Type of the API? (REST/SOAP/GraphQL/other)
  REST

Integration details
https://docs.google.com/spreadsheets/d/1s-MAwI5d3eBlBOD8II_sZM7pw5FmZtAJsx1KJjVRFNU/edit?usp=sharing