Airbyte Native Hubspot connector

[sherifnada]

Tell us about the problem you're trying to solve

We've recently received a few requests to support replicating more entities from our Hubspot connector. The connector is currently Singer-based and replicates the following entities:

• "subscription_changes"
• "email_events"
• "forms"
• "workflows"
• "owners"
• "campaigns"
• "contact_lists"
• "contacts"
• "companies"
• "deals"
• "deal_pipelines"
• "engagements"
• "contacts_by_company"

We've received requests to support the following entities:

• tickets
• products
• line_items
• quotes

Plus some extra fields in various entities (Contacts, Deals, Companies):

property_country.updatedByUserId
property_country.versions
property_createdate.updatedByUserId
property_createdate.versions
property_d_u_n_s_number_c.updatedByUserId
property_d_u_n_s_number_c.versions
property_description.updatedByUserId
property_description.versions
property_dnb_business_name.updatedByUserId

Describe the solution you’d like

We'd like to have a hubspot source which:

• replicates the current set of entities
• replicates the new requested entities and properties
• replicates data in an incremental fashion

Preferably this connector is written from scratch for the Airbyte protocol rather than forked from the singer hubspot tap.

Describe the alternative you’ve considered or used

Fork tap-hubspot, which is what our current connector is based on, to add support for these entities. The upside is that it's probably less work. The downside is that it'll carry Singer baggage in the code.

[sherifnada]

@eugene-kulak has volunteered to scope and provide breakdown (and implement). Initial breakdown:

• Populate the schema/ folder Source Hubspot: Create native connector with schema folder populated #2150
• Make sure all streams in the API have data Source Hubspot: ensure that every stream has a data to read #2160
• implement all current streams as full refresh. (including nested) Source Hubspot: native client should support all streams in full sync mode #2195
• load properties dynamically on entities schema Source Hubspot: schema should load properties dynamically #2219
• ensure we use start_date as initial offset to read Source Hubspot: all streams should use start_date #2220
• implement incremental on streams that support it at the moment. Source Hubspot: add incremental support #2222
• Ensure best practices are applied (rate limiting, tests all streams etc.) Source Hubspot: ensure best practices are applied (rate limiting, tests all streams etc.) #2221

[sherifnada]

@eugene-kulak will create tickets and populate in the connector roadmap Project

[sowla]

Hi, I see contacts_by_company listed in this and similar tickets (#2150, #2160), but I don't see it anywhere in the code or the Hubspot source documentation. I was wondering if there are plans to add this stream too?