Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉Source Hubspot: Add contacts associations to Deals stream. #5693

Merged
merged 14 commits into from
Sep 20, 2021
Merged

🎉Source Hubspot: Add contacts associations to Deals stream. #5693

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
7ba2b69
Source Hubspot: add new stream deal_associations
vladimir-remar Aug 26, 2021
9caab78
update: change stream name
vladimir-remar Aug 27, 2021
1d885c7
Update: refactor the class DealToContactAssociationsStream and bump …
vladimir-remar Aug 30, 2021
e0b483a
fix: fix conflicts
vladimir-remar Aug 30, 2021
1cf0c2b
Source Hubspot: add new stream deal_associations
vladimir-remar Aug 26, 2021
ebdef51
update: change stream name
vladimir-remar Aug 27, 2021
b57fb36
Update: refactor the class DealToContactAssociationsStream and bump …
vladimir-remar Aug 30, 2021
d51f1f5
Merge branch 'vladimir/source_hubspot_deal_associations' of github.co…
vladimir-remar Sep 9, 2021
e69541b
bump versions
vladimir-remar Sep 9, 2021
59f4eb5
Merge branch 'airbytehq:master' into vladimir/source_hubspot_deal_ass…
vladimir-remar Sep 13, 2021
b7a6fb1
update: remove class DealToContactsAssociationsStream and add field …
vladimir-remar Sep 14, 2021
a44003e
update: remove associations from deals stream, change DealStream to C…
vladimir-remar Sep 16, 2021
673dc4c
remove deal_to_contact_associations from schemas
vladimir-remar Sep 17, 2021
4f09a61
remove stream deal_to_contact_associations and add to deals the conta…
vladimir-remar Sep 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...ain/resources/config/STANDARD_SOURCE_DEFINITION/36c891d9-4bd9-43ac-bad2-10e12756272c.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"sourceDefinitionId": "36c891d9-4bd9-43ac-bad2-10e12756272c",
"name": "Hubspot",
"dockerRepository": "airbyte/source-hubspot",
"dockerImageTag": "0.1.13",
"dockerImageTag": "0.1.14",
"documentationUrl": "https://docs.airbyte.io/integrations/sources/hubspot",
"icon": "hubspot.svg"
}
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/source_definitions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@
- sourceDefinitionId: 36c891d9-4bd9-43ac-bad2-10e12756272c
name: Hubspot
dockerRepository: airbyte/source-hubspot
dockerImageTag: 0.1.13
dockerImageTag: 0.1.14
documentationUrl: https://docs.airbyte.io/integrations/sources/hubspot
icon: hubspot.svg
- sourceDefinitionId: 95e8cffd-b8c4-4039-968e-d32fb4a69bde
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-hubspot/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ RUN pip install .

ENV AIRBYTE_ENTRYPOINT "/airbyte/base.sh"

LABEL io.airbyte.version=0.1.13
LABEL io.airbyte.version=0.1.14
LABEL io.airbyte.name=airbyte/source-hubspot
8 changes: 8 additions & 0 deletions airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -636,6 +636,14 @@ def list(self, fields) -> Iterable:
yield record


class DealToContactAssociationsStream(CRMObjectStream):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vladimir-remar it seems better from a UX standpoint to include the deal-to-contact association as a field on the deals stream. This way in the destination it will be normalized to a deal_contacts table. WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sherifnada thanks for the review, about the suggestion make sense for me, I will include the field as "associations", since "contacts" are one of them , for me it would be something like this.

class DealStream(CRMObjectStream):
    """Deals, API v3"""

    def __init__(self, associations: List[str] = None, **kwargs):
        super().__init__(entity="deal", associations=associations, **kwargs)
        self._stage_history = DealStageHistoryStream(**kwargs)

    def list(self, fields) -> Iterable:
        history_by_id = {}
        for record in self._stage_history.list(fields):
            if all(field in record for field in ("id", "dealstage")):
                history_by_id[record["id"]] = record["dealstage"]
        for record in super().list(fields):
            if record.get("id") and int(record["id"]) in history_by_id:
                record["dealstage"] = history_by_id[int(record["id"])]
            yield record

and I will call it in client.py
"deal_to_contact_associations": DealStream(associations=["contacts"], **common_params),

Copy link
Contributor

@sherifnada sherifnada Sep 14, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry I was a little unclear: why wouldn't we include this by default in the Deals stream instead of a separate deals_to_contacts_associations? That would be my first inclination although there may be a reason why that's not favorable.

CRMObjectStream already accepts associations as input so you actually don't need to change anything about DealStream, you only need to change the calling context to initialize it with associations=['contacts']

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @sherifnada and thanks again.
On the first question, my first approach was to replicate the functionality of the Deals Streams instead of modifying it, since associations were not included by default in the Deals stream, I did it thinking about the possible inclusion of the remaining associations. Why it was not included, I really do not know, perhaps to obtain the values of the associations we do not need the history of the deals or because the rest of the associations do not generate a good output, I do not know the result of the iteration with the rest of the associations Maybe someone on the team can answer that question.

Finally calling CRMObjectStream like this "deal_to_contact_associations": CRMObjectStream(entity="deal", associations=['contacts'], **common_params) would also get the response.

So my question is, at this point what it is the good approach to implement the associations in the current streams?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vladimir-remar that approach (including associations=['contacts']) is the one I'd recommend!

"""
Deals to Contacts associations
"""
entity = "deal"
associations = ["contacts"]


class DealPipelineStream(Stream):
"""Deal pipelines, API v1,
This endpoint requires the contacts scope the tickets scope.
Expand Down
2 changes: 2 additions & 0 deletions airbyte-integrations/connectors/source-hubspot/source_hubspot/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@
OwnerStream,
SubscriptionChangeStream,
WorkflowStream,
DealToContactAssociationsStream,
)


Expand Down Expand Up @@ -69,6 +70,7 @@ def __init__(self, start_date, credentials, **kwargs):
"subscription_changes": SubscriptionChangeStream(**common_params),
"tickets": CRMObjectStream(entity="ticket", **common_params),
"workflows": WorkflowStream(**common_params),
"deal_to_contact_associations": DealToContactAssociationsStream(**common_params),
}

super().__init__(**kwargs)
Expand Down
325 changes: 325 additions & 0 deletions ...ations/connectors/source-hubspot/source_hubspot/schemas/deal_to_contact_associations.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,325 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": ["null", "object"],
"properties": {
"id": {
"type": ["null", "string"]
},
"properties": {
"type": ["null", "object"],
"properties": {
"amount": {
"type": ["null", "string"]
},
"amount_in_home_currency": {
"type": ["null", "string"]
},
"closed_lost_reason": {
"type": ["null", "string"]
},
"closed_won_reason": {
"type": ["null", "string"]
},
"closedate": {
"type": ["null", "string"],
"format": "date-time"
},
"createdate": {
"type": ["null", "string"],
"format": "date-time"
},
"days_to_close": {
"type": ["null", "string"]
},
"dealname": {
"type": ["null", "string"]
},
"dealstage": {
"type": ["null", "string"]
},
"dealtype": {
"type": ["null", "string"]
},
"description": {
"type": ["null", "string"]
},
"engagements_last_meeting_booked": {
"type": ["null", "string"]
},
"engagements_last_meeting_booked_campaign": {
"type": ["null", "string"]
},
"engagements_last_meeting_booked_medium": {
"type": ["null", "string"]
},
"engagements_last_meeting_booked_source": {
"type": ["null", "string"]
},
"hs_acv": {
"type": ["null", "string"]
},
"hs_all_accessible_team_ids": {
"type": ["null", "string"]
},
"hs_all_assigned_business_unit_ids": {
"type": ["null", "string"]
},
"hs_all_owner_ids": {
"type": ["null", "string"]
},
"hs_all_team_ids": {
"type": ["null", "string"]
},
"hs_analytics_source": {
"type": ["null", "string"]
},
"hs_analytics_source_data_1": {
"type": ["null", "string"]
},
"hs_analytics_source_data_2": {
"type": ["null", "string"]
},
"hs_arr": {
"type": ["null", "string"]
},
"hs_closed_amount": {
"type": ["null", "string"]
},
"hs_closed_amount_in_home_currency": {
"type": ["null", "string"]
},
"hs_created_by_user_id": {
"type": ["null", "string"]
},
"hs_createdate": {
"type": ["null", "string"],
"format": "date-time"
},
"hs_date_entered_9567448": {
"type": ["null", "string"]
},
"hs_date_entered_9567449": {
"type": ["null", "string"]
},
"hs_date_entered_appointmentscheduled": {
"type": ["null", "string"]
},
"hs_date_entered_closedlost": {
"type": ["null", "string"]
},
"hs_date_entered_closedwon": {
"type": ["null", "string"]
},
"hs_date_entered_contractsent": {
"type": ["null", "string"]
},
"hs_date_entered_customclosedwonstage": {
"type": ["null", "string"]
},
"hs_date_entered_decisionmakerboughtin": {
"type": ["null", "string"]
},
"hs_date_entered_presentationscheduled": {
"type": ["null", "string"]
},
"hs_date_entered_qualifiedtobuy": {
"type": ["null", "string"]
},
"hs_date_exited_9567448": {
"type": ["null", "string"]
},
"hs_date_exited_9567449": {
"type": ["null", "string"]
},
"hs_date_exited_appointmentscheduled": {
"type": ["null", "string"]
},
"hs_date_exited_closedlost": {
"type": ["null", "string"]
},
"hs_date_exited_closedwon": {
"type": ["null", "string"]
},
"hs_date_exited_contractsent": {
"type": ["null", "string"]
},
"hs_date_exited_customclosedwonstage": {
"type": ["null", "string"]
},
"hs_date_exited_decisionmakerboughtin": {
"type": ["null", "string"]
},
"hs_date_exited_presentationscheduled": {
"type": ["null", "string"]
},
"hs_date_exited_qualifiedtobuy": {
"type": ["null", "string"]
},
"hs_deal_amount_calculation_preference": {
"type": ["null", "string"]
},
"hs_deal_stage_probability": {
"type": ["null", "string"]
},
"hs_deal_stage_probability_shadow": {
"type": ["null", "string"]
},
"hs_forecast_amount": {
"type": ["null", "string"]
},
"hs_forecast_probability": {
"type": ["null", "string"]
},
"hs_is_closed": {
"type": ["null", "string"]
},
"hs_is_closed_won": {
"type": ["null", "string"]
},
"hs_lastmodifieddate": {
"type": ["null", "string"],
"format": "date-time"
},
"hs_latest_meeting_activity": {
"type": ["null", "string"]
},
"hs_likelihood_to_close": {
"type": ["null", "string"]
},
"hs_line_item_global_term_hs_discount_percentage": {
"type": ["null", "string"]
},
"hs_line_item_global_term_hs_discount_percentage_enabled": {
"type": ["null", "string"]
},
"hs_line_item_global_term_hs_recurring_billing_period": {
"type": ["null", "string"]
},
"hs_line_item_global_term_hs_recurring_billing_period_enabled": {
"type": ["null", "string"]
},
"hs_line_item_global_term_hs_recurring_billing_start_date": {
"type": ["null", "string"]
},
"hs_line_item_global_term_hs_recurring_billing_start_date_enabled": {
"type": ["null", "string"]
},
"hs_line_item_global_term_recurringbillingfrequency": {
"type": ["null", "string"]
},
"hs_line_item_global_term_recurringbillingfrequency_enabled": {
"type": ["null", "string"]
},
"hs_manual_forecast_category": {
"type": ["null", "string"]
},
"hs_merged_object_ids": {
"type": ["null", "string"]
},
"hs_mrr": {
"type": ["null", "string"]
},
"hs_next_step": {
"type": ["null", "string"]
},
"hs_num_target_accounts": {
"type": ["null", "string"]
},
"hs_object_id": {
"type": ["null", "string"]
},
"hs_predicted_amount": {
"type": ["null", "string"]
},
"hs_predicted_amount_in_home_currency": {
"type": ["null", "string"]
},
"hs_priority": {
"type": ["null", "string"]
},
"hs_projected_amount": {
"type": ["null", "string"]
},
"hs_projected_amount_in_home_currency": {
"type": ["null", "string"]
},
"hs_sales_email_last_replied": {
"type": ["null", "string"]
},
"hs_tcv": {
"type": ["null", "string"]
},
"hs_unique_creation_key": {
"type": ["null", "string"]
},
"hs_updated_by_user_id": {
"type": ["null", "string"]
},
"hs_user_ids_of_all_notification_followers": {
"type": ["null", "string"]
},
"hs_user_ids_of_all_notification_unfollowers": {
"type": ["null", "string"]
},
"hs_user_ids_of_all_owners": {
"type": ["null", "string"],
"format": "date"
},
"hubspot_owner_assigneddate": {
"type": ["null", "string"],
"format": "date-time"
},
"hubspot_owner_id": {
"type": ["null", "string"]
},
"hubspot_team_id": {
"type": ["null", "string"]
},
"notes_last_contacted": {
"type": ["null", "string"]
},
"notes_last_updated": {
"type": ["null", "string"]
},
"notes_next_activity_date": {
"type": ["null", "string"]
},
"num_associated_contacts": {
"type": ["null", "string"]
},
"num_contacted_notes": {
"type": ["null", "string"]
},
"num_notes": {
"type": ["null", "string"]
},
"pipeline": {
"type": ["null", "string"]
}
}
},
"createdAt": {
"type": ["null", "string"],
"format": "date-time"
},
"updatedAt": {
"type": ["null", "string"],
"format": "date-time"
},
"archived": {
"type": ["null", "boolean"]
},
"companies": {
"type": ["null", "array"],
"items": {
"type": ["null", "integer"]
}
},
"contacts": {
"type": ["null", "array"],
"items": {
"type": ["null", "integer"]
}
}
}
}