Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Source PostHog: Use account information for checking the connection #4692

Merged
merged 15 commits into from
Jul 15, 2021
Merged

Source PostHog: Use account information for checking the connection #4692

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
f37b011
this should fix the check if no records in annotations stream
vovavovavovavova Jul 12, 2021
0610c7b
update schemas for new SAT requirements && apply user hint upgrade on…
vovavovavovavova Jul 14, 2021
2b477a7
save schema upd
vovavovavovavova Jul 14, 2021
87a5bcd
Merge remote-tracking branch 'origin/master' into valdemar/fix_postho…
vovavovavovavova Jul 14, 2021
b591822
upd insights schema
vovavovavovavova Jul 14, 2021
d0bd7a3
upd insights schema2
vovavovavovavova Jul 14, 2021
c6fca0d
upd insights schema3
vovavovavovavova Jul 14, 2021
74672b2
upd insights schema4
vovavovavovavova Jul 14, 2021
eb7ad89
upd insights schema5 (null is joking)
vovavovavovavova Jul 14, 2021
fdf198f
upd insights schema6 (null is joking)
vovavovavovavova Jul 14, 2021
94e36b0
upd insights schema7
vovavovavovavova Jul 14, 2021
61415db
upd insights schema8
vovavovavovavova Jul 14, 2021
1061ddf
upd insights schema8
vovavovavovavova Jul 14, 2021
3de0e54
Merge remote-tracking branch 'origin/master' into valdemar/fix_postho…
vovavovavovavova Jul 15, 2021
5b7b9b4
bump version && docs
vovavovavovavova Jul 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions airbyte-integrations/connectors/source-posthog/integration_tests/configured_catalog.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,6 @@
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": null,
"default_cursor_field": null,
"source_defined_primary_key": [["id"]],
"namespace": null
},
"sync_mode": "full_refresh",
Expand All @@ -97,7 +96,6 @@
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": null,
"default_cursor_field": null,
"source_defined_primary_key": [["id"]],
"namespace": null
},
"sync_mode": "full_refresh",
Expand Down Expand Up @@ -127,7 +125,6 @@
"supported_sync_modes": ["full_refresh"],
"source_defined_cursor": null,
"default_cursor_field": null,
"source_defined_primary_key": [["id"]],
"namespace": null
},
"sync_mode": "full_refresh",
Expand Down
8 changes: 4 additions & 4 deletions airbyte-integrations/connectors/source-posthog/source_posthog/schemas/annotations.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,17 @@
"type": "integer"
},
"content": {
"type": "string"
"type": ["string", "null"]
},
"date_marker": {
"type": "string",
"type": ["string", "null"],
"format": "date-time"
},
"creation_type": {
"type": "string"
"type": ["string", "null"]
},
"dashboard_item": {
"type": "string"
"type": ["string", "null"]
},
"created_by": {
"type": "object",
Expand Down
16 changes: 8 additions & 8 deletions airbyte-integrations/connectors/source-posthog/source_posthog/schemas/cohorts.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,18 @@
"type": "string"
},
"groups": {
"type": "array",
"type": ["array", "object"],
"items": {
"type": "object",
"properties": {
"days": {
"type": "string"
"type": ["string", "null"]
},
"action_id": {
"type": "string"
"type": ["string", "null"]
},
"properties": {
"type": "array",
"type": ["array"],
"items": {
"type": "object"
}
Expand All @@ -46,19 +46,19 @@
"type": "string"
},
"first_name": {
"type": "string"
"type": ["string", "null"]
},
"email": {
"type": "string"
"type": ["string", "null"]
}
}
},
"created_at": {
"type": "string",
"type": ["string", "null"],
"format": "date-time"
},
"last_calculation": {
"type": "string"
"type": ["string", "null"]
},
"errors_calculating": {
"type": "integer"
Expand Down
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-posthog/source_posthog/schemas/events.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"type": "string"
},
"event": {
"type": "string"
"type": ["string", "object"]
},
"timestamp": {
"type": "string",
Expand All @@ -34,7 +34,7 @@
"elements": {
"type": "array",
"items": {
"type": "string"
"type": ["string", "object"]
}
},
"elements_chain": {
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-posthog/source_posthog/schemas/feature_flags.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"type": "string"
},
"rollout_percentage": {
"type": "integer"
"type": ["integer", "null"]
},
"filters": {
"type": "object",
Expand Down
56 changes: 28 additions & 28 deletions airbyte-integrations/connectors/source-posthog/source_posthog/schemas/insights.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,118 +5,118 @@
"type": "integer"
},
"name": {
"type": "string"
"type": ["string", "null"]
},
"filters": {
"type": "object",
"type": ["object", "null"],
"properties": {
"events": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {
"type": "string"
"type": ["string", "null"]
},
"math": {
"type": "string"
"type": ["string", "null"]
},
"name": {
"type": "string"
"type": ["string", "null"]
},
"type": {
"type": "string"
"type": ["string", "null"]
},
"order": {
"type": "integer"
"type": ["integer", "null"]
},
"properties": {
"type": "array",
"items": {
"type": "string"
"type": ["string", "null", "object"]
}
},
"math_property": {
"type": "string"
"type": ["string", "null"]
}
}
}
},
"display": {
"type": "string"
"type": ["string", "null"]
},
"filters": {
"type": "array",
"items": {
"type": "string"
"type": ["string", "null"]
}
},
"insight": {
"type": "string"
"type": ["string", "null"]
},
"session": {
"type": "string"
"type": ["string", "null"]
},
"interval": {
"type": "string"
"type": ["string", "integer"]
},
"pagination": {
"type": "object"
"type": ["object", "null"]
}
}
},
"filters_hash": {
"type": "string"
"type": ["string", "null"]
},
"order": {
"type": "string"
"type": ["string", "null", "object"]
},
"deleted": {
"type": "boolean"
},
"dashboard": {
"type": "string"
"type": ["string", "integer", "null"]
},
"layouts": {
"type": "object"
"type": ["object", "null"]
},
"color": {
"type": "string"
"type": ["string", "null"]
},
"last_refresh": {
"type": "string",
"type": ["string", "null"],
"format": "date-time"
},
"refreshing": {
"type": "boolean"
},
"result": {
"type": "string"
"type": ["string", "null", "object"]
},
"created_at": {
"type": "string",
"type": ["string", "null"],
"format": "date-time"
},
"saved": {
"type": "boolean"
},
"created_by": {
"type": "object",
"type": ["object", "null"],
"properties": {
"id": {
"type": "integer"
},
"uuid": {
"type": "string"
"type": ["string", "null"]
},
"distinct_id": {
"type": "string"
"type": ["string", "null"]
},
"first_name": {
"type": "string"
"type": ["string", "null"]
},
"email": {
"type": "string"
"type": ["string", "null"]
}
}
}
Expand Down
7 changes: 5 additions & 2 deletions airbyte-integrations/connectors/source-posthog/source_posthog/source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@
# SOFTWARE.
#


from typing import Any, List, Mapping, Tuple

import pendulum
import requests
from airbyte_cdk.logger import AirbyteLogger
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources import AbstractSource
Expand All @@ -41,6 +41,7 @@
InsightsPath,
InsightsSessions,
Persons,
PingMe,
Trends,
)

Expand All @@ -50,11 +51,13 @@ def check_connection(self, logger: AirbyteLogger, config: Mapping[str, Any]) ->
try:
_ = pendulum.parse(config["start_date"], strict=True)
authenticator = TokenAuthenticator(token=config["api_key"])
stream = Cohorts(authenticator=authenticator)
stream = PingMe(authenticator=authenticator)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the error message the user sees if the API token is incorrect? it should help them understand what the issue is

Copy link
Contributor Author

@vovavovavovavova vovavovavovavova Jul 13, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The status wll be failed and log will get the following output in this case:
HTTPError('401 Client Error: Unauthorized for url: https://app.posthog.com/api/users/@me')

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets catch HTTPError separately and check

if exc.status_code == requests.codes.unauthorized:
   return False, f"Please check you api_key. Error: {repr(exc)}"

WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

applied but modified working version

records = stream.read_records(sync_mode=SyncMode.full_refresh)
_ = next(records)
return True, None
except Exception as e:
if isinstance(e, requests.exceptions.HTTPError) and e.response.status_code == 401:
return False, f"Please check you api_key. Error: {repr(e)}"
return False, repr(e)

def streams(self, config: Mapping[str, Any]) -> List[Stream]:
Expand Down
13 changes: 13 additions & 0 deletions airbyte-integrations/connectors/source-posthog/source_posthog/streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,3 +228,16 @@ class Trends(PosthogStream):

def path(self, stream_slice: Mapping[str, Any] = None, **kwargs) -> str:
return "insight/trend"


class PingMe(PosthogStream):
"""
Docs: https://posthog.com/docs/api/user
"""

def path(self, stream_slice: Mapping[str, Any] = None, **kwargs) -> str:
return "users/@me"

def parse_response(self, response: requests.Response, stream_state: Mapping[str, Any], **kwargs) -> Iterable[Mapping]:
response_json = response.json()
yield response_json