Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛 Source Sendgrid: add start_time config and correct primary_key #4682

Merged
merged 4 commits into from
Jul 10, 2021
Merged

🐛 Source Sendgrid: add start_time config and correct primary_key #4682

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
28a016f
add start_time config and correct primary_key
marcosmarxm Jul 10, 2021
d91679c
correct integration tests
marcosmarxm Jul 10, 2021
918518a
correct type
marcosmarxm Jul 10, 2021
9e2320e
config txt and primary_key
marcosmarxm Jul 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...ain/resources/config/STANDARD_SOURCE_DEFINITION/fbb5fbe2-16ad-4cf4-af7d-ff9d9c316c87.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"sourceDefinitionId": "fbb5fbe2-16ad-4cf4-af7d-ff9d9c316c87",
"name": "Sendgrid",
"dockerRepository": "airbyte/source-sendgrid",
"dockerImageTag": "0.2.4",
"dockerImageTag": "0.2.5",
"documentationUrl": "https://hub.docker.com/r/airbyte/source-sendgrid",
"icon": "sendgrid.svg"
}
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/source_definitions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@
- sourceDefinitionId: fbb5fbe2-16ad-4cf4-af7d-ff9d9c316c87
name: Sendgrid
dockerRepository: airbyte/source-sendgrid
dockerImageTag: 0.2.4
dockerImageTag: 0.2.5
documentationUrl: https://hub.docker.com/r/airbyte/source-sendgrid
icon: sendgrid.svg
- sourceDefinitionId: 9e0556f4-69df-4522-a3fb-03264d36b348
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-sendgrid/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,5 @@ RUN pip install .
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.2.4
LABEL io.airbyte.version=0.2.5
LABEL io.airbyte.name=airbyte/source-sendgrid
55 changes: 0 additions & 55 deletions ...egrations/connectors/source-sendgrid/sample_files/no_spam_reports_configured_catalog.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -354,35 +354,6 @@
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
{
"stream": {
"name": "blocks",
"json_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"created": {
"type": "integer"
},
"email": {
"type": "string"
},
"reason": {
"type": "string"
},
"status": {
"type": "string"
}
}
},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["created"]
},
"sync_mode": "incremental",
"cursor_field": ["created"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "bounces",
Expand Down Expand Up @@ -411,32 +382,6 @@
"sync_mode": "incremental",
"cursor_field": ["created"],
"destination_sync_mode": "append"
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why these two were removed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they are returning zero records in tests and break it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, then I think the name of the file slightly off

{
"stream": {
"name": "invalid_emails",
"json_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"created": {
"type": "integer"
},
"email": {
"type": "string"
},
"reason": {
"type": "string"
}
}
},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["created"]
},
"sync_mode": "incremental",
"cursor_field": ["created"],
"destination_sync_mode": "append"
}
]
}
8 changes: 4 additions & 4 deletions airbyte-integrations/connectors/source-sendgrid/source_sendgrid/schemas/contacts.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"type": "string"
},
"alternate_emails": {
"type": "array"
"type": ["null", "array"]
},
"city": {
"type": "string"
Expand All @@ -36,13 +36,13 @@
"type": "string"
},
"list_ids": {
"type": "array"
"type": ["null", "array"]
},
"created_at": {
"type": "array"
"type": "string"
},
"updated_at": {
"type": "array"
"type": "string"
},
"_metadata": {
"type": "object",
Expand Down
2 changes: 1 addition & 1 deletion ...e-integrations/connectors/source-sendgrid/source_sendgrid/schemas/suppression_groups.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"type": "object",
"properties": {
"id": {
"type": "string"
"type": "integer"
},
"name": {
"type": "string"
Expand Down
5 changes: 5 additions & 0 deletions airbyte-integrations/connectors/source-sendgrid/source_sendgrid/spec.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@
"apikey": {
"type": "string",
"description": "API Key, use <a href=\"https://app.sendgrid.com/settings/api_keys/\">admin</a> to generate this key."
},
"start_time": {
"type": "integer",
"description": "Start time in timestamp integer format. Any data before this timestamp will not be replicated.",
"examples": [1558359837]
}
}
}
Expand Down
10 changes: 10 additions & 0 deletions airbyte-integrations/connectors/source-sendgrid/source_sendgrid/streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,8 @@ def initial_path() -> str:


class GlobalSuppressions(SendgridStreamOffsetPagination, SendgridStreamIncrementalMixin):
primary_key = "email"

def path(self, **kwargs) -> str:
return "suppression/unsubscribes"

Expand All @@ -205,20 +207,28 @@ def path(self, **kwargs) -> str:


class Blocks(SendgridStreamOffsetPagination, SendgridStreamIncrementalMixin):
primary_key = "email"

def path(self, **kwargs) -> str:
return "suppression/blocks"


class Bounces(SendgridStream, SendgridStreamIncrementalMixin):
primary_key = "email"

def path(self, **kwargs) -> str:
return "suppression/bounces"


class InvalidEmails(SendgridStreamOffsetPagination, SendgridStreamIncrementalMixin):
primary_key = "email"

def path(self, **kwargs) -> str:
return "suppression/invalid_emails"


class SpamReports(SendgridStreamOffsetPagination, SendgridStreamIncrementalMixin):
primary_key = "email"

def path(self, **kwargs) -> str:
return "suppression/spam_reports"