🎉 New Source: Mailersend [low-code CDK] #18669
Conversation
github-actions
bot
added
area/connectors
There was a problem hiding this comment.
Hello @cirdes, Marcos from Airbyte here 👋 . We received more than 25 new contributions along the weekend. One is yours 🎉 thank so much for! Our team is limited and maybe the review process can take longer than expected. As described in the Airbyte's Hacktoberfest your contribution was submitted before November 2nd and it is eligible to win the prize. The review process will validate other requirements. I ask to you patience until someone from the team review it.
Because I reviewed some contributions for Hacktoberfest so far I saw some common patterns you can check in advance:
- Make sure you have added connector documentation to
/docs/integrations/ - Remove the file
catalogfrom/integration_tests - Edit the
sample_config.jsoninside/integration_tests - For the
configured_catalogyou can use onlyjson_schema: {} - Add title to all properties in the
spec.yaml - Make sure the
documentationUrlin thespec.yamlredirect to Airbyte's future connector page, eg: connector Airtable thedocumentationUrl: https://docs.airbyte.com/integrations/sources/airtable - Review now new line at EOF (end-of-file) for all files.
If possible send to me a DM in Slack with the tests credentials, this process will make easier to us run integration tests and publish your connector. If you only has production keys, make sure to create a bootstrap.md explaining how to get the keys.
| date_from: | ||
| type: string | ||
| description: Timestamp is assumed to be UTC. Must be lower than date_to. | ||
| examples: | ||
| - "1443651141" | ||
| date_to: | ||
| type: string | ||
| description: Timestamp is assumed to be UTC. Must be higher than date_from. | ||
| examples: | ||
| - "1443651141" |
There was a problem hiding this comment.
Can you rename this to start_date and end_date?
There was a problem hiding this comment.
There was a problem hiding this comment.
I think we can rename because the user don't need to know the name the API uses, probably is better to use a more common terminology inside Airbyte.
airbyte-integrations/connectors/source-mailersend/source_mailersend/spec.yaml
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-mailersend/source_mailersend/mailersend.yaml
Show resolved
Hide resolved
airbyte-integrations/connectors/source-mailersend/source_mailersend/mailersend.yaml
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
@cirdes two comments, one about the parameters and other about the incremental implementation.
| activity_stream: | ||
| $ref: "*ref(definitions.base_stream)" | ||
| $options: | ||
| name: "activity" | ||
| primary_key: "id" | ||
| path: "/activity/{{ config['domain_id'] }}" |
There was a problem hiding this comment.
I saw the API supports the query parameter from/to using dates, this allow using incremental syncs. @cirdes what about implementing this feature for this connector? I can help you if you need some help.
There was a problem hiding this comment.
@marcosmarxm, the API does supports query parameter using dates. But it expect the params to be a integer like 1443651141 (utc in seconds). DatetimeStreamSlicer is using date as strings "2021-02-01T00:00:00.000000+0000".
Is it possible to use DatetimeStreamSlicer in this case?
There was a problem hiding this comment.
@girarda can you have me if it is possible to convert to timestamp?
There was a problem hiding this comment.
you can set the date format to %s to convert it to a timestamp. the sendgrid connector is an example
There was a problem hiding this comment.
thanks guy, I will have a look on it!
There was a problem hiding this comment.
I don't know if a got the point here.
The challenge is not about the stream_slicer itself.
In the spec.yml I'm asking the user to enter start_time(or date_from) as a number because I need to send start_time as number to Mailersend's API.
mailersend.yml
request_options_provider:
request_parameters:
limit: "100"
start_time: "{{ config['date_from'] }}"
But if a change start_time to be a string link "2022-11-05" I will need to cast the string into Date and then get the unix timestamp but its seems to be not possible in the yml.
In sendgrid exemple it is doing that using python and pendulum lib -> https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-sendgrid/source_sendgrid/streams.py#L98
It's seems to be a complex scenario or I'm missing a simple solution here?
cc: @marcosmarxm
There was a problem hiding this comment.
@cirdes, you can configure the datetime format of the stream slicer's fields. The %s directive converts the datetime to a timestamp.
Using Sendgrid as an example:
stream_slicer:
type: "DatetimeStreamSlicer"
start_datetime:
datetime: "{{ config['start_time'] }}"
datetime_format: "%s"
There was a problem hiding this comment.
@cirdes let's schedule a time tomorrow to implement the stream slicer.
|
Hello! I'm going to be out of the office this Friday and won't be able to review your contribution again today, I return next Monday. So far, most contributions look solid and are almost done to be approved. As said in Chris' comment all contributions made before 2-November are eligible to receive the prize and have 2 weeks to merge the contributions. But I ensure next week we're going to have your contribution merged. If you have questions about the implementation you can send them in Sorry the inconvenience and see you again next week, thank you so much for your contribution! |
| request_options_provider: | ||
| request_parameters: | ||
| limit: "100" | ||
| stream_slicer: |
There was a problem hiding this comment.
@girarda , I'm trying to add stream slicer support but I'm getting this error:
{"message": "Something went wrong in the connector. See the logs for more details.", "internal_message": "invalid literal for int() with base 10: '2022-11-06'", "stack_trace": "Traceback (most recent call last):\n File \"/Users/cirdes/Workspace/airbyte/airbyte-integrations/connectors/source-mailersend/main.py\", line 13
The api should be called like this:
https://api.mailersend.com/v1/activity/{domain_id}/?date_from=1667746844
date_from is the request_parameters and it's a Unix Timestamp (integer)
in spec.yml I have start_date
start_date:
type: string
description: Timestamp is assumed to be UTC. Must be lower than date_to.
examples:
- "2022-11-06"
And I'm using "2022-11-06" as start_date.
|
/test connector=connectors/source-mailersend
Build FailedTest summary info: |
|
/test connector=connectors/source-mailersend
Build PassedTest summary info: |
|
/publish connector=connectors/source-mailersend
if you have connectors that successfully published but failed definition generation, follow step 4 here |
* New source Mailersend * first review * trying to add stream_slicer * change spec.yml * add streamslicer * Update spec.yaml * add mailersend to source def * auto-bump connector version Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
What
Developing new connector for the source: Mailersend
How
Developed using low-code cdk
Streams
Recommended reading order
🚨 User Impact 🚨
No breaking changes, Just addition of new source
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampledocs/integrations/README.mdairbyte-integrations/builds.mdAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing/publishcommand described hereTests
Integration & Acceptance