🎉 Source Iterable: add new streams #5915
Conversation
|
/test connector=connectors/source-iterable
|
|
/test connector=connectors/source-iterable
|
github-actions
bot
added
the
area/documentation
|
/test connector=connectors/source-iterable
|
airbyte-integrations/connectors/source-iterable/source_iterable/api.py
Outdated
Show resolved
Hide resolved
| @@ -0,0 +1,8 @@ | |||
| { | |||
| "properties": { | |||
There was a problem hiding this comment.
is there is a reason why we have so generic schema?
There was a problem hiding this comment.
As I can see there's no a strict schema for this
There was a problem hiding this comment.
I don't think this can be useful for users, because it lacks campaign id, also we definitely know at least some structure:
metric: <string>
value: <number>
There was a problem hiding this comment.
the id of campaign is inside of data, lets maybe rename it to metrics? or even better move it to the top level of object? i.e. campaign_metrics will looks like:
{
<id>: <campaign_id>,
<metric_i>: <value_i>,
...
}instead of
{
data: {...}
}
airbyte-integrations/connectors/source-iterable/source_iterable/api.py
Outdated
Show resolved
Hide resolved
| { | ||
| "properties": { | ||
| "data": { | ||
| "type": ["null", "object"] |
There was a problem hiding this comment.
same question regarding events
There was a problem hiding this comment.
Same here, custom fields
| try: | ||
| result[key] = int(value) | ||
| except ValueError: | ||
| result[key] = float(value) |
There was a problem hiding this comment.
Could you please explain what is it for?
There was a problem hiding this comment.
The response from the campaign_metrics endpoint is a csv file, the first line contains the name of the values, the second contains the values. There are 2 types of values: integer and floating point, so here we are casting from a string to the actual value type in a pretty naive way.
| { | ||
| "api_key": "test-api-key", | ||
| "start_date": "2020-12-12T00:00:00Z" | ||
| } |
| "requests==2.25.1", | ||
| "airbyte-cdk~=0.1", | ||
| "pendulum~=1.2", | ||
| "requests", |
There was a problem hiding this comment.
Why you didn't set version?
| yield {"data": self._parse_csv_string_to_dict(content)} | ||
|
|
||
| @staticmethod | ||
| def _parse_csv_string_to_dict(csv_string: str) -> Dict[str, Any]: |
There was a problem hiding this comment.
important to note that we assume that there is only one line in csv input
There was a problem hiding this comment.
mb move it outside of the class and make it return list of objects?
There was a problem hiding this comment.
Technically there are two lines, the first one is for variable names, and the second – for values.
This is used only here at the moment, and we can extract it when necessary.
We return CampaignMetircs as a raw json, so converting to object and then back to dict looks unnecessary, what do you think?
|
|
||
| return params | ||
|
|
||
| def stream_slices(self, **kwargs) -> Iterable[Optional[Mapping[str, any]]]: |
There was a problem hiding this comment.
How about slice not by one campaign, but by 20-30, since we already doing a lot of sub reads
# Conflicts: # airbyte-integrations/connectors/source-iterable/Dockerfile # airbyte-integrations/connectors/source-iterable/source_iterable/api.py # docs/integrations/sources/iterable.md
|
/test connector=connectors/source-iterable
|
|
/test connector=connectors/source-iterable
|
|
/test connector=connectors/source-iterable
|
|
/test connector=connectors/source-iterable
|
|
/test connector=connectors/source-iterable
|
|
/test connector=connectors/source-iterable
|
|
/publish connector=connectors/source-iterable
|
* Add new streams * Upd requirements versions * Upd docs * Remove tests for the templates stream * Upd csv field parsing * Fix file permissions * Set dependency version * Refactor * Merge * Upd licence * Add bulk metrics retrieving * Actualize schema
What
New streams added: Campaign Metrics, Events
How
Recommended reading order
source.pyapi.pyschemas/campaigns_metrics.jsonschemas/events.jsonPre-merge Checklist
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing./publishcommand described here