🐛 Source Intercom: Fix "updated_at" data type #6087

ReptilianBrain · 2021-09-15T14:19:56Z

The Intercom API send a integer for the "updated_at" field in the companies endpoint. Which cause issue later on when dbt tries to create the table.

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

The Intercom API send a integer for the "updated_at" field in the companies endpoint. Which cause issue later on when dbt tries to create the table.

marcosmarxm · 2021-09-15T14:25:43Z

Thanks @ReptilianBrain could you share the logs from the failures?

ReptilianBrain · 2021-09-15T14:29:22Z

here's the log I recveived from airbyte.

2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - Completed with 2 errors and 0 warnings: 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - Database Error in model intercom__company_attributes (models/generated/airbyte_tables/dev_sandbox/intercom__company_attributes.sql) 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - Invalid data 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - DETAIL: 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - ----------------------------------------------- 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - error: Invalid data 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - code: 8001 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - context: Invalid format or data given: 1507831043 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - query: 132707242 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - location: funcs_timestamp.cpp:274 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - process: query3_46_132707242 [pid=2261] 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - ----------------------------------------------- 2021-09-15 14:25:54 INFO () LineGobbler(voidCall):85 - compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/dev_sandbox/intercom__company_attributes.sql

also here's a response from the intercom api if it can help:

{ "type": "company", "company_id": "<REDACTED>", "id": "<REDACTED>", "app_id": "<REDACTED>", "name": "<REDACTED>", "created_at": 1507897642, "updated_at": 1631306673, "last_request_at": 1631306303, "monthly_spend": 0, "session_count": 0, "user_count": <REDACTED>, "tags": { "type": "tag.list", "tags": [] }, "segments": { "type": "segment.list", "segments": [] }, "plan": {}, "custom_attributes": {} }

prasrvenkat · 2021-09-15T15:15:35Z

@marcosmarxm FYI, this could be for all intercom streams having updated_at btw. We had similar problem with some other streams, we are not using dbt so we handle it ourselves in our flow downstream (pubsub destination -> our services).

This PR #5908 changed most such data-time references when it was coming as integer from API, but it did not touch updated_at. I am not sure if its because it's used as a cursor and changing would impact incremental sync in someway or just missed.

marcosmarxm · 2021-09-17T19:58:19Z

Hi @ReptilianBrain we're going to review this later monday! thanks for the contribution again.

lazebnyi · 2021-09-20T11:45:05Z

/test connector=connectors/source-intercom

🕑 connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/1253381443
✅ connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/1253381443
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        45     45     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  776    430    45%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                          Stmts   Miss  Cover
	 -------------------------------------------------
	 source_intercom/__init__.py       2      0   100%
	 source_intercom/source.py       143     62    57%
	 -------------------------------------------------
	 TOTAL                           145     62    57%

lazebnyi · 2021-09-20T12:02:27Z

/publish connector=connectors/source-intercom

🕑 connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/1253430362
✅ connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/1253430362

Intercom: Fix "updated_at" data type

feeee90

The Intercom API send a integer for the "updated_at" field in the companies endpoint. Which cause issue later on when dbt tries to create the table.

github-actions bot added the area/connectors Connector related issues label Sep 15, 2021

marcosmarxm added the community_pr label Sep 15, 2021

marcosmarxm self-assigned this Sep 15, 2021

marcosmarxm requested a review from lazebnyi September 17, 2021 19:56

lazebnyi added 3 commits September 20, 2021 14:05

Add start date fromating to timestamp

7ff5c81

Update updated_at field for all schemas and integration tests

16f1ec5

Bumped version

104fbbd

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 20, 2021

lazebnyi approved these changes Sep 20, 2021

View reviewed changes

lazebnyi changed the title ~~Intercom: Fix "updated_at" data type~~ 🐛 Source Intercom: Fix "updated_at" data type Sep 20, 2021

lazebnyi merged commit 65b42c8 into airbytehq:master Sep 20, 2021

lazebnyi mentioned this pull request Sep 22, 2021

Fix "android_last_seen_at" data type #6082

Merged

38 tasks

jrhizor mentioned this pull request Sep 25, 2021

Bump Airbyte version from 0.29.21-alpha to 0.29.22-alpha #6450

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Intercom: Fix "updated_at" data type #6087

🐛 Source Intercom: Fix "updated_at" data type #6087

ReptilianBrain commented Sep 15, 2021 •

edited by lazebnyi

Loading

marcosmarxm commented Sep 15, 2021

ReptilianBrain commented Sep 15, 2021

prasrvenkat commented Sep 15, 2021

marcosmarxm commented Sep 17, 2021

lazebnyi commented Sep 20, 2021 •

edited by github-actions bot

Loading

lazebnyi commented Sep 20, 2021 •

edited by github-actions bot

Loading

🐛 Source Intercom: Fix "updated_at" data type #6087

🐛 Source Intercom: Fix "updated_at" data type #6087

Conversation

ReptilianBrain commented Sep 15, 2021 • edited by lazebnyi Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

marcosmarxm commented Sep 15, 2021

ReptilianBrain commented Sep 15, 2021

prasrvenkat commented Sep 15, 2021

marcosmarxm commented Sep 17, 2021

lazebnyi commented Sep 20, 2021 • edited by github-actions bot Loading

lazebnyi commented Sep 20, 2021 • edited by github-actions bot Loading

ReptilianBrain commented Sep 15, 2021 •

edited by lazebnyi

Loading

lazebnyi commented Sep 20, 2021 •

edited by github-actions bot

Loading

lazebnyi commented Sep 20, 2021 •

edited by github-actions bot

Loading