Jira source normalization step error

[GiannisDimitriou]

OS Version / Instance: Ubuntu 20.04
Memory / Disk: 7.5Gb
Deployment: Docker
Airbyte Version: 0.29.17-alpha
Source name/version: Jira 0.2.10
Destination name/version: Postgresql 0.3.11
Step: Sync job
Description:

Current Behavior

When basic normalization is enabled for the Jira source connector the job fails with the following error
io.airbyte.workers.WorkerException: Normalization process wasn't successful

Expected Behavior

After successful syncing, the normalization step should be successful.

Logs

airbyte_log.txt

Steps to Reproduce

1. docker-compose up
2. Setup Source (Jira) and Destination (PostgreSQL)
3. Setup Connection using the source/destination created above, choose issues as stream and enable basic normalization

[marcosmarxm]

thanks for reporting this @GiannisDimitriou! I added to the connector roadmap and we're going to solve this soon.

[keu]

@sherifnada @eliziario ,
I think we should add the cause and investigation information here, it is not clear what was the issue, and how we going to fix it.
All I see is a bug report from the user and the actual PR with some changes.

[ChristopheDuong]

@sherifnada @eliziario ,
I think we should add the cause and investigation information here, it is not clear what was the issue, and how we going to fix it.
All I see is a bug report from the user and the actual PR with some changes.

From the logs, I would guess it seems the jira source is not declaring the updated column in the catalog of some stream but it is declaring it as a cursor field anyway (and then fails to use it when building the SCD model)

2021-09-13 07:19:38 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/stream_processor.py", line 244, in process
2021-09-13 07:19:38 ERROR () LineGobbler(voidCall):85 -     self.generate_scd_type_2_model(from_table, column_names) + where_clause,
2021-09-13 07:19:38 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/stream_processor.py", line 587, in generate_scd_type_2_model
2021-09-13 07:19:38 ERROR () LineGobbler(voidCall):85 -     cursor_field=self.get_cursor_field(column_names),
2021-09-13 07:19:38 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/lib/python3.8/site-packages/normalization/transform_catalog/stream_processor.py", line 602, in get_cursor_field
2021-09-13 07:19:38 ERROR () LineGobbler(voidCall):85 -     cursor = column_names[self.cursor_field[0]][0]
2021-09-13 07:19:38 ERROR () LineGobbler(voidCall):85 - KeyError: 'updated'

Can we take a closer look at the catalog.json for this source?