🐛 Destination S3: avro and parquet formats have issues with JsonToAvroSchemaConverter #8574

VitaliiMaltsev · 2021-12-07T09:47:58Z

What

Sync failed for Github Source (stream commits)
Logs:

�[35mdestination�[0m - 2021-11-26 13:17:54 INFO () DefaultAirbyteStreamFactory(lambda$create$0):61 - 2021-11-26 13:17:54 �[33mWARN�[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):58 - {} - Airbyte message consumer: failed.
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - Exception in thread "main" org.apache.avro.SchemaParseException: Can't redefine: author
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at org.apache.avro.Schema$Names.put(Schema.java:1542)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:805)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:967)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1234)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:995)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:979)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1234)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:995)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:979)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at org.apache.avro.Schema.toString(Schema.java:419)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at org.apache.avro.Schema.toString(Schema.java:391)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.destination.s3.writer.ProductionWriterFactory.create(ProductionWriterFactory.java:42)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.destination.s3.S3Consumer.startTracked(S3Consumer.java:53)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer.start(FailureTrackingAirbyteMessageConsumer.java:34)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:142)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:128)
�[35mdestination�[0m - 2021-11-26 13:17:54 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.destination.s3.S3Destination.main(S3Destination.java:29)
2021-11-26 13:17:54 INFO () DefaultReplicationWorker(run):141 - Destination thread complete.
2021-11-26 13:17:54 ERROR () DefaultReplicationWorker(run):145 - Sync worker failed.

How

According to Apache Avro documentation https://avro.apache.org/docs/current/spec.html#names
A schema or protocol may not contain multiple definitions of a fullname.
Added namespace to avro records

🚨 User Impact 🚨

There should not be any user impact in scope of Airbyte streams processing. This PR only add unique namespace into avro schemas for record types, previous namespace value for avro records was null. Only one potential impact is for further processing avro/parquet files outside of airbyte, cause earlier fullname of avro records was consist from name, with these changes would be consits from namespace.name

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

CLAassistant · 2021-12-07T09:48:12Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ alexandr-shegeda
❌ vmaltsev

vmaltsev seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

VitaliiMaltsev · 2021-12-07T10:11:55Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1548901668
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1548901668
No Python unittests run

…pace # Conflicts: # docs/integrations/destinations/s3.md

alexandr-shegeda

LGTM

sherifnada

@VitaliiMaltsev could you describe the user impact? Is this causing a breaking change for users?

sherifnada · 2021-12-08T01:06:52Z

...nation-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/AvroNameTransformer.java

@@ -26,4 +30,12 @@ private String checkFirsCharInStreamName(final String name) {
    }
  }

+  public String resovleNamespace(String fieldName, List<String> recordFieldNames) {


typo: resolve

sherifnada

Also, can you add a test case to DAT to catch this in the future for all destinations?

VitaliiMaltsev · 2021-12-08T09:45:42Z

@VitaliiMaltsev could you describe the user impact? Is this causing a breaking change for users?

@sherifnada updated user impact section

VitaliiMaltsev · 2021-12-08T11:13:35Z

Also, can you add a test case to DAT to catch this in the future for all destinations?

Please advise which test case do you mean for all destinations? This PR makes changes only to the conversion of the Json schema to the Avro schema in the JsonToAvroSchemaConverter class, since the avro schema, according to the documentation, does not allow having records with the same names. JsonToAvroSchemaConverter is only used for Destination S3 and GCS. JsonToAvroSchemaConverter class is covered by unit tests with in the JsonToAvroConverterTest class (test case "schema_with_the_same_object_names")

VitaliiMaltsev · 2021-12-21T14:28:04Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1607219513
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1607219513
No Python unittests run

VitaliiMaltsev · 2021-12-21T14:28:38Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1607221646
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1607221646
No Python unittests run

…port-namespace' into vmaltsev/8242-destination-s3-support-namespace # Conflicts: # airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonToAvroSchemaConverter.java

VitaliiMaltsev · 2021-12-21T15:01:38Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1607351786
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1607351786
No Python unittests run

VitaliiMaltsev · 2021-12-21T15:10:31Z

/publish connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1607392144
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1607392144

VitaliiMaltsev · 2021-12-21T15:10:58Z

/publish connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1607394219
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1607394219

VitaliiMaltsev · 2021-12-21T15:13:07Z

Thanks a lot for the update.

Can you format the code before merging the PR? Some of the formats are off.

formatted

VitaliiMaltsev · 2021-12-21T15:28:29Z

/publish connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1607468923
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1607468923

vmaltsev added 3 commits December 5, 2021 16:17

add namespace to avro record type

fff9beb

refactoring

2f259cd

Add unit tests

487360d

VitaliiMaltsev temporarily deployed to more-secrets December 7, 2021 09:48 Inactive

github-actions bot added the area/connectors Connector related issues label Dec 7, 2021

added CHANGELOG

f75c85e

github-actions bot added the area/documentation Improvements or additions to documentation label Dec 7, 2021

VitaliiMaltsev temporarily deployed to more-secrets December 7, 2021 09:58 Inactive

jrhizor temporarily deployed to more-secrets December 7, 2021 10:14 Inactive

VitaliiMaltsev requested review from etsybaev, sashaNeshcheret and alexandr-shegeda December 7, 2021 10:33

Merge branch 'master' into vmaltsev/8242-destination-s3-support-names…

0add1d8

…pace # Conflicts: # docs/integrations/destinations/s3.md

VitaliiMaltsev temporarily deployed to more-secrets December 7, 2021 10:39 Inactive

VitaliiMaltsev linked an issue Dec 7, 2021 that may be closed by this pull request

Destination S3: avro and parquet formats have issues with JsonToAvroSchemaConverter #8242

Closed

alexandr-shegeda approved these changes Dec 7, 2021

View reviewed changes

alexandr-shegeda marked this pull request as ready for review December 7, 2021 11:52

alexandr-shegeda requested a review from sherifnada December 7, 2021 11:52

VitaliiMaltsev linked an issue Dec 7, 2021 that may be closed by this pull request

🐛 Destination S3: Issues with converting JsonSchema to Avro format #8261

Closed

sherifnada reviewed Dec 8, 2021

View reviewed changes

fix typo in method name

8ca1165

VitaliiMaltsev temporarily deployed to more-secrets December 8, 2021 09:32 Inactive

fix typo in method name

dcde062

VitaliiMaltsev temporarily deployed to more-secrets December 8, 2021 10:03 Inactive

VitaliiMaltsev mentioned this pull request Dec 9, 2021

🎉 Source Mailchimp: updated Mailchimp schemas #7975

Merged

40 tasks

VitaliiMaltsev temporarily deployed to more-secrets December 21, 2021 14:23 Inactive

jrhizor temporarily deployed to more-secrets December 21, 2021 14:30 Inactive

jrhizor temporarily deployed to more-secrets December 21, 2021 14:31 Inactive

alexandr-shegeda added 2 commits December 21, 2021 16:38

reformat the code

1e16cb2

alexandr-shegeda temporarily deployed to more-secrets December 21, 2021 14:59 Inactive

jrhizor temporarily deployed to more-secrets December 21, 2021 15:03 Inactive

bump version

9d0d0ad

VitaliiMaltsev temporarily deployed to more-secrets December 21, 2021 15:11 Inactive

jrhizor temporarily deployed to more-secrets December 21, 2021 15:12 Inactive

bump Bigquery Denormalized version

affce39

VitaliiMaltsev temporarily deployed to more-secrets December 21, 2021 15:29 Inactive

jrhizor temporarily deployed to more-secrets December 21, 2021 15:30 Inactive

VitaliiMaltsev merged commit a3100cb into master Dec 21, 2021

VitaliiMaltsev deleted the vmaltsev/8242-destination-s3-support-namespace branch December 21, 2021 15:51

jrhizor mentioned this pull request Dec 22, 2021

Bump Airbyte version from 0.34.4-alpha to 0.35.0-alpha #9023

Merged

karinakuz added connectors/destination/bigquery-denormalized connectors/destinations-api connectors/destinations-warehouse and removed connectors/destinations-api labels Jan 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Destination S3: avro and parquet formats have issues with JsonToAvroSchemaConverter #8574

🐛 Destination S3: avro and parquet formats have issues with JsonToAvroSchemaConverter #8574

VitaliiMaltsev commented Dec 7, 2021 •

edited

Loading

CLAassistant commented Dec 7, 2021 •

edited

Loading

VitaliiMaltsev commented Dec 7, 2021 •

edited by github-actions bot

Loading

alexandr-shegeda left a comment

sherifnada left a comment

sherifnada Dec 8, 2021 •

edited

Loading

VitaliiMaltsev Dec 8, 2021

sherifnada left a comment

VitaliiMaltsev commented Dec 8, 2021

VitaliiMaltsev commented Dec 8, 2021

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Dec 21, 2021

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading

🐛 Destination S3: avro and parquet formats have issues with JsonToAvroSchemaConverter #8574

🐛 Destination S3: avro and parquet formats have issues with JsonToAvroSchemaConverter #8574

Conversation

VitaliiMaltsev commented Dec 7, 2021 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

CLAassistant commented Dec 7, 2021 • edited Loading

VitaliiMaltsev commented Dec 7, 2021 • edited by github-actions bot Loading

alexandr-shegeda left a comment

Choose a reason for hiding this comment

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Dec 8, 2021 • edited Loading

Choose a reason for hiding this comment

VitaliiMaltsev Dec 8, 2021

Choose a reason for hiding this comment

sherifnada left a comment

Choose a reason for hiding this comment

VitaliiMaltsev commented Dec 8, 2021

VitaliiMaltsev commented Dec 8, 2021

VitaliiMaltsev commented Dec 21, 2021 • edited by github-actions bot Loading

VitaliiMaltsev commented Dec 21, 2021 • edited by github-actions bot Loading

VitaliiMaltsev commented Dec 21, 2021 • edited by github-actions bot Loading

VitaliiMaltsev commented Dec 21, 2021 • edited by github-actions bot Loading

VitaliiMaltsev commented Dec 21, 2021 • edited by github-actions bot Loading

VitaliiMaltsev commented Dec 21, 2021

VitaliiMaltsev commented Dec 21, 2021 • edited by github-actions bot Loading

VitaliiMaltsev commented Dec 7, 2021 •

edited

Loading

CLAassistant commented Dec 7, 2021 •

edited

Loading

VitaliiMaltsev commented Dec 7, 2021 •

edited by github-actions bot

Loading

sherifnada Dec 8, 2021 •

edited

Loading

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Dec 21, 2021 •

edited by github-actions bot

Loading