You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The base S3 destination connector has had substantial updates made to it since the last time that we ran a build of the S3-Glue destination. There are also pull requests that were never merged upstream which we are relying on in our build of the connector. We need to generate an updated build that incorporates the improvements from the base S3 destination to increase the performance of our syncs.
Plan/Design
The un-merged changes in the upstream code need to be re-created against the current state of the repository:
destination-s3
Added a new "stringify" argument to the JsonLSerialized buffer
If true, conditional logic will execute that gathers root level objects as strings, including airbyte_data.
Added a new Stringify utility.
Threaded the new stringify argument through the necessary classes and methods, incl. S3JsonlFormatConfig.
destination-s3-glue
Added new s3_glue interfaces MetastoreFormatConfig and MetastoreJsonlFormatConfig with methods to get input, output, and return a serializationLibrary.
Passing in an S3FormatConfig object to the GlueDestinationConfig.
Passing in a MetastoreFormatConfig object instead of the serializationLibrary in the operations code. This is config is passed as an additional argument to transformSchemaRecursive in GlueOperations or upsertTable in MetastoreOperations.
Abstracted default values for glue db, serialization library, text input and output formats to MetastoreConstants.
Once these changes have been implemented and any other structural changes required in the s3-glue code are fixed then we will build a new version of the s3-glue connector for testing in our QA environment.
The text was updated successfully, but these errors were encountered:
Description/Context
The base S3 destination connector has had substantial updates made to it since the last time that we ran a build of the S3-Glue destination. There are also pull requests that were never merged upstream which we are relying on in our build of the connector. We need to generate an updated build that incorporates the improvements from the base S3 destination to increase the performance of our syncs.
Plan/Design
The un-merged changes in the upstream code need to be re-created against the current state of the repository:
destination-s3
Added a new "stringify" argument to the JsonLSerialized buffer
If true, conditional logic will execute that gathers root level objects as strings, including airbyte_data.
Added a new Stringify utility.
Threaded the new stringify argument through the necessary classes and methods, incl. S3JsonlFormatConfig.
destination-s3-glue
Added new s3_glue interfaces MetastoreFormatConfig and MetastoreJsonlFormatConfig with methods to get input, output, and return a serializationLibrary.
Passing in an S3FormatConfig object to the GlueDestinationConfig.
Passing in a MetastoreFormatConfig object instead of the serializationLibrary in the operations code. This is config is passed as an additional argument to transformSchemaRecursive in GlueOperations or upsertTable in MetastoreOperations.
Abstracted default values for glue db, serialization library, text input and output formats to MetastoreConstants.
Once these changes have been implemented and any other structural changes required in the s3-glue code are fixed then we will build a new version of the s3-glue connector for testing in our QA environment.
The text was updated successfully, but these errors were encountered: