Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add optional compression support in avro2json workflow #1371

Open
marekhorst opened this issue Aug 3, 2022 · 0 comments
Open

Add optional compression support in avro2json workflow #1371

marekhorst opened this issue Aug 3, 2022 · 0 comments
Assignees
@marekhorst

Comments

@marekhorst
Copy link
Member

Currently avro2json workflow is responsible for translating avro input into json output.

It will be convenient, also for #918, to allow enabling compression by providing e.g. compression_method input parameter which could result in adding:

.option("compression", params.compressionMethod)

before .json() spark method producing json out of the Dataset.

Compression should be disabled when set to null, empty or $UNDEFINED$ value.
marekhorst added a commit that referenced this issue Aug 3, 2022
@marekhorst
Closes #1371: Add optional compression support in avro2json workflow
7790a51 
Accepting `compression_method` input parameter set to `$UNDEFINED$` value by default (compression disabled).
@marekhorst marekhorst self-assigned this Aug 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@marekhorst marekhorst

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@marekhorst