Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: AvroUtils.toAvroSchema supports a logicaltypes.Date type #23299

Closed
gabrywu opened this issue Sep 20, 2022 · 6 comments · Fixed by #23969 or #24756
Closed

[Feature Request]: AvroUtils.toAvroSchema supports a logicaltypes.Date type #23299

gabrywu opened this issue Sep 20, 2022 · 6 comments · Fixed by #23969 or #24756

Comments

@gabrywu
Copy link
Member

gabrywu commented Sep 20, 2022

What would you like to happen?

Hi, folks,
I'm using a SQLTransform to read & write a Date field to parquet files. However, AvroUtils.toAvroSchema can't convert org.apache.beam.sdk.schemas.logicaltypes.Date to a avro date type. Because its identifier is beam:logical_type:date:v1, which can't be handled in method org.apache.beam.sdk.schemas.utils.AvroUtils.genericFromBeamField

Issue Priority

Priority: 2

Issue Component

Component: sdk-java-core

@gabrywu
Copy link
Member Author

gabrywu commented Sep 22, 2022

Schema beamSchema = new Schema.Builder() .addLogicalTypeField("date1", SqlTypes.DATE) .build(); Assert.assertThrows(RuntimeException.class, () -> AvroUtils.toAvroSchema(beamSchema));

AvroUtils.toAvroSchema can't process SqlTypes.DATE.

@gabrywu
Copy link
Member Author

gabrywu commented Sep 22, 2022

@reuvenlax @kanterov @piter75

@chamikaramj chamikaramj added the io label Oct 25, 2022
@Abacn
Copy link
Contributor

Abacn commented Oct 31, 2022

.take-issue

@Abacn
Copy link
Contributor

Abacn commented Oct 31, 2022

beam:logical_type:date:v1 is a portable logical type (with URN). The support of portable logical types is currently underway. For now as a workaround, could create a PTransform that convert Row with beam:logical_type:date:v1 type to Date type in its schema.

@gabrywu
Copy link
Member Author

gabrywu commented Nov 1, 2022

beam:logical_type:date:v1 is a portable logical type (with URN). The support of portable logical types is currently underway. For now as a workaround, could create a PTransform that convert Row with beam:logical_type:date:v1 type to Date type in its schema.

I'm creating a table using a Beam SQL, can't apply a PTransform before it's created.

@Abacn
Copy link
Contributor

Abacn commented Dec 2, 2022

Reopen because it is rolled back #24489

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment