Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deephaven.ParquetTools.readTable() doesn't support Parquet date32 or date64 data types #1253

Closed
mikeblas opened this issue Sep 11, 2021 · 1 comment
Labels
bug Something isn't working triage

Comments

@mikeblas
Copy link
Contributor

Description

Using readTable() I've found that I can't read files that contain columns with the date32 or date64 data types. An exception is thrown. If we don't support it that's cool -- let's document it, and try to express a workaround about it if possible. But it would be nice to support the types ...

Steps to reproduce

from deephaven.ParquetTools import readTable

s = readTable("/data/equity-quotes-2021-03-04.parquet")

Expected results

Table is imported successfully

Actual results

An exception is thrown; here's how it starts out:

r-Scheduler-Serial-1 | i.d.g.s.SessionState      | Internal Error 'c5ef3418-ec87-4e7d-9b81-c1d7da6bfd02' java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'RuntimeError'>
Value: io.deephaven.UncheckedDeephavenException: Unable to read column [Date]: DateLogicalType not supported
	at io.deephaven.db.v2.parquet.ParquetSchemaReader.lambda$readParquetSchema$2(ParquetSchemaReader.java:272)
	at java.util.Optional.orElseThrow(Optional.java:290)
	at io.deephaven.db.v2.parquet.ParquetSchemaReader.readParquetSchema(ParquetSchemaReader.java:266)
	at io.deephaven.db.tables.utils.ParquetTools.convertSchema(ParquetTools.java:650)
	at io.deephaven.db.tables.utils.ParquetTools.readTableInternal(ParquetTools.java:386)
	at io.deephaven.db.tables.utils.ParquetTools.readTable(ParquetTools.java:110)
	at org.jpy.PyLib.executeCode(Native Method)
:
:

Additional details and attachments

parquet_date.zip

@mikeblas mikeblas added bug Something isn't working triage labels Sep 11, 2021
@jcferretti
Copy link
Member

Fixed by #1254

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

No branches or pull requests

2 participants