Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disallow creating Delta tables partitioned by all columns #14001

Closed
ebyhr opened this issue Sep 6, 2022 · 0 comments · Fixed by #18354
Closed

Disallow creating Delta tables partitioned by all columns #14001

ebyhr opened this issue Sep 6, 2022 · 0 comments · Fixed by #18354
Assignees
Labels
bug Something isn't working

Comments

@ebyhr
Copy link
Member

ebyhr commented Sep 6, 2022

Delta doesn't support creating such tables.

Cannot use all columns for partition columns

Steps to reproduce:

CREATE TABLE test1 (c1 int) WITH (partitioned_by = array['c1']);
INSERT INTO test1 VALUES (1);

java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
	at java.base/java.util.Objects.checkIndex(Objects.java:359)
	at java.base/java.util.ArrayList.get(ArrayList.java:427)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:1538)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1485)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:583)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:777)
	at org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:646)
	at io.trino.plugin.deltalake.DeltaLakeWriter.readStatistics(DeltaLakeWriter.java:187)
	at io.trino.plugin.deltalake.DeltaLakeWriter.getDataFileInfo(DeltaLakeWriter.java:169)
	at io.trino.plugin.deltalake.DeltaLakePageSink.closeWriter(DeltaLakePageSink.java:425)
	at io.trino.plugin.deltalake.DeltaLakePageSink.doFinish(DeltaLakePageSink.java:233)
	at io.trino.hdfs.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:25)
	at io.trino.hdfs.HdfsEnvironment.doAs(HdfsEnvironment.java:94)
	at io.trino.plugin.deltalake.DeltaLakePageSink.finish(DeltaLakePageSink.java:226)
	at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorPageSink.finish(ClassLoaderSafeConnectorPageSink.java:77)
	at io.trino.operator.TableWriterOperator.finish(TableWriterOperator.java:212)
	at io.trino.operator.Driver.processInternal(Driver.java:429)
	at io.trino.operator.Driver.lambda$process$10(Driver.java:314)
	at io.trino.operator.Driver.tryWithLock(Driver.java:706)
	at io.trino.operator.Driver.process(Driver.java:306)
	at io.trino.operator.Driver.processForDuration(Driver.java:277)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:737)
	at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:164)
	at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:514)
	at io.trino.$gen.Trino_testversion____20220906_065104_71.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Below INSERT also fails:

CREATE TABLE test2 (c1 int, c2 int) WITH (partitioned_by = array['c1', 'c2']);
INSERT INTO test2 VALUES (1, 2);
@ebyhr ebyhr added the bug Something isn't working label Sep 6, 2022
@ebyhr ebyhr self-assigned this Sep 8, 2022
@ebyhr ebyhr removed their assignment Mar 3, 2023
@ebyhr ebyhr changed the title Inserting into Delta Lake tables partitioned by all columns fail Disallow creating Delta tables partitioned by all columns Jul 20, 2023
@ebyhr ebyhr self-assigned this Jul 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Development

Successfully merging a pull request may close this issue.

1 participant