Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Iceberg maxOpenWriters configurable #6650

Closed
guyco33 opened this issue Jan 20, 2021 · 4 comments
Closed

Make Iceberg maxOpenWriters configurable #6650

guyco33 opened this issue Jan 20, 2021 · 4 comments
Labels
good first issue Good for newcomers
Milestone

Comments

@guyco33
Copy link
Member

guyco33 commented Jan 20, 2021

Creating more than 100 partitions fails with the following error:

[84148226] Query failed (#20210118_061400_09812_m328m): Exceeded limit of 100 open writers for partitions

private final int maxOpenWriters = 100; // TODO: make this configurable

Possible solution is to take advantage of hive.max-partitions-per-writers configuration that can be set also in iceberg

@phd3
Copy link
Member

phd3 commented Jan 20, 2021

I think it might be better to just add a new config in the IcebergConfig class, independent of the hive config.

@phd3 phd3 added the good first issue Good for newcomers label Jan 20, 2021
@akshayrai
Copy link
Contributor

Taking a look! I will raise a PR for this!

@findepi
Copy link
Member

findepi commented Feb 19, 2021

@phd3 has this been covered in 352?

@phd3
Copy link
Member

phd3 commented Feb 19, 2021

yes, closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Development

No branches or pull requests

5 participants