Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hive Catalog cannot create table with TimestamptzType field #583

Closed
HonahX opened this issue Apr 5, 2024 · 0 comments · Fixed by #585
Closed

Hive Catalog cannot create table with TimestamptzType field #583

HonahX opened this issue Apr 5, 2024 · 0 comments · Fixed by #585

Comments

@HonahX
Copy link
Contributor

HonahX commented Apr 5, 2024

Apache Iceberg version

0.6.0 (latest release)

Please describe the bug 🐞

This bug was reported by Jorge Arada on slack: https://apache-iceberg.slack.com/archives/C029EE6HQ5D/p1712309474600169

Reported example to reproduce:

catalog = load_hive("iceberg_catalog", {"uri": "some_url"})
 fields = [NestedField( field_id=0, name="teste_tz", field_type=TimestamptzType(), required=False)]
schema = Schema(*fields)
catalog.create_table(identifier="raw.test_table", schema=schema)

Error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/revolut/app/.venv/lib/python3.11/site-packages/pyiceberg/catalog/hive.py", line 315, in create_table
    sd=_construct_hive_storage_descriptor(schema, location),
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/revolut/app/.venv/lib/python3.11/site-packages/pyiceberg/catalog/hive.py", line 155, in _construct_hive_storage_descriptor
    [FieldSchema(field.name, visit(field.field_type, SchemaToHiveConverter()), field.doc) for field in schema.fields],
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/revolut/app/.venv/lib/python3.11/site-packages/pyiceberg/catalog/hive.py", line 155, in <listcomp>
    [FieldSchema(field.name, visit(field.field_type, SchemaToHiveConverter()), field.doc) for field in schema.fields],
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/functools.py", line 909, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/revolut/app/.venv/lib/python3.11/site-packages/pyiceberg/schema.py", line 847, in _
    return visitor.primitive(obj)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/revolut/app/.venv/lib/python3.11/site-packages/pyiceberg/catalog/hive.py", line 228, in primitive
    return HIVE_PRIMITIVE_TYPES[type(primitive)]
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: <class 'pyiceberg.types.TimestamptzType'>

It seems we miss the TimestampzTypein the Iceberg to Hive type conversion map.
A similar issue has been observed and fixed in glue: #366

It would be good to include the fix in 0.6.1

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant