Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pydantic warning when appending data #971

Closed
ndrluis opened this issue Jul 27, 2024 · 2 comments · Fixed by #972
Closed

Pydantic warning when appending data #971

ndrluis opened this issue Jul 27, 2024 · 2 comments · Fixed by #972

Comments

@ndrluis
Copy link
Collaborator

ndrluis commented Jul 27, 2024

Apache Iceberg version

0.7.0rc2

Please describe the bug 🐞

In 0.7.0rc1 we don't have this problem.

Code to reproduce the warning

from pyiceberg.catalog import load_catalog
import pyarrow as pa
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, StringType, DoubleType

catalog = load_catalog(
    "demo",
    **{
        "type": "rest",
        "uri": "http://localhost:8181",
        "s3.endpoint": "http://localhost:9000",
        "s3.access-key-id": "admin",
        "s3.secret-access-key": "password",
        "warehouse": "demo",
    },
)

catalog.create_namespace_if_not_exists("default")

schema = Schema(
    NestedField(1, "city", StringType(), required=False),
    NestedField(2, "lat", DoubleType(), required=False),
    NestedField(3, "long", DoubleType(), required=False),
)

tbl = catalog.create_table_if_not_exists("default.cities", schema=schema)

df = pa.Table.from_pylist(
    [
        {"city": "Amsterdam", "lat": 52.371807, "long": 4.896029},
        {"city": "San Francisco", "lat": 37.773972, "long": -122.431297},
        {"city": "Drachten", "lat": 53.11254, "long": 6.0989},
        {"city": "Paris", "lat": 48.864716, "long": 2.349014},
    ],
)

tbl.append(df)

The append process is working but we receive this warning

UserWarning: Pydantic serializer warnings:
  Expected `TableIdentifier` but got `dict` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_json(
ndrluis added a commit to ndrluis/iceberg-python that referenced this issue Jul 27, 2024
This will help us avoid propagating warnings to our users, as occurred in apache#971.
@ndrluis
Copy link
Collaborator Author

ndrluis commented Jul 27, 2024

Hi @sungwy,

I got this warning message when I was testing the 0.7.0rc2. I have mixed feelings about not adding this to the final release. It does not raise an error, but this warning is annoying, and the fix is a simple adjustment.

@sungwy
Copy link
Collaborator

sungwy commented Jul 28, 2024

Hi @ndrluis - first off, thank you very much for jumping in and helping test out the 0.7.0 release.

It's a very large release, and I'm sincerely appreciative of all your help in testing and validating this release thoroughly 🙂

I agree with your point in that although the warning is annoying and the fix is simple (and ready to commit! 🏎️ Thank you!), it thankfully isn't a critical issue that impacts the actual function.

Hence, I feel that we should have this fix ready to commit, and bundle them up with either:

  • the next RC if there are critical issues that arise that need to be fixed
  • the next patch release, if there are no such issues that will block the release

tagging @HonahX and @Fokko for awareness

ndrluis added a commit to ndrluis/iceberg-python that referenced this issue Jul 31, 2024
This will help us avoid propagating warnings to our users, as occurred in apache#971.
ndrluis added a commit to ndrluis/iceberg-python that referenced this issue Jul 31, 2024
This will help us avoid propagating warnings to our users, as occurred in apache#971.
ndrluis added a commit to ndrluis/iceberg-python that referenced this issue Jul 31, 2024
This will help us avoid propagating warnings to our users, as occurred in apache#971.
sungwy pushed a commit that referenced this issue Aug 1, 2024
* Treat warning as error in CI/Dev

This will help us avoid propagating warnings to our users, as occurred in #971.

* fixup! Treat warning as error in CI/Dev
sungwy pushed a commit to sungwy/iceberg-python that referenced this issue Dec 7, 2024
* Treat warning as error in CI/Dev

This will help us avoid propagating warnings to our users, as occurred in apache#971.

* fixup! Treat warning as error in CI/Dev
sungwy pushed a commit to sungwy/iceberg-python that referenced this issue Dec 7, 2024
* Treat warning as error in CI/Dev

This will help us avoid propagating warnings to our users, as occurred in apache#971.

* fixup! Treat warning as error in CI/Dev
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants