Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: Fix warnings from pytest #6703

Closed
wants to merge 8 commits into from
+11 −0
Closed

Python: Fix warnings from pytest #6703

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
56c80ee
Python: Fix warnings from PyLint
Fokko Jan 30, 2023
2506fd3
Update python/pyproject.toml
Fokko Jan 30, 2023
42024ed
Add `.close()`
Fokko Feb 2, 2023
af4a849
Merge branch 'fd-fix-warnings' of github.com:Fokko/incubator-iceberg …
Fokko Feb 2, 2023
9d314ce
Merge branch 'master' of github.com:apache/iceberg into fd-fix-warnings
Fokko Mar 12, 2023
b4db336
Merge branch 'master' of github.com:apache/iceberg into fd-fix-warnings
Fokko Apr 11, 2023
205ff5f
Merge branch 'master' of github.com:apache/iceberg into fd-fix-warnings
Fokko Aug 15, 2023
d7e1708
Update
Fokko Aug 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,10 @@ markers = [
"s3: marks a test as requiring access to s3 compliant storage (use with --aws-access-key-id, --aws-secret-access-key, and --endpoint-url args)",
"adlfs: marks a test as requiring access to adlfs compliant storage (use with --adlfs.account-name, --adlfs.account-key, and --adlfs.endpoint args)"
]
# Turns a warning into an error
filterwarnings = [
"error"
]

[tool.black]
line-length = 130
Expand Down
13 changes: 12 additions & 1 deletion python/tests/catalog/test_glue.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,18 @@ def get_random_databases(n: int) -> Set[str]:

@pytest.fixture(name="_bucket_initialize")
def fixture_s3_bucket(_s3) -> None: # type: ignore
_s3.create_bucket(Bucket=BUCKET_NAME)
bucket = _s3.create_bucket(Bucket=BUCKET_NAME)
yield bucket
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this change the return type?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this is a bit of an interesting fixture. It didn't return anything before, and now it just blocks the yield until the test is passed, and then cleans it up.


response = _s3.list_objects_v2(
Bucket=BUCKET_NAME,
)
while response["KeyCount"] > 0:
_s3.delete_objects(Bucket=BUCKET_NAME, Delete={"Objects": [{"Key": obj["Key"]} for obj in response["Contents"]]})
Copy link
Contributor

@amogh-jahagirdar amogh-jahagirdar Jan 30, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't checked the details of this test, but delete_objects can delete at max 1000 keys for a given bucket. We're probably not creating close to that much here which is why this is passing, but just so the logic is robust we probably want to handle that (if not here, we can do in a separate PR as well)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also no expert on the subject, but it looks like the max number of key returned is also 1000:
image
From: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects_v2

This will iterate until KeyCount is zero

response = _s3.list_objects_v2(
Bucket=BUCKET_NAME,
)
_s3.delete_bucket(Bucket=BUCKET_NAME)


@mock_glue
Expand Down
1 change: 1 addition & 0 deletions python/tests/test_transforms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,7 @@ def test_void_transform() -> None:

class TestType(IcebergBaseModel):
__root__: Transform[Any, Any]
__test__ = False
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this fixed in the other PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that was a PR to the 0.3.0 branch, so we can create another RC from that branch without having the latest pyarrow changes released along with it.



def test_bucket_transform_serialize() -> None:
Expand Down