Skip to content

Commit

Permalink
fix: Remove opening file object when validating S3 parquet source (#3217
Browse files Browse the repository at this point in the history
)

* Remove opening the file object

Let pyarrow handle opening the path using the filesystem.

Signed-off-by: Max Z <max.zwiessele@babylonhealth.com>

* fix: linting error

Signed-off-by: Max Z <max.zwiessele@babylonhealth.com>

Signed-off-by: Max Z <max.zwiessele@babylonhealth.com>
  • Loading branch information
mzwiessele authored Sep 16, 2022
1 parent ae37b20 commit a906018
Showing 1 changed file with 1 addition and 3 deletions.
4 changes: 1 addition & 3 deletions sdk/python/feast/infra/offline_stores/file_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -160,9 +160,7 @@ def get_table_column_names_and_types(
if filesystem is None:
schema = ParquetDataset(path).schema.to_arrow_schema()
else:
schema = ParquetDataset(
filesystem.open_input_file(path), filesystem=filesystem
).schema
schema = ParquetDataset(path, filesystem=filesystem).schema

return zip(schema.names, map(str, schema.types))

Expand Down

0 comments on commit a906018

Please sign in to comment.