Fix open_mfdataset for list of fsspec files #9785
Conversation
|
mypy needs a fix. Can you expand that logic out to an explicit for-loop please. The comprehension is quite hard to read now. |
|
I added type ignores, mypy should have raised earlier as well since the else clause was incorrect, but type inference was probably a bit smarter here. I really don't know enough about mypy to properly troubleshoot this |
|
The type hints are generally incorrect in open_mfdataset (I think) since the fsspec version doesn't produce a string or os.PathLike |
| paths.append(_normalize_path(p)) | ||
| elif isinstance(p, list): | ||
| paths.append(_normalize_path_list(p)) # type: ignore[arg-type] | ||
| else: |
There was a problem hiding this comment.
Can you tell me what the expected type here is?
The type hints only "allow" str, os.PathLike or a list of these.
There was a problem hiding this comment.
Basically whatever
fs = fsspec.filesystem("file")
fs.open(my_file)
returns
There was a problem hiding this comment.
FYI trying to determine what type fs.open returns could be a massive rabbit hole - see fsspec/filesystem_spec#579 (comment)
There was a problem hiding this comment.
Ok, I guess a minimal protocol similar to how pandas handles it should be the solution then.
Do you know what this returns? Is it some buffered reader or similar? |
|
No, no clue. Nothing is typed over there and there are so many subclasses out there... The class AbstractFileSystem implements the open but again not really clear We typed this as in pandas, but this is very generic. The implementation lives here: https://github.com/pandas-dev/pandas/blob/ee3c18f51b393893ed6e31214c7be2f9427ce0c9/pandas/_typing.py#L270 |
|
OK merging since this fixes a regression. We can followup with typing improvements. |
* main: Add download stats badges (pydata#9786) Fix open_mfdataset for list of fsspec files (pydata#9785) add 'User-Agent'-header to pooch.retrieve (pydata#9782) Optimize `ffill`, `bfill` with dask when `limit` is specified (pydata#9771)
* main: (24 commits) Bump minimum versions (#9796) Namespace-aware `xarray.ufuncs` (#9776) Add prettier and pygrep hooks to pre-commit hooks (#9644) `rolling.construct`: Add `sliding_window_kwargs` to pipe arguments down to `sliding_window_view` (#9720) Bump codecov/codecov-action from 4.6.0 to 5.0.2 in the actions group (#9793) Buffer types (#9787) Add download stats badges (#9786) Fix open_mfdataset for list of fsspec files (#9785) add 'User-Agent'-header to pooch.retrieve (#9782) Optimize `ffill`, `bfill` with dask when `limit` is specified (#9771) fix cf decoding of grid_mapping (#9765) Allow wrapping `np.ndarray` subclasses (#9760) Optimize polyfit (#9766) Use `map_overlap` for rolling reductions with Dask (#9770) fix html repr indexes section (#9768) Bump pypa/gh-action-pypi-publish from 1.11.0 to 1.12.2 in the actions group (#9763) unpin array-api-strict, as issues are resolved upstream (#9762) rewrite the `min_deps_check` script (#9754) CI runs ruff instead of pep8speaks (#9759) Specify copyright holders in main license file (#9756) ...
* main: Bump minimum versions (pydata#9796) Namespace-aware `xarray.ufuncs` (pydata#9776) Add prettier and pygrep hooks to pre-commit hooks (pydata#9644) `rolling.construct`: Add `sliding_window_kwargs` to pipe arguments down to `sliding_window_view` (pydata#9720) Bump codecov/codecov-action from 4.6.0 to 5.0.2 in the actions group (pydata#9793) Buffer types (pydata#9787) Add download stats badges (pydata#9786) Fix open_mfdataset for list of fsspec files (pydata#9785) add 'User-Agent'-header to pooch.retrieve (pydata#9782) Optimize `ffill`, `bfill` with dask when `limit` is specified (pydata#9771) fix cf decoding of grid_mapping (pydata#9765) Allow wrapping `np.ndarray` subclasses (pydata#9760) Optimize polyfit (pydata#9766) Use `map_overlap` for rolling reductions with Dask (pydata#9770)
cc @dcherian