You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the quantile method raises an exception when it encounters a Dask array.
ifisinstance(self.data, dask_array_type):
raiseTypeError(
"quantile does not work for arrays stored as dask ""arrays. Load the data via .compute() or .load() ""prior to calling this method."
)
I think it's because taking a quantile needs to see all the data in the dimension it's quantile-ing, or blocked/approximate methods weren't on hand when the feature was added. Dask arrays where the dimension being quantile-ed was exactly one chunk in extent seem like a special case where no blocked algorithm is needed.
The problem with following the suggestion of the exception (loading the array into memory) is that "wide and shallow" arrays are too big to load into memory, yet each chunk is statistically independent if the quantile dimension is the "shallow" dimension.
I'm not necessarily proposing delegating to Dask's quantile (unless it's super easy), but wanted to explore this special case described above.
Currently the
quantile
method raises an exception when it encounters a Dask array.I think it's because taking a quantile needs to see all the data in the dimension it's quantile-ing, or blocked/approximate methods weren't on hand when the feature was added. Dask arrays where the dimension being quantile-ed was exactly one chunk in extent seem like a special case where no blocked algorithm is needed.
The problem with following the suggestion of the exception (loading the array into memory) is that "wide and shallow" arrays are too big to load into memory, yet each chunk is statistically independent if the quantile dimension is the "shallow" dimension.
I'm not necessarily proposing delegating to Dask's quantile (unless it's super easy), but wanted to explore this special case described above.
Related links:
Thank you!
EDIT: added stackoverflow link
The text was updated successfully, but these errors were encountered: