-
-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
REF: BlockManager.apply_allow_failures #34714
Labels
Milestone
Comments
jbrockmendel
added
Bug
Needs Triage
Issue that has not been reviewed by a pandas team member
labels
Jun 11, 2020
I don’t know about deprecating. I think I tried to remove this pattern from GroupBy in the past and it killed performance. Would be on board with consolidating for sure though
…Sent from my iPhone
On Jun 11, 2020, at 10:44 AM, jbrockmendel ***@***.***> wrote:
cc @mroeschke w/r/t rolling ._apply, @WillAyd w/r/t _cython_agg_blocks, and anyone else w/r/t DataFrame._reducewith numeric_only=None, apply.FrameApply with ignore_failures=True.
All of these do something along the lines of:
results = []
exclude = []
for i, block in enumerate(mgr.blocks):
try:
res = func(block.values)
results.append(res)
except:
exclude.append(i)
out = reconstruct(results, exclude)
Two things should be done with this pattern:
Deprecate it, since it is a disproportionate maintainence burden
implement something like BlockManager.apply_with_ignore_failures that would resemble BlockManager.apply but with the try/except logic*
* We could add that logic into BlockManager.apply, but BM.apply assumes the output has the same shape as the original frame, which I don't think holds for the cases mentioned above, so that would be a little more invasive than just adding a try/except.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
FWIW, if we're open to allowing numba in the internals (with some opt-in, global execution engine setting), this is a great candidate for numba parallelism and can ameliorate the performance hit if we decide to deprecate this behavior. |
TomAugspurger
removed
the
Needs Triage
Issue that has not been reviewed by a pandas team member
label
Jun 11, 2020
jbrockmendel
added
Internals
Related to non-user accessible pandas implementation
Refactor
Internal refactoring of code
labels
Jun 13, 2020
This was referenced Jul 14, 2020
This was referenced Aug 14, 2020
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
cc @mroeschke w/r/t rolling
._apply
, @WillAyd w/r/t_cython_agg_blocks
, and anyone else w/r/tDataFrame._reduce
withnumeric_only=None
,apply.FrameApply
withignore_failures=True
.All of these do something along the lines of:
Two things should be done with this pattern:
BlockManager.apply_with_ignore_failures
that would resemble BlockManager.apply but with the try/except logic** We could add that logic into BlockManager.apply, but BM.apply assumes the output has the same shape as the original frame, which I don't think holds for the cases mentioned above, so that would be a little more invasive than just adding a try/except.
The text was updated successfully, but these errors were encountered: