-
-
Notifications
You must be signed in to change notification settings - Fork 30.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gh-119127: functools.partial placeholders #119827
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I personally would advocate for (see #119827 (comment))PLACEHOLDER
instead of Placeholder
to stress that 1) this is not a global constant, just a module constant, 2) this is not a class, 3) this is named similarly to dataclasses.KW_ONLY
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for more nitpicks.
Not very convenient to match exact exception message string, which, at least from my experience, is more common case. #123013 |
I just looked at the Placeholder singleton logic and think it is fine because it mirrors the None singleton. It is a little complicated but not in a way that will impair maintenance. |
|
|
@dg-pb Congratulations on landing a new feature. |
Thank you @rhettinger. And thank you everyone for your help. Especially @serhiy-storchaka. I hope this will prove to be a useful feature. Was pleasure working with you all. |
I'm a very often user of |
As I already had implementation I though PR might be helpful for others to see and evaluate.
From all the different extensions of
functools.partial
I think this one is the best. It is relatively simple and exposes all missing functionality. Otherpartial
extensions that I have seen lack functionality and would not provide complete argument ordering capabilities and/or are too complicated in relation to what they offer.Implementation can be summarised as follows:
a) Trailing placeholders are not allowed. (Makes things simpler)
b) Throws exception if not all placeholders are filled on call
c) retains optimization benefits of application on other
partial
instances.Performance penalty compared to current
functools.partial
is minimal for extension class. + 20-30 ns for initialisation and <4 ns when called with or without placeholders.To put it simply, new functionality extends
functools.partial
so that it has flexibility oflambda
/def
approach (in terms of argument ordering), but call overhead is 2x smaller.The way I see it is that this could only be justified if this extension provided completeness and no new functionality is going to be needed anywhere near in the future. I have thought about it and tried various alternatives and I think there is a good chance that this is the case. Personally, I don't think I would ever need anything more from
partial
class.Current implementation functions reliably.
Benchmark
There is nothing new here in terms of performance. The performance after this PR will be (almost) the same as the performance of
partial
until now.Placeholders
only provide flexibility for taking advantage of performance benefits where it is important.So far I have identified 2 such cases:
operator
module. This allows for new strategies in making performantiterator
recipes.Partializing
input target function. Examples of this are optimizers and similar. I.e. cases where the function will be called over and over within the routine with number of arguments. But the input target function needs partial substitution for positionals and keywords.Good example of this is
scipy.optimize.minimize
.Its signature is:
scipy.optimize.minimize(fun, x0, args=(), ...)
Note, it does not have
kwds
. Why? I don't know. But good reason for it could be:will need to expand
**kwds
on every call (even if it is empty), whilepartial
will make the most optimal call. (see benchmarks below). So theminimize
function can leave outkwds
given there is a good way to source callable with already substituted keywords.This extension allows pre-substituting both positionals and keywords. This allows optimizer signature to leave out both
kwds
andargs
resulting in simpler interfacescipy.optimize.minimize(fun, x0, ...)
and gaining slightly better performance - function calls are at the center of such problems after all.Benchmark Results for
__call__
Code for Cases
CPython Results
PyPy Results
Setup:
lambda
otherpartial
.CPython:
__call__
. Run times are very close of current and new version with Placeholders.pos2kw2_kwe
(emptykwds
) is much faster ofpartial
call.pos2kw2_kw
(non-emptykwds
) is currently slower, however gh-119109: functool.partial vectorcall supports pto->kw & fallback to tp_call removed #120783 will likely to improve its speed so that it outperforms lambda.PyPy:
Placeholders
results in very poor performance. However, this has no material implication aslambda
is more performant thanpartial
in all cases and is an optimal choice.Benchmark Results for
__new__
To sum up
This extension:
partial
to few more important (at least from my POV) cases.lambda/def
behaviour. Thus, allowingpartial
to be used forpartialmethod
application which allows for some simplifications in handling these in other parts of the library - i.e.inspect
.📚 Documentation preview 📚: https://cpython-previews--119827.org.readthedocs.build/