functools.lru_cache omits __defaults__ attribute from wrapped function

[gpshead]

BPO	44003
Nosy	@rhettinger, @gpshead, @ncoghlan, @serhiy-storchaka
PRs	bpo-44003: expose default args on the wrapped functools.lru_cache function #25800

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-02.04:35:18.116>
created_at = <Date 2021-05-01.19:49:53.450>
labels = ['type-feature', 'library', '3.11']
title = 'functools.lru_cache omits __defaults__ attribute from wrapped function'
updated_at = <Date 2021-05-02.06:04:28.545>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2021-05-02.06:04:28.545>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-02.04:35:18.116>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2021-05-01.19:49:53.450>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 44003
keywords = ['patch', '3.5regression']
message_count = 12.0
messages = ['392623', '392627', '392643', '392644', '392645', '392646', '392647', '392649', '392651', '392652', '392662', '392668']
nosy_count = 4.0
nosy_names = ['rhettinger', 'gregory.p.smith', 'ncoghlan', 'serhiy.storchaka']
pr_nums = ['25800']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue44003'
versions = ['Python 3.11']

[gpshead]

When the C implementation of functools.lru_cache was added in bpo/issue14373, it appears to have omitted setting .__defaults__ on its wrapped function.

Python 3.10.0a7+ (heads/master-dirty:823fbf4e0e, May  1 2021, 11:10:30) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import functools
>>> def func(b=5): pass
... 
>>> @functools.lru_cache
... def cached_func(b=5): pass
... 
>>> func.__defaults__
(5,)
>>> cached_func.__defaults__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'functools._lru_cache_wrapper' object has no attribute '__defaults__'

functools.update_wrapper() does set __defaults__ so this appears to just be an oversight in Modules/_functoolsmodule.c.

[serhiy-storchaka]

Where does functools.update_wrapper() set __defaults__?

[gpshead]

That was anecdotal evidence:

Python 3.9.1 (v3.9.1:1e5d33e9b9, Dec  7 2020, 12:10:52) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def func(arg=1, *, kwarg=2): pass
... 
>>> import functools
>>> from functools import lru_cache, update_wrapper
>>> @lru_cache
... def cached_func(arg=1, *, kwargs=2): pass
... 
>>> def x(*args, **kwargs): func(*args, **kwargs)
... 
>>> updated_x = update_wrapper(func, x)
>>> x
<function x at 0x7feff5892b80>
>>> updated_x
<function x at 0x7feff5828a60>
>>> updated_x.__defaults__
(1,)
>>> updated_x.__kwdefaults__
{'kwarg': 2}
>>> func.__defaults__
(1,)
>>> func.__kwdefaults__
{'kwarg': 2}
>>> cached_func.__defaults__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'functools._lru_cache_wrapper' object has no attribute '__defaults__'
>>> cached_func.__kwdefaults__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'functools._lru_cache_wrapper' object has no attribute '__kwdefaults__'

[gpshead]

the pure python functools.lru_cache doesn't get this right either. here's a desirable testcase for this bug:

    def test_lru_defaults_bug44003(self):
        @self.module.lru_cache(maxsize=None)
        def func(arg='ARG', *, kw='KW'):
            return arg, kw

        self.assertEqual(func.__wrapped__.__defaults__, ('ARG',))
        self.assertEqual(func.__wrapped__.__kwdefaults__, {'kw': 'KW'})
        self.assertEqual(func.__defaults__, ('ARG',))
        self.assertEqual(func.__kwdefaults__, {'kw': 'KW'})

results in

ERROR: test_lru_defaults_bug44003 (test.test_functools.TestLRUC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/greg/oss/python/gpshead/Lib/test/test_functools.py", line 1738, in test_lru_defaults_bug44003
    self.assertEqual(func.__defaults__, ('ARG',))
AttributeError: 'functools._lru_cache_wrapper' object has no attribute '__defaults__'

======================================================================
FAIL: test_lru_defaults_bug44003 (test.test_functools.TestLRUPy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/greg/oss/python/gpshead/Lib/test/test_functools.py", line 1738, in test_lru_defaults_bug44003
    self.assertEqual(func.__defaults__, ('ARG',))
AssertionError: None != ('ARG',)

[rhettinger]

I don't think this should be done. We want the lru_cache to be a pass-through. Applying defaults or keyword-only/positional-only restrictions is the responsibility of the inner function.

FWIW, here are the fields that Nick selected to be included in update_wrapper(): ('__module__', '__name__', '__qualname__', '__doc__', '__annotations__').

Those are sufficient to get help() to work which is all we were aiming for:

>>> from functools import *
>>> @lru_cache
def cached_func(b=5):
    pass

>>> help(cached_func)
Help on _lru_cache_wrapper in module __main__:

cached_func(b=5)

[gpshead]

__defaults__ and __kwdefaults__ get used for code introspection. Just as __annotations__ does. __annotations__ is already available on the lru_cache wrapped function. All of those seem to go together from a runtime inspection point of view.

[gpshead]

An inner function can't know if somebody else might want to inspect it.

This is a decorator that does not change anything about the argument signature of the wrapped function, carrying over the reference to meta-information about that by default seems to make sense.

[gpshead]

https://bugs.python.org/issue41232 covers the more general case of suggesting changing update_wrapper's behavior. That would alleviate the need to fix the pure python implementation within my own PR.

[rhettinger]

I don't really like it. Carrying forward these attributes isn't the norm for wrapping functions.

The __defaults__ argument is normally only used where it has an effect rather than in a wrapper where it doesn't. Given that it is mutable, it invites a change that won't work. For example:

    >>> def pow(base, exp=2):
        return base ** exp

    >>> pow.__defaults__
    (2,)
    >>> pow.__defaults__ = (3,)
    >>> pow(2)                     
    8

Also, an introspection function can only meaningfully use defaults when accompanied by the names of the fields:

    >>> pow.__code__.co_varnames
    ('base', 'exp')

However, these aren't visible by directly introspecting the wrapper.

FWIW, we've never had a user reported issue regarding the absence of __defaults__. If ain't broke, let's don't "fix" it.

Nick and Serhiy, any thoughts?

[gpshead]

Oh, I didn't realize mutating those would actually change the code runtime behavior. But it makes sense, those are needed before the code object is entered.

Yeah that is different, and suggests making this the default is not actually desired. (this issue and the other one)

I guess our rule is that introspection code really must check for and be ready to handle .__wrapped__ if its goal is robustness?

[gpshead]

rejecting. code trying to make direct use of __defaults__ is likely better off using inspect.signature(). there might be an issue with inspect in some cases (https://bugs.python.org/issue41232) but I do not believe that is true for lru_cache wrapped things.

[serhiy-storchaka]

I meant that I looked up the code of functools.update_wrapper() and did not see that it sets the __defaults__ attribute.

In your example in msg392643 you use functools.update_wrapper() incorrectly. The first argument is wrapper, and the second argument is wrapped. updated_x is func.