Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sys.setrecursionlimit docs are incorrect in 3.12 and 3.13 #112282

Open
dimpase opened this issue Nov 20, 2023 · 1 comment
Open

sys.setrecursionlimit docs are incorrect in 3.12 and 3.13 #112282

dimpase opened this issue Nov 20, 2023 · 1 comment
Assignees
Labels
docs Documentation in the Doc dir

Comments

@dimpase
Copy link
Contributor

dimpase commented Nov 20, 2023

Since 3.12, sys.setrecursionlimit does not control the recursion limit of the C stack (it's hardcoded in Py_C_RECURSION_LIMIT, due to #91079), while the 3.13 docs say

Set the maximum depth of the Python interpreter stack to limit. 
This limit prevents infinite recursion from causing an overflow of the C stack and crashing Python.

The fallout can be seen in #112215 (basically, any code that uses lru_cache in combination with sys.setrecursionlimit is potentially, or actually, broken in 3.12, while still working in 3.11) or in #107263 (which was papered over with an increase of Py_C_RECURSION_LIMIT, not really fixed).

This is an obvious change of behaviour, as code working in 3.11 can easily be broken in 3.12, as found in #112215

import sys
sys.setrecursionlimit(1001)
from functools import cache
@cache
def fib(n):
    if n<1: return 0
    if n==1: return 1
    return fib(n-1) + fib(n-2)

print(fib(500))
$ python3.11 <fib.py 
139423224561697880139724382870407283950070256587697307264108962948325571622863290691557658876222521294125
$ python3.12 <fib.py 
Traceback (most recent call last):
  File "<stdin>", line 15, in <module>
  File "<stdin>", line 13, in fib
  File "<stdin>", line 13, in fib
  File "<stdin>", line 13, in fib
  [Previous line repeated 496 more times]
RecursionError: maximum recursion depth exceeded

Such changes should carry a warning in the documentation.

Potentially, any recursive Python code combining C recursion (in a module, e.g. lru_cache, via @lru_cache decorator) with increased (beyond the hardcoded Py_C_RECURSION_LIMIT) sys.setrecursionlimit is bound to be broken in 3.12.

I believe the whole hardcoding of Py_C_RECURSION_LIMIT has to be fixed (perhaps with a separate API provided to change it), as it's a regression and an incompatible change of code behaviour, but meanwhile the docs have to reflect the reality.

@Jiehong
Copy link

Jiehong commented Oct 9, 2024

I confirm this breaks some code out there.

In particular, we are unable to move from python 3.11 to anything higher, since we can no longer pickle.dump our objects, even if sys.setrecursionlimit is called beforehand for example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Documentation in the Doc dir
Projects
None yet
Development

No branches or pull requests

3 participants