Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-90699: fix ref counting of static immortal strings #94850

Merged
merged 4 commits into from
Jul 20, 2022

Conversation

kumaraditya303
Copy link
Contributor

@kumaraditya303 kumaraditya303 commented Jul 14, 2022

Without this they can cause negative refcount in _Py_RefTotal.

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @kumaraditya303 for commit f530250 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 14, 2022
@kumaraditya303 kumaraditya303 added the needs backport to 3.11 only security fixes label Jul 14, 2022
@corona10
Copy link
Member

corona10 commented Jul 14, 2022

Thanks for the work, Would you like to provide a reproducible way for detecting leaks?

@kumaraditya303
Copy link
Contributor Author

Thanks for the work, Would you like to provide a reproducible way for detecting leaks?

The following script causes negative ref count:

for i in range(1000):
    a = repr(False)

Output:

@kumaraditya303 ➜ /workspaces/cpython (main ✗) $ ./python -X showrefcount main.py 
[-1001 refs, 0 blocks]

@kumaraditya303
Copy link
Contributor Author

@corona10: The following script causes negative refcount with TextIOWrapper:

# main.py
from io import TextIOWrapper

a = TextIOWrapper(open('main.py','rb'))
for i in a:
    pass
a.close()

Output:

@kumaraditya303 ➜ /workspaces/cpython (main ✗) $ ./python -X showrefcount main.py 
[-2 refs, 0 blocks]

Copy link
Member

@corona10 corona10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

I reproduced the issue with the attached scripts and issues are solved by @kumaraditya303 's patch :)

Thanks for the work!

Copy link
Member

@corona10 corona10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kumaraditya303

Sorry Kumar, Looks like leaks exist through this patch.
PTAL

With patch


beginning 6 repetitions
123456
......
test_io leaked [2, 2, 2] references, sum=6
test_io failed (reference leak) in 3 min 23 sec

== Tests result: FAILURE ==

1 test failed:
    test_io

Total duration: 3 min 23 sec
Tests result: FAILURE

Without Patch


Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 2.90 Run tests sequentially
0:00:00 load avg: 2.90 [1/1] test_io
beginning 6 repetitions
123456
......
test_io passed in 3 min 23 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 3 min 23 sec
Tests result: SUCCESS

@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@kumaraditya303
Copy link
Contributor Author

Looking, I wonder why only two bots failed if there are refleaks.

@kumaraditya303
Copy link
Contributor Author

It is also leaking at my local environment :(

Yeah, I am also able to reproduce, bisecting which one is causing leak

@kumaraditya303
Copy link
Contributor Author

I am getting refleak on main branch too, test.test_io.CTextIOWrapperTest.test_reconfigure_locale is failing.

@kumaraditya303
Copy link
Contributor Author

kumaraditya303 commented Jul 15, 2022

Refleak was fixed by #94858

@kumaraditya303 kumaraditya303 added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 18, 2022
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @kumaraditya303 for commit 0a76abd 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 18, 2022
@kumaraditya303
Copy link
Contributor Author

Refleak buildbots are failing on main too, see #94979

Copy link
Member

@corona10 corona10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Thank you, Kumar!

@corona10 corona10 merged commit 1834133 into python:main Jul 20, 2022
@miss-islington
Copy link
Contributor

Thanks @kumaraditya303 for the PR, and @corona10 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 20, 2022
…-94850)

(cherry picked from commit 1834133)

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
@bedevere-bot
Copy link

GH-95037 is a backport of this pull request to the 3.11 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Jul 20, 2022
@kumaraditya303 kumaraditya303 deleted the cached-strs branch July 20, 2022 06:28
@kumaraditya303
Copy link
Contributor Author

Thanks for the review @corona10!

miss-islington added a commit that referenced this pull request Jul 20, 2022
(cherry picked from commit 1834133)

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants