Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-119518: Stop interning strings in pathlib #123356

Merged
merged 1 commit into from
Sep 2, 2024
Merged

Conversation

barneygale
Copy link
Contributor

@barneygale barneygale commented Aug 26, 2024

Remove sys.intern(str(x)) calls when normalizing a path in pathlib. This speeds up str(Path('foo/bar')) by about 10%.

As mentioned in the issue, I've never been able to establish if/when string interning is useful in pathlib, nor that we're using it in the best way. It feels negligent to leave it in the code.

Remove `sys.intern(str(x))` calls when normalizing a path in pathlib. This
speeds up `str(Path('foo/bar'))` by about 10%.
Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I am not convinced that interning strings is a good idea according to issues listed in #119518.

@encukou encukou merged commit 5002f17 into python:main Sep 2, 2024
39 checks passed
@barneygale
Copy link
Contributor Author

Thank you both!

@geertj
Copy link

geertj commented Oct 23, 2024

Can this be back ported to 3.13? The interning causes memory usage to grow indefinitely on a free threaded build.

@vstinner
Copy link
Member

Can this be back ported to 3.13? The interning causes memory usage to grow indefinitely on a free threaded build.

IMO it's too late to backport such change. cc @colesbury

@colesbury
Copy link
Contributor

Maybe we could consider something like the following in 3.13:

if sysconfig.get_config_var("Py_GIL_DISABLED"):
    return drv, root, [x for x in rel.split(sep) if x and x != '.']

parsed = [sys.intern(str(x)) for x in rel.split(sep) if x and x != '.']
return drv, root, parsed

In other words, avoid interning in the free-threaded build, but don't change the behavior in the 3.13 default build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Performance or resource usage topic-pathlib
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants