-
-
Notifications
You must be signed in to change notification settings - Fork 30.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gh-87729: improve hit rate of LOAD_SUPER_ATTR specialization #104270
Conversation
* main: (35 commits) pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304) pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441) pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096) pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217) pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312) pythongh-104240: return code unit metadata from codegen (python#104300) pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277) pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298) pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320) require-pr-label.yml: Add missing "permissions:" (python#104309) pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939) pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181) pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188) pythonGH-104308: socket.getnameinfo should release the GIL (python#104307) pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311) pythongh-99113: A Per-Interpreter GIL! (pythongh-104210) pythonGH-104284: Fix documentation gettext build (python#104296) pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251) pythongh-104223: Fix issues with inheriting from buffer classes (python#104227) pythongh-99108: fix typo in Modules/Setup (python#104293) ...
* main: pythonGH-102181: Improve specialization stats for SEND (pythonGH-102182) pythongh-103000: Optimise `dataclasses.asdict` for the common case (python#104364) pythongh-103538: Remove unused TK_AQUA code (pythonGH-103539) pythonGH-87695: Fix OSError from `pathlib.Path.glob()` (pythonGH-104292) pythongh-104263: Rely on Py_NAN and introduce Py_INFINITY (pythonGH-104202) pythongh-104010: Separate and improve docs for `typing.get_origin` and `typing.get_args` (python#104013) pythongh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic (python#104355) pythongh-103960: Dark mode: invert image brightness (python#103983) pythongh-104252: Immortalize Py_EMPTY_KEYS (pythongh-104253) pythongh-101819: Clean up _io windows console io after pythongh-104197 (python#104354) pythongh-101819: Harden _io init (python#104352) pythongh-103247: clear the module cache in a test in test_importlib/extensions/test_loader.py (pythonGH-104226) pythongh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format (python#103849) pythongh-74895: adjust tests to work on Solaris (python#104326) pythongh-101819: Refactor _io in preparation for module isolation (python#104334) pythongh-90953: Don't use deprecated AST nodes in clinic.py (python#104322) pythongh-102327: Extend docs for "url" and "headers" parameters to HTTPConnection.request() pythongh-104328: Fix typo in ``typing.Generic`` multiple inheritance error message (python#104335)
* main: pythongh-87849: fix SEND specialization family definition (pythonGH-104268) pythongh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic (python#104384) pythongh-101819: Adapt _io._Buffered* methods to Argument Clinic (python#104367) pythongh-101819: Refactor `_io` futher in preparation for module isolation (python#104369) pythongh-101819: Adapt _io.TextIOBase methods to Argument Clinic (python#104383) pythongh-101117: Improve accuracy of sqlite3.Cursor.rowcount docs (python#104287) pythonGH-92184: Convert os.altsep to '/' in filenames when creating ZipInfo objects (python#92185) pythongh-104357: fix inlined comprehensions that close over iteration var (python#104368) pythonGH-90208: Suppress OSError exceptions from `pathlib.Path.glob()` (pythonGH-104141)
I think this is fine for 3.12. For 3.13 we will probably want something like we had before. With larger regions of optimization which handle polymorphism through code duplication prior to specialization, we will want the information about what the value of |
Yes, I agree; once we have a reasonable way to handle specialization in polymorphic cases, we'll want to reintroduce this. But for now I don't think we can keep it; it results in too many failed specializations. |
* main: pythongh-104057: Fix direct invocation of test_support (pythonGH-104069) pythongh-87729: improve hit rate of LOAD_SUPER_ATTR specialization (python#104270) pythongh-101819: Fix inverted debug preprocessor check in winconsoleio.c (python#104388)
@carljm FYI I've identified this commit as the source of a backwards incompatibility in Rubicon-Objc. I'm trying to narrow down a simpler reproduction case. |
We observed that in stats collection on pyperformance (+ pyston macro-benchmarks), the hit rate of the
LOAD_SUPER_ATTR
specialization toLOAD_SUPER_ATTR_METHOD
was not as good as expected. Investigation revealed a few causes for this.It is not uncommon to have super method calls that use argument unpacking (e.g.
super().method(*args, **kwargs)
) and so need to use e.g.CALL_FUNCTION_EX
. This fails the checks for the load-method optimization in the compiler, so it emits aLOAD_SUPER_ATTR
without the load-method optimization.Loading things that are not actually method descriptors via a call that looks like
super().method()
is also not uncommon. E.g.super().__new__()
where the base class is e.g. builtinstr
.The type of
self
at a particular call-site ofsuper()
is commonly polymorphic, whenever the inheritance chain is longer than length 2. E.g. ifC
inheritsB
inheritsA
, then a call tosuper()
in a method ofB
may commonly seeself
of type eitherB
orC
.The fixes to (1) and (2) are simple enough: we should also specialize for
LOAD_SUPER_ATTR_ATTR
.(3) is a much trickier problem, since it means we can't inline-cache the results of a
super()
lookup at all. The results of asuper()
lookup are dependent on the type ofself
, and we can't effectively specialize for the type ofself
(barring polymorphic inline caching, which we don't generally do and would make for a much large inline cache.)This means that specialization of
LOAD_SUPER_ATTR
is just splitting out the "shadowed global super" case from the load-method and attr cases. This still preserves most of the performance improvement fromLOAD_SUPER_ATTR
, which always came from avoiding allocation of single-usesuper
objects.With this change, in my tests the hit rate of
LOAD_SUPER_ATTR
specialization in pyperformance is 100%. (Effectively this just means we don't have a case in pyperformance of shadowing the namesuper
.)