Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Isolate the itertools extension module #101277

Closed
erlend-aasland opened this issue Jan 24, 2023 · 3 comments
Closed

Isolate the itertools extension module #101277

erlend-aasland opened this issue Jan 24, 2023 · 3 comments
Assignees
@erlend-aasland
Labels
type-feature A feature request or enhancement

Comments

@erlend-aasland
Copy link
Contributor

erlend-aasland commented Jan 24, 2023 •
edited by bedevere-bot
Loading

Feature or enhancement

Following https://github.com/ericsnowcurrently/multi-core-python/wiki/0-The-Plan we need to convert the itertools extension module to use module state.

There are multiple static global type objects:

cpython/Tools/c-analyzer/cpython/globals-to-fix.tsv

Lines 338 to 359 in e244401

Modules/itertoolsmodule.c - _grouper_type -
Modules/itertoolsmodule.c - accumulate_type -
Modules/itertoolsmodule.c - batched_type -
Modules/itertoolsmodule.c - chain_type -
Modules/itertoolsmodule.c - combinations_type -
Modules/itertoolsmodule.c - compress_type -
Modules/itertoolsmodule.c - count_type -
Modules/itertoolsmodule.c - cwr_type -
Modules/itertoolsmodule.c - cycle_type -
Modules/itertoolsmodule.c - dropwhile_type -
Modules/itertoolsmodule.c - filterfalse_type -
Modules/itertoolsmodule.c - groupby_type -
Modules/itertoolsmodule.c - islice_type -
Modules/itertoolsmodule.c - pairwise_type -
Modules/itertoolsmodule.c - permutations_type -
Modules/itertoolsmodule.c - product_type -
Modules/itertoolsmodule.c - repeat_type -
Modules/itertoolsmodule.c - starmap_type -
Modules/itertoolsmodule.c - takewhile_type -
Modules/itertoolsmodule.c - tee_type -
Modules/itertoolsmodule.c - teedataobject_type -
Modules/itertoolsmodule.c - ziplongest_type -

We need to convert these to heap types, add module state, and implement multi-phase init.

See also PEP-687.

I've got an old PR (#24065) that I'm planning on resurrecting. I'll re-submit it as multiple PRs; this is going to be a large change.

For this particular module, it could make sense to add a module state pointer to each type context, for easy and cheap state access.

Linked PRs
@erlend-aasland erlend-aasland added the type-feature A feature request or enhancement label Jan 24, 2023
@erlend-aasland erlend-aasland self-assigned this Jan 24, 2023
@sobolevn
Copy link
Member

cc @rhettinger as the module owner.

@bedevere-bot bedevere-bot mentioned this issue Jan 24, 2023
Merged
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: establish global state
99406f8
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add groupby and _grouper types to module state
50410bf
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add cycle type to module state
98636d2
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add dropwhile type to module state
54c43a4
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add takewhile type to module state
2ede7ad
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add starmap type to module state
aca9ef5
@bedevere-bot bedevere-bot mentioned this issue Jan 24, 2023
Merged
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add combinations and cwr types to module state
64be5ab
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add permutations type to module state
8a063a7
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add accumulate type to module state
19d43d8
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add compress type to module state
446b751
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add count type to module state
1d2d93c
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add pairwise type to module state
38bf6e5
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add ziplongest type to module state
19ca7b1
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add product type to module state
1c3828d
@bedevere-bot bedevere-bot mentioned this issue Jan 24, 2023
Merged
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add repeat type to module state
a3c78ad
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add islice type to module state
4a30327
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add chain type to module state
27ee72c
@bedevere-bot bedevere-bot mentioned this issue Jan 24, 2023
Merged
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add tee type to module state
343abef
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 24, 2023
@erlend-aasland
pythongh-101277: Add tee data object type to module state
ff77cab
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 25, 2023
@erlend-aasland
pythongh-101277: Add accumulate type to module state
89b8e34
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 25, 2023
@erlend-aasland
pythongh-101277: Add compress type to module state
04e8c29
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 25, 2023
@erlend-aasland
pythongh-101277: Add count type to module state
1e0c23d
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Jan 25, 2023
@erlend-aasland
pythongh-101277: Add pairwise type to module state
bbfc0f3
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 1, 2023
@erlend-aasland
pythongh-101277: Add compress type to module state
4784128
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 1, 2023
@erlend-aasland
pythongh-101277: Add count type to module state
8708e94
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 1, 2023
@erlend-aasland
pythongh-101277: Add pairwise type to module state
783e0f2
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 1, 2023
@erlend-aasland
pythongh-101277: Add ziplongest type to module state
336ca9b
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 1, 2023
@erlend-aasland
pythongh-101277: Add product type to module state
cfa334f
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 1, 2023
@erlend-aasland
pythongh-101277: Add repeat type to module state
b64540c
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 1, 2023
@erlend-aasland
pythongh-101277: Add islice type to module state
50316a7
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 1, 2023
@erlend-aasland
pythongh-101277: Add chain type to module state
da955ae
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 1, 2023
@erlend-aasland
pythongh-101277: Add tee type to module state
75bb5da
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 1, 2023
@erlend-aasland
pythongh-101277: Add tee data object type to module state
b7c21be
carljm added a commit to carljm/cpython that referenced this issue Feb 1, 2023
@carljm
Merge branch 'main' into inlinecomp2
8b76051 
* main:
  pythongh-98831: rewrite PUSH_EXC_INFO and conditional jumps in the instruction definition DSL (python#101481)
  pythongh-98831: Modernize the LOAD_ATTR family (python#101488)
  pythongh-101498 : Fix asyncio.Timeout example in docs (python#101499)
  pythongh-101454: fix documentation for END_ASYNC_FOR (python#101455)
  pythongh-101277: Isolate itertools, add group and _grouper types to module state (python#101302)
  pythongh-101317: Add `ssl_shutdown_timeout` parameter for `asyncio.StreamWriter.start_tls` (python#101335)
  datetime.rst: fix combine() signature (python#101490)
kumaraditya303 pushed a commit that referenced this issue Feb 3, 2023
@erlend-aasland
gh-101277: Port more itertools static types to heap types (#101303)
a52cc98 
Add dropwhile, takewhile, starmap, combinations*, and permutations types to module state.
iritkatriel pushed a commit to iritkatriel/cpython that referenced this issue Feb 3, 2023
@erlend-aasland @iritkatriel
pythongh-101277: Port more itertools static types to heap types (py…
fef6510 
…thon#101303)

Add dropwhile, takewhile, starmap, combinations*, and permutations types to module state.
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 3, 2023
@erlend-aasland
pythongh-101277: Add accumulate type to module state
6d5d4d2
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 3, 2023
@erlend-aasland
pythongh-101277: Add compress type to module state
e42e834
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 3, 2023
@erlend-aasland
pythongh-101277: Add count type to module state
ecb447f
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 3, 2023
@erlend-aasland
pythongh-101277: Add pairwise type to module state
fb0e18d
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 3, 2023
@erlend-aasland
pythongh-101277: Add ziplongest type to module state
5a27550
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 3, 2023
@erlend-aasland
pythongh-101277: Add product type to module state
8baf0e1
@rhettinger rhettinger mentioned this issue Feb 5, 2023
Closed
erlend-aasland added a commit that referenced this issue Feb 8, 2023
@erlend-aasland
gh-101277: Port more itertools static types to heap types (#101304)
de3669e 
Add accumulate, compress, count, filterfalse, pairwise, product,
and zip_longest types to module state.
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 8, 2023
@erlend-aasland
pythongh-101277: Add repeat type to module state
9a4b14c
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 8, 2023
@erlend-aasland
pythongh-101277: Add islice type to module state
a66b888
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 8, 2023
@erlend-aasland
pythongh-101277: Add chain type to module state
d18169b
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 8, 2023
@erlend-aasland
pythongh-101277: Add tee type to module state
e574de1
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Feb 8, 2023
@erlend-aasland
pythongh-101277: Add tee data object type to module state
053b759
carljm added a commit to carljm/cpython that referenced this issue Feb 9, 2023
@carljm
Merge branch 'main' into inlinecomp2
be3becc 
* main: (82 commits)
  pythongh-101670: typo fix in PyImport_ExtendInittab() (python#101723)
  pythonGH-99293: Document that `Py_TPFLAGS_VALID_VERSION_TAG` shouldn't be used. (#pythonGH-101736)
  no-issue: Add Dong-hee Na as the cjkcodecs codeowner (pythongh-101731)
  pythongh-101678: Merge math_1_to_whatever() and math_1() (python#101730)
  pythongh-101678: refactor the math module to use special functions from c11 (pythonGH-101679)
  pythongh-85984: Remove legacy Lib/pty.py code. (python#92365)
  pythongh-98831: Use opcode metadata for stack_effect() (python#101704)
  pythongh-101283: Version was just released, so should be changed in 3.11.3 (pythonGH-101719)
  pythongh-101283: Fix use of unbound variable (pythonGH-101712)
  pythongh-101283: Improved fallback logic for subprocess with shell=True on Windows (pythonGH-101286)
  pythongh-101277: Port more itertools static types to heap types (python#101304)
  pythongh-98831: Modernize CALL and family (python#101508)
  pythonGH-101696: invalidate type version tag in `_PyStaticType_Dealloc` (python#101697)
  pythongh-100221: Fix creating dirs in `make sharedinstall` (pythonGH-100329)
  pythongh-101670: typo fix in PyImport_AppendInittab() (pythonGH-101672)
  pythongh-101196: Make isdir/isfile/exists faster on Windows (pythonGH-101324)
  pythongh-101614: Don't treat python3_d.dll as a Python DLL when checking extension modules for incompatibility (pythonGH-101615)
  pythongh-100933: Improve `check_element` helper in `test_xml_etree` (python#100934)
  pythonGH-101578: Normalize the current exception (pythonGH-101607)
  pythongh-47937: Note that Popen attributes are read-only (python#93070)
  ...
miss-islington pushed a commit that referenced this issue Feb 10, 2023
@erlend-aasland
gh-101277: Finalise isolating itertools (GH-101305)
826bf0e 
Add repeat, islice, chain, tee, teedataobject, and batched types to module state.

Automerge-Triggered-By: GH:erlend-aasland
carljm added a commit to carljm/cpython that referenced this issue Feb 10, 2023
@carljm
Merge branch 'main' into inlinecomp2
36b2917 
* main:
  Docs: Fix getstatus() -> getcode() typos (python#101296)
  Docs: use parameter list for sqlite3.Cursor.execute* (python#101782)
  pythongh-101763: Update bundled copy of libffi to 3.4.4 on Windows (pythonGH-101784)
  pythongh-101517: make bdb avoid looking up in linecache with lineno=None (python#101787)
  pythongh-101759: Update Windows installer to SQLite 3.40.1 (python#101762)
  pythongh-101277: Finalise isolating itertools (pythonGH-101305)
  pythongh-101759: Update macOS installer to SQLite 3.40.1 (python#101761)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@erlend-aasland erlend-aasland

Labels
type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sobolevn @erlend-aasland