Wrong type of module.__path__?

[kytta]

Environment data

• Language Server version: 2021.3.2
• OS and version: darwin x64 (macOS 10.15.7. Catalina)
• Python version (and distribution if applicable, e.g. Anaconda): 3.9.2
• python.analysis.indexing: undefined
• python.analysis.typeCheckingMode: strict

Expected behaviour

Pylance treats __path__ as an Iterable[str] (it should be treated as such according to the Python docs) and doesn't show an error when trying to assign the result of pkgutil.extend_path to it.

Actual behaviour

Pylance says the type of __path__ is list[str] and shows an error saying

__path__: list[str]
Expression of type "Iterable[str]" cannot be assigned to declared type "list[str]"
  "Iterable[str]" is incompatible with "list[str]"PylancereportGeneralTypeIssues

Logs

Python Language Server Log

[Info  - 12:47:50] Pylance language server 2021.3.2 (pyright 85309906) starting
[Info  - 12:47:50] Server root directory: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist
[Info  - 12:47:51] Background analysis(1) root directory: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist
[Info  - 12:47:51] Background analysis(1) started
[Info  - 12:47:51] No configuration file found.
[Info  - 12:47:51] Setting pythonPath for service "stw-on-mensa-bot": "/Users/nick/Library/Caches/pypoetry/virtualenvs/stw-on-mensa-bot-UafpWPe7-py3.9/bin/python"
[Error - 12:47:51] venvPath /Users/nick/dev/stw-on-mensa-bot/~/Library/Caches/pypoetry/virtualenvs is not a valid directory.
[Error - 12:47:51] stubPath /Users/nick/dev/stw-on-mensa-bot/typings is not a valid directory.
[Info  - 12:47:51] Assuming Python version 3.9
[Info  - 12:47:51] Assuming Python platform Darwin
[Info  - 12:47:51] Searching for source files
[Info  - 12:47:51] Found 3 source files
Background analysis message: setConfigOptions
Background analysis message: setTrackedFiles
Background analysis message: markAllFilesDirty
Background analysis message: ensurePartialStubPackages
[FG] parsing: /Users/nick/dev/stw-on-mensa-bot/mensabot/__init__.py (22ms)
[FG] parsing: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 3ms] (100ms)
[FG] binding: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stdlib/builtins.pyi (41ms)
[FG] binding: /Users/nick/dev/stw-on-mensa-bot/mensabot/__init__.py (1ms)
Background analysis message: setFileOpened
IntelliCode model /Users/nick/.vscode/extensions/visualstudioexptteam.vscodeintellicode-1.2.12/cache/E61945A9A512ED5E1A3EE3F1A2365B88F8FE_E4E9EADA96734F01970E616FAB2FAC19
Loading ONNX runtime...
Loaded ONNX runtime. Creating IntelliCode session...
2021-03-28 11:47:52.113956 [I:onnxruntime:, inference_session.cc:226 operator()] Flush-to-zero and denormal-as-zero are off
2021-03-28 11:47:52.113999 [I:onnxruntime:, inference_session.cc:233 ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true
2021-03-28 11:47:52.166840 [I:onnxruntime:, inference_session.cc:1107 Initialize] Initializing session.
2021-03-28 11:47:52.166867 [I:onnxruntime:, inference_session.cc:1132 Initialize] Adding default CPU execution provider.
2021-03-28 11:47:52.170814 [I:onnxruntime:, reshape_fusion.cc:37 ApplyImpl] Total fused reshape node count: 0
2021-03-28 11:47:52.171579 [I:onnxruntime:, reshape_fusion.cc:37 ApplyImpl] Total fused reshape node count: 0
2021-03-28 11:47:52.177451 [V:onnxruntime:, inference_session.cc:893 TransformGraph] Node placements
2021-03-28 11:47:52.177475 [V:onnxruntime:, inference_session.cc:895 TransformGraph] All nodes have been placed on [CPUExecutionProvider].
2021-03-28 11:47:52.177692 [V:onnxruntime:, session_state.cc:76 CreateGraphInfo] SaveMLValueNameIndexMapping
2021-03-28 11:47:52.177763 [V:onnxruntime:, session_state.cc:122 CreateGraphInfo] Done saving OrtValue mappings.
2021-03-28 11:47:52.178018 [I:onnxruntime:, session_state_utils.cc:103 SaveInitializedTensors] Saving initialized tensors.
2021-03-28 11:47:52.178093 [I:onnxruntime:, session_state_utils.cc:177 SaveInitializedTensors] [Memory] SessionStateInitializer statically allocates 35748096 bytes for Cpu

2021-03-28 11:47:52.200760 [I:onnxruntime:, session_state_utils.cc:219 SaveInitializedTensors] Done saving initialized tensors
2021-03-28 11:47:52.206054 [I:onnxruntime:, inference_session.cc:1282 Initialize] Session successfully initialized.
Created IntelliCode session.
Background analysis message: analyze
[BG(1)] analyzing: /Users/nick/dev/stw-on-mensa-bot/mensabot/__init__.py ...
[BG(1)]   parsing: /Users/nick/dev/stw-on-mensa-bot/mensabot/__init__.py (21ms)
[BG(1)]   parsing: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stdlib/builtins.pyi [fs read 2ms] (100ms)
[BG(1)]   binding: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stdlib/builtins.pyi (38ms)
[BG(1)]   binding: /Users/nick/dev/stw-on-mensa-bot/mensabot/__init__.py (1ms)
[BG(1)]   checking: /Users/nick/dev/stw-on-mensa-bot/mensabot/__init__.py ...
[BG(1)]     parsing: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stdlib/pkgutil.pyi [fs read 1ms] (6ms)
[BG(1)]     binding: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stdlib/pkgutil.pyi (1ms)
[BG(1)]     parsing: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi [fs read 0ms] (10ms)
[BG(1)]     binding: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stdlib/_typeshed/__init__.pyi (1ms)
[BG(1)]     parsing: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stdlib/typing.pyi [fs read 1ms] (33ms)
[BG(1)]     binding: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stdlib/typing.pyi (12ms)
[BG(1)]     parsing: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stubs/typing-extensions/typing_extensions.pyi [fs read 0ms] (3ms)
[BG(1)]     binding: /Users/nick/.vscode/extensions/ms-python.vscode-pylance-2021.3.2/dist/typeshed-fallback/stubs/typing-extensions/typing_extensions.pyi (2ms)
[BG(1)]   checking: /Users/nick/dev/stw-on-mensa-bot/mensabot/__init__.py (100ms)
[BG(1)] analyzing: /Users/nick/dev/stw-on-mensa-bot/mensabot/__init__.py (262ms)
Background analysis message: resumeAnalysis

Code Snippet / Additional information

from pkgutil import extend_path

__version__ = "0.0.1"


__path__ = extend_path(__path__, __name__)  # this line shows error

[erictraut]

Thanks for the bug report. You're correct, the type of __path__ should be Iterable[str], but it's currently hard-coded to List[str]. This will be fixed in the next release of pylance.

[jakebailey]

This issue has been fixed in version 2021.3.4, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/main/CHANGELOG.md#202134-31-march-2021

[cdce8p]

Sorry for reopening it, but I came across this issue while updating mypy to 0.920.
It seems list[str] might actually be the better type for __path__.

Technically __path__ can be anything that is an iterable of strings, which is why the Python docs are correct. In most situations though, it's a list[str].

The stubs from pkgutil.extend_path have been update recently to use list[str] https://github.com/python/typeshed/pull/5222/files and __path__ itself is defined in typeshed as MutableSequence[str]: typeshed/types.pyi -> ModuleType.

With mypy 0.920, reveal_type(__path__) also returns builtins.list[builtins.str]].

The original mypy issue: python/mypy#1422

--
In particular Iterable[str] doesn't define __getitem__. This here will work fine.

# file: some_module/__init__.py
print(__path__[0])

[erictraut]

Based on the Python docs, it would be legitimate to assign an immutable iterable type to __path__, correct? For example: __path__ = tuple(__path__). That means it would be unsafe to assume that it's mutable, no?

[cdce8p]

Based on the Python docs, it would be legitimate to assign an immutable iterable type to __path__, correct? For example: __path__ = tuple(__path__). That means it would be unsafe to assume that it's mutable, no?

Probably? I don't know enough about it to be sure.
I saw this comment in the docstring from pkgutil.extend_path (cpython/pkgutil.py):

    If the input path is not a list (as is the case for frozen
    packages) it is returned unchanged.  The input path is not
    modified; an extended copy is returned.  Items are only appended
    to the copy at the end.

That would indeed suggest that __path__ can be immutable, too.
A few lines later, this is the first check in extend_path.

    if not isinstance(path, list):
        # This could happen e.g. when this is called from inside a
        # frozen package.  Return the path unchanged in that case.
        return path

--
Thinking out loud for a moment, if mutable types are unsafe, that would leave Sequence[str] which still defines __getitem__. The types for extend_path would need to be updated to allow a TypeVar with bound Iterable[str] as first argument (and return type), instead of list[str], but that could work.

Another aspect to consider, how common is it for users to try to modify __path__ manually, without extend_path?

[erictraut]

I don't have a strong opinion about this, but I don't want to keep changing it. I've already changed this once, and the current code appears to match the Python documentation. Before changing it again, I'd like to get a definitive answer — or at least an indication that there's consensus within the typing and typeshed communities on the right answer. If you are so motivated, please drive that process and once there's consensus, file a new pyright feature request for the change. Thanks!