bug: BuiltinModuleError in PyTorch 2.3.0

[sbrugman]

Amazing work on griffe!

Description of the bug

For torch 2.3.0 we get the following error:

  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/models.py", line 1503, in filepath
    raise BuiltinModuleError(self.name)
_griffe.exceptions.BuiltinModuleError: _profiler

Version 2.4.0 runs without issue.

To Reproduce

pip install "torch==2.3.0"
griffe dump torch

Full traceback

Full traceback

INFO       Loading package torch
Traceback (most recent call last):
  File "/Users/me/Documents/code/project/venv/bin/griffe", line 8, in <module>
    sys.exit(main())
  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/cli.py", line 560, in main
    return commands[subcommand](**opts_dict)
  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/cli.py", line 387, in dump
    loader = _load_packages(
  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/cli.py", line 99, in _load_packages
    loader.load(package, try_relative_path=True, find_stubs_package=find_stubs_package)
  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/loader.py", line 179, in load
    top_module = self._load_package(package, submodules=submodules)
  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/loader.py", line 511, in _load_package
    top_module = self._load_module(package.name, package.path, submodules=submodules)
  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/loader.py", line 535, in _load_module
    return self._load_module_path(module_name, module_path, submodules=submodules, parent=parent)
  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/loader.py", line 565, in _load_module_path
    self._load_submodules(module)
  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/loader.py", line 570, in _load_submodules
    self._load_submodule(module, subparts, subpath)
  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/loader.py", line 621, in _load_submodule
    parent_module.set_member(submodule_name, submodule)
  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/mixins.py", line 208, in set_member
    if value.is_module and value.filepath != member.filepath:
  File "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/_griffe/models.py", line 1503, in filepath
    raise BuiltinModuleError(self.name)
_griffe.exceptions.BuiltinModuleError: _profiler

Expected behavior

Output the json information, e.g.

{
  "torch": {
    "docstring": {
      "endlineno": 11,
      "lineno": 3,
      "value": "The torch package contains data structures for multi-dimensional\ntensors and defines mathematical operations over these tensors.\nAdditionally, it provides many utilities for efficient serialization of\nTensors and arbitrary types, and other useful utilities.\n\nIt has a CUDA counterpart, that enables you to run your tensor computations\non an NVIDIA GPU with compute capability >= 3.0."
    },
    "filepath": "/Users/me/Documents/code/project/venv/lib/python3.10/site-packages/torch/__init__.py",
    "kind": "module",
    "labels": [],
 ...

Environment information

• System: macOS-14.6.1-arm64-arm-64bit
• Python: cpython 3.10.13 (/Users/me/Documents/code/project/venv/bin/python)
• Environment variables:
• Installed packages:
  • griffe v1.2.0

[pawamoy]

Hey, thanks for the kind words and for the report!

It looks like the traceback is incomplete? I'll try to reproduce it anyway, the given info should be enough 🙂

[sbrugman]

Hey, the three lines posted above were missing. Added them now. Again, great tool!

[pawamoy]

So, with only torch==2.3.0 (no dependencies), Griffe loads and dumps it without any issue. If I install torch's dependencies as well, I'm getting an error similar to yours: _griffe.exceptions.BuiltinModuleError: _cudnn. I suppose that fixing mine will fix yours too. Continuing investigation.

[pawamoy]

OK, looks like torch has some compiled modules (torch._C) that contain other compiled modules (torch._C._cudnn) 🤯 In my case, torch._C._cudnn doesn't have a file path, so when trying to merge stubs found in the corresponding .pyi file, Griffe tries to compare file paths and crashes. I guess I can just catch and ignore the exception.

[pawamoy]

Should be fixed in v1.3.0 🙂