Use PEP 484 stub files when documenting native extensions with autodoc

[hoefling]

Is your feature request related to a problem? Please describe.

Restating the comment:

My use case is to provide documentation for a native C extension in a way that it is identical with docs for pure Python modules. Right now, Sphinx can't read function/method signatures for C extensions so the argument lists remain empty.

I have created a sample repository with a minimal reproducible example. If building the docs from it, the rendered result for the fizz native extension is:

As can be seen from the above screenshot, the Buzz.fuzz method has no arguments documented.

Describe the solution you'd like

If a native extension has PEP 484 stub files available, Sphinx could use them to extract the missing information about function/method signatures, resulting in a richer docs:

Another possible, although much less common, scenario could be even extracting the function signatures along with the docstrings from the stub files if available. PEP 484 doesn't forbid the stub files to contain docstrings, so maybe a switch in config could decide whether to draw the docs from the compiled extension object or the stub files.

Describe alternatives you've considered

The only similar issue I've found is #4824, however, Sphinx installed from it doesn't resolve stub files (maybe the code is too old now).

Additional context

• sample repository to play with
• PEP 484
• PEP 561

[hoefling]

I have just discovered autodoc_docstring_signature which suits my use case well. The only issue with it is that I have to duplicate the signatures in the docstrings of the C code. I'm thus fine with closing the issue if you consider it irrelevant!

[tk0miya]

Thank you for filing. I'd like to support .pyi file in the future. So I keep this opened.

[hoefling]

@tk0miya thanks! This would be especially useful when e.g. building the docs on https://readthedocs.org where I can't build the C extensions because RTD doesn't allow installing native libs.

[tk0miya]

I think it would be a helper of autodoc. So that situation would not be changed.

[daggy1234]

Hey small necrobump, but any update. I have a lib that isn't pure python and therefore for the docs would love to be able to read my stubs. do lmk!

[Cheaterman]

I would love this too, FWIW. :-)

EDIT: Our use-case is an embedded Python library, so pretty much the exact same as OP's :-)

[schnemensch]

I would like to push this issue as well.
We have several reasons for using .pyi stub files for annotations and documentation.

[Amourspirit]

I also would like to push thus issue.

[kdschlosser]

not sure why this is an issue for so many. rename the extension module extension to py_ or whatever and then rename the stub file extension from pyi to py. You have to make sure that your code isn't going to run when sphinx builds the documentation but that should not be all that hard to accomplish.

This is typical of what would be seen in a stub file.

ATTRIBUTE: int = ...


class Test(object):
    """
    This is a doc string
    """

    def method(self, test1, test2: str, test3: int = 0) -> None:
        """
        method docstring
        """
        ...


def function(test1, test2: str, test3: int = 0) -> float:
    """
    function docstring
    """
    ...

This runs just fine as python code and sphinx should be able to do it's thing without an issue so long as the stub file is able to run as python code.

[hoefling]

@kdschlosser unfortunately, this is not a viable solution for two reasons:

1. sphinx will import the renamed stubs when collecting objects in the autodoc extension, so this will only work for simple use cases. If you use typing stuff not defined at runtime (e.g. from _typeshed import Self & friends), it will break the doc build. This could be avoided by introducing if TYPE_CHECKING: blocks, but is not meant for the stubs.
2. This requires putting the docstrings into stubs. This leads to multiple problems as well: you have to keep the docstrings in the extension source code so help(my_extension_module) still works having to maintain duplicate docstrings, this will conflict with e.g. flake8-pyi rules etc. Also I have a use case where I'm assembling different docstrings depending on the platform tag (docs are generated for different platforms and architectures, choosable via dropdown, and render different values for constants from macrodefs). I would probably have to keep multiple versions of platform-specific stubs files with different docstrings with this approach.

[kdschlosser]

what I have done in the past was the generation of the stub file using doxygen xml output for extension modules. So there was never any python doctrings that would be available at runtime anyhow... same goes for type checking. Never going to be available at runtime do to it being an extension module. The if TYPE_CHECKING statement being is is the fame thing as doing if False. The code never runs but an IDK does not know that so an IDE is able to work properly. It is a trick I have used for extension modules before stub files were used in Python. I would create a wrapper module that would load the extension module and in that wrapper module was in False: followed by class and function declarations that contained the docstrings with sphinx style type hints. to build documentation the if False: statement was removed but the class , function and module attribute declarations remained. I also would remove the dynamic loading of the extension module. Some enough to do in a build script.

Python does not do type checking at runtime so using a stub file to generate documentation should be a non issue because nothing is actually being run. Documentation generation does not require the script to be running in order to create it. It only requires that the code be loaded.

And you also stated that you generate different docstrings based on the platform. So you are already having your build program write code to have those docstrings used, how hard would it be to have it write a dummy python module that gets used to build the documentation from at the same time?..

I have done this for generating stub files when compiling a Cython project. Cython allows for python docstrings but it doesn't handle type hinting. That is where I use sphinx style type hinting in the docstring. I compile the extension module and in the build program I have it load the extension module and read the docstrings, parse the parameter names, default values and types from the docstring and generate a stub file. With Cython there is no type hinting but sphinx still works properly because it still uses type hinting contained within the doctsrings.

C Extensions would be written and have documentation that is handled by doxygen and a build program would then read the XML files from doxygen which contain all of the data types, default parameter values, object names and documentation...

Yes it is more code that would need to be written to handle the documentation being created properly. But trust me when I say it would be far easier to do at the user end then it would be for the developers of sphinx to write code that would read a stub file line by line and determine the types and documentation. It would have to be written so it could handle different coding styles like hanging indents for parameters and newline for each parameter, tabs or spaces.. 2 spaces or 4 spaces, 4 spaces for indents on parameters or 8 spaces, etc... It ends up becoming a real headache on their end. Being able to load the code and run it means type annotations are easily gotten using inspect and docstrings are easily collected as well. No need to read the code line by line and parse each line to get what is needed.

I have never written any python code that actually needs to "run" to be able to have documentation generated or type hinting to work properly. And IDE is not running the code in order for it to tell you that you are not passing a valid data type. So why would the code need to run in or for sphinx to generate documentation properly? If the code has to run in order for documentation to be set or whatever and type hinting to be set then that completely removes the purpose of docstrings and type hinting which is to have the information accessible from an IDE.

[jpmckinney]

I’m confused. Why would there be code that cares about indentation in a pyi file? The type information can be extracted using the same standard libraries as for py files.

[iluvcapra]

I'd like this feature too 👍 I'm trying to document classes auto-generated by protoc in a protobuf/grpc client stub.

[duburcqa]

Any update ? I would love this feature too !

[kdschlosser]

I have a fantastic use case for this.

Some may have heard of MicroPython and some may not. MicroPython is as the name suggests it is Python made to have a really small footprint. Low memory and low disk use. The intended use is for running on MCUs or single chip computers that have limited resources. Something that has 500K of RAM and things of that nature.

While MIcroPython does make use of extension modules they are not a traditional type of extension module and it is not something that is able to be loaded using CPython so sphinx would not be able to access the module to collect any information from within it.

There is a graphics library that has been made and it is not specific to MicroPython. That library is written in C code and there is a program that has been made that uses pycparser to port the code so it will work in MicroPython. The issue here is the naming conventions change from the C version to the Python version and the data types also change. So using the generated documentation using doxygen form the C code does not align with the port to MicroPython. I have written a script that makes a mapping of the c names to the python names and the type changes as well. The script then reads the xml files that are generated by doxygen to collect the documentation and any types that have not been changed. I use that information to build a stub file. The stub file I am able to place into the root of a project I am working on and that then gives me the IDE features like auto complete and type checking. There is still no documentation that is available because there is no extension module that can be loaded using CPython and Sphinx does not use a .pyi file to collect the information.

It is not a matter of having to maintain documentation that is in 2 different locations. In my use case everything is generated at build time so there is actually only one place the documentation exists which in the c code. Because of the nature of what is being made and there being resource limitations documentation is not something that needs to be available at runtime nor does the type hinting. If those things where available at runtime it would consume far to much storage space and also memory. The purpose of the stub file is only for providing type hinting and autocomplete in an IDE and the second part would be to generate documentation.

This is easy enough for me to handle during the build process because I can simply change the extension of the file from .pyi to .py and have sphinx build the documentation for the module using CPython. It would be nice if Sphinx had the ability to use a stub file internally as it would save me from having to rename the file. The only way I could see something like this working properly is if sphinx loaded an extension module and also checked for the existence of a stub file for that module If there was a stub file then that would be used for the generation of the documentation and if it was not available then it would collect documentation from the extension module itself. In the event an import of an extension module failed then the stub file would be used if there was one available. That would take care of version specific extension modules and making sure the documentation is created from the same version of Python the extension module was created for. That would not longer be an issue if a stub file existed.

The am pretty sure the original purpose of the stub file was to provide information to an IDE for extension modules. This is because an IDE is not able to read the contents of an extension module unless the extension module has been loaded and python is running. The intention of it was not to be used for python source files and any type hinting and documentation for a python source file is supposed to be located in said source file. The mechanics of an IDE will use a stub file first if available and then a source file second. So what has happened is people have started to use stub files in combination with python source files as a mechanism to make it easier to read without all of the type hinting cluttering up the source file. Stub files were intended to be created at compile time of an extension module by the build program. optionally a person could create the stub file manually if wanted but that would be more work to maintain then something that is automatically generated.

In my specific use case it would be impossible for me to build the documentation from the extension module using CPython. I am able to generate a CPython stub file but Sphinx does not support reading those files to generate the documentation. So the only option is to rename the stub file to a Python source file and then generate the documentation from that.

Does this specific feature need to be built into Sphinx.. I say no it doesn't because it is something that is pretty easy to deal with in the build code. However, It would be nice if it did as it would save me from having to write code in the build program to handle this specific issue. Nice to have? Yes definitely, Is it a must to have have into sphinx? This is subjective and it would depend on the person you ask. I say no but others may say yes..