Recurse through submodules in extension modules

[robamler]

Fixes #319

The original method of finding submodules by recursing through the
directory structure didn't work with extension modules (aka native "C"
extensions) because there is no directory structure to recurse through.
This commit adds code to recurse through submodules by inspecting
obj.__all__, i.e., using the same mechanism that is already used for
finding variables, functions, and classes.

This is a minimalistic first sketch of a solution. It works on a native
extension that I tried it on but it introduces some duplication since
there are now two strategies to search for submodules. For a native
module, the first strategy (searching the directory structure) will fail
but the second (new) strategy (using obj.__all__) will work. For
regular modules, both strategies will likely work and so the second
strategy will just overwrite the results from the first strategy with
identical objects. This should technically work but it seems
unnecessary. I just don't feel confident enough to remove the first
strategy without further guidance.

[kernc]

Can you construct a minimal, self-contained test with a module that has no path, but contains a reference to another module, and assert both modules are discovered? Like:

pdoc/pdoc/test/__init__.py

Line 42 in aef1917

EMPTY_MODULE = ModuleType('empty')

For regular modules, the second strategy will just overwrite the results

The procedure actually skips modules that have the same name as one that was already defined:

pdoc/pdoc/__init__.py

Lines 734 to 737 in aef1917

	for root in iter_modules(self.obj.__path__):
	# Ignore if this module was already doc'd.
	if root in self.doc:
	continue

Though this method seems unreliable.

Further, if I:

import .foo as bar

module foo will now be be documented as foo and as bar ... 🤔

[robamler]

Thanks for the quick feedback! I've moved the new code into the if self.is_package - else block and added a test.

Studying the existing code for "normal" submodule discovery, I realized I should be more careful to avoid capturing cycles and reexports (not sure if they're possible in native extension modules though). So I made sure the new code only runs if the parent module explicitly exposes public members through the __all__ attribute. This restriction seems to be necessary because the is_from_this_module function doesn't work on submodules (inspect.getmodule(x) returns x rather than x's parent if x is a submodule). This restriction doesn't seem to be a problem for the native extension modules I tried, but I'd be happy to relax it if you have an idea how.

Further, if I:

import .foo as bar

module foo will now be be documented as foo and as bar ... 🤔

Is that possible with native extension modules? I know too little about Python's module system to tell.

[robamler]

One problem I encountered: the HTML documentation for a native extension module now places all files in the top level directory. The contents of the files shows the correct nested hierarchy of submodules but the paths for the corresponding HTML files doesn't reflect this, so paths may clash if, e.g., a submodule has the same name as its parent module. Any idea how to fix this?