Support Pydantic model attributes

[StephenBrown2]

I'm using Pydantic for several models in my code, and I would like mkdocstrings to handle them. The issue is, class attributes get sucked up into a __fields__ attribute with separate __annotations__, and so they get filtered out by filter_name_out due to the global_filters.

Example __dict__.items() on a Pydantic class with two fields (debugged from just before https://github.com/pawamoy/mkdocstrings/blob/ca82f6964e223b7d8b0a3d0e0fe4d9ea8f4296ce/src/mkdocstrings/documenter.py#L340):

class_.__dict__.items(): (
    dict_items(
        [
            ('__config__', <class 'pydantic.main.Config'>),
            ('__fields__', {
                'start': ModelField(name='start', type=DateTime, required=True),
                'end': ModelField(name='end', type=DateTime, required=True)
            }),
            ('__field_defaults__', {}),
            ('__validators__', {
                'start': [<pydantic.class_validators.Validator object at 0x10257df10>],
                'end': [<pydantic.class_validators.Validator object at 0x10257df10>]
            }),
            ('__pre_root_validators__', []),
            ('__post_root_validators__', []),
            ('__schema_cache__', {}),
            ('__json_encoder__', <staticmethod object at 0x10260e978>),
            ('__custom_root_type__', False),
            ('__module__', 'mypackage.models.available_times'),
            ('__annotations__', {
                'start': <class 'pendulum.datetime.DateTime'>,
                'end': <class 'pendulum.datetime.DateTime'>
            }),
            ('__doc__', '### TimeSlot\n\n    A slot of time in which a maintenance may be scheduled.\n    '),
            ('__lt__', <function TimeSlot.__lt__ at 0x1026152f0>),
            ('parse_times', <classmethod object at 0x10260e828>),
            ('__weakref__', <attribute '__weakref__' of 'TimeSlot' objects>),
            ('__abstractmethods__', frozenset()),
            ('_abc_registry', <_weakrefset.WeakSet object at 0x10260e908>),
            ('_abc_cache', <_weakrefset.WeakSet object at 0x10260ea58>),
            ('_abc_negative_cache', <_weakrefset.WeakSet object at 0x10260eac8>),
            ('_abc_negative_cache_version', 42)
        ]
    )
)

A crude implementation for an older version of pydantic was put forward here: pydantic/pydantic#638 but not accepted into Pydantic proper. Perhaps mkdocstrings could take some of the logic and parse through the field attributes?

Update: I should probably also note that I started down this path by attempting to add some docstrings to the attributes as suggested by mkdocstrings docs, but then got this error:

  File "/Users/step7212/git/rack/emcee/.nox/docs-3-6/lib/python3.6/site-packages/mkdocstrings/documenter.py", line 437, in node_to_annotated_names
    name = node.target.attr
AttributeError: 'Name' object has no attribute 'attr'

Changing node.target.attr to node.target.id fixed that for me avoided the error, but I'm not sure how relevant it is, just means I can't docstring my pydantic models (yet) since it's filtering the dunder-attrs.

[pawamoy]

I'm favorable to this. I'll just have to play with pydantic a bit before tackling it.

[shyamd]

I'm actually very interested in this. Once I can get my head around where you're going with the refactor, I'm more than happy to tackle this.

[pawamoy]

Great! So, as I said in the refactor issue, I'll write a proper summary of how mkdocstrings can be implemented, and the advantages/disadvantages of each method. Hopefully it will help us choosing the best architecture.

[StephenBrown2]

It may be possible to utilize the JSON Schema generation for Pydantic models as has been done with sphinx-pydantic[1] via sphinx-jsonschema[2].

From Model Properties:

.schema()
    returns a dictionary representing the model as JSON Schema; cf. Schema
.schema_json()
    returns a JSON string representation of schema(); cf. Schema

[1] https://pypi.org/project/sphinx-pydantic/
[2] https://pypi.org/project/sphinx-jsonschema/

[demospace]

FWIW: After testing sphinx-pydantic (which is light a wrapper around sphinx-jsonschema), I wish a more robust approach were available.

Sphinx's autodoc plugin is very good about creating linked references to inherited models, related models, params, and a host of other well-worn doc features which are elided/unavailable/ugly in the jsonschema approach.

json_schema is great for many things, but it can quickly become unreadable. Self-documenting pydantic models with just a little more functionality than the json_schema docs provide would be extremely useful when working with other dev teams, PMs, etc.

A Sphinx pre-processor plugin would probably get the job done with zero run time overhead, but populating docstrings on Fields() would be useful for many other static analysis libs that rely on robust docstring information.

[pawamoy]

If you don't mind, I'll transfer the issue over to pytkdocs, since it's the project that is now responsible for loading documentation from Python code!

[pawamoy]

@shyamd now that the refactor is done, you're free to try and come up with a PR 🙂
I've made the code a bit more structured, robust and readable. I guess supporting Pydantic models would require to update the get_class_documentation method: https://github.com/pawamoy/pytkdocs/blob/8dfd0bbf2e73977fc1cc9528e62057f3764a5a07/src/pytkdocs/loader.py#L302

[shyamd]

So both dataclass and pydantic models have __annotations__. I can keep it strict to just that, or I can add in code specific to pydantic. Do you have any issue with adding in pydantic specific compatibility code?

[pawamoy]

Do you have any issue with adding in pydantic specific compatibility code?

Not at all! This should simply be properly commented / documented 🙂

[StephenBrown2]

Might want to start with the more generic approach, being able to knock out both stdlib dataclasses and Pydantic models would be a great benefit!

[pawamoy]

I think that's what @shyamd meant?

I can keep it strict to just that, or I can add in code specific to pydantic

• just that = at least dataclass support, maybe basic pydantic support
• add in = both dataclass and pydantic support

[pawamoy]

Hi @StephenBrown2, @demospace!

@shyamd just got their pull request merged in master! If you want to try them now, before I make a new release, you can simply pip install git+https://github.com/pawamoy/pytkdocs in your repository to see if Pydantic and dataclass attributes are correctly picked up and rendered by mkdocstrings 🙂

[StephenBrown2]

So I tried real hard, but it seems that it's not working in my main project, but it does work on a minimal demo project. Probably something on my end, but it's a good start!

Also, that project is getting sunset, so I won't be able to test it out much there. I think that this is probably good to go, since it works in a fresh environment. I would prefer that it gathered doc-comments as well if it didn't get the FieldInfo, since dataclasses also don't have a description, but that's not critical for a first release.

[pawamoy]

About dataclasses fields not having a description, yeah, we decided to do that in another PR, because it needs a refactoring of the attributes parser. I'll go ahead and put this in the "to release" column 🙂

[pawamoy]

Released in v0.4.0. Please open new issues for any encountered problem with this feature!