python: some libraries have failing docfx sessions

[busunkim96]

I recently made some changes to the (1) type references in the docstrings and (2) the docs directory layout. I am unsure which is the problem. Many but not all libraries are failing.

Failed: googleapis/python-datalabeling#65
Passed: googleapis/python-data-qna#21

Expected Behavior

docfx session succeeds when the docs session passes.

Actual Behavior

docfx session fails with the following error:

...
reading sources... [ 50%] datalabeling_v1beta1/data_labeling_service
Can't inspect type <class 'property'>: google.cloud.datalabeling_v1beta1.services.data_labeling_service.DataLabelingServiceClient.transport
Can't inspect type <class 'property'>: google.cloud.datalabeling_v1beta1.services.data_labeling_service.DataLabelingServiceAsyncClient.transport
/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/util/nodes.py:54: FutureWarning:
   The iterable returned by Node.traverse()
   will become an iterator instead of a list in Docutils > 0.16.
  for classifier in reversed(node.parent.traverse(nodes.classifier)):
reading sources... [ 62%] datalabeling_v1beta1/services
reading sources... [ 75%] datalabeling_v1beta1/types
Can't inspect type <class 'proto.message.MessageMeta'>: google.cloud.datalabeling_v1beta1.types.ImageSegmentationAnnotation.AnnotationColorsEntry
Can't inspect type <class 'proto.message.MessageMeta'>: google.cloud.datalabeling_v1beta1.types.LabelStats.ExampleCountEntry
Can't inspect type <class 'proto.message.MessageMeta'>: google.cloud.datalabeling_v1beta1.types.EvaluationJobConfig.BigqueryImportKeysEntry
Can't inspect type <class 'property'>: google.cloud.datalabeling_v1beta1.types.ListDatasetsResponse.raw_page

Traceback (most recent call last):
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/autodoc.py", line 687, in process_doc
    self.env.app.emit('autodoc-process-docstring',
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/application.py", line 589, in emit
    results.append(callback(self, *args))
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/napoleon/__init__.py", line 357, in _process_docstring
    docstring = NumpyDocstring(result_lines, app.config, app, what, name,
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/napoleon/docstring.py", line 824, in __init__
    super(NumpyDocstring, self).__init__(docstring, config, app, what,
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/napoleon/docstring.py", line 158, in __init__
    self._parse()
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/napoleon/docstring.py", line 492, in _parse
    self._parsed_lines.extend(self._parse_attribute_docstring())
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/napoleon/docstring.py", line 516, in _parse_attribute_docstring
    _type, _desc = self._consume_inline_attribute()
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/napoleon/docstring.py", line 238, in _consume_inline_attribute
    line = next(self._line_iter)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/napoleon/iterators.py", line 65, in __next__
    return getattr(self, 'next')(n)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/napoleon/iterators.py", line 117, in next
    raise StopIteration
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/cmdline.py", line 296, in main
    app.build(opts.force_all, filenames)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/application.py", line 333, in build
    self.builder.build_update()
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 249, in build_update
    self.build(to_build,
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 264, in build
    updated_docnames = set(self.env.update(self.config, self.srcdir,
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/environment/__init__.py", line 556, in update
    self._read_serial(docnames, app)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/environment/__init__.py", line 576, in _read_serial
    self.read_doc(docname, app)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/environment/__init__.py", line 684, in read_doc
    pub.publish()
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/core.py", line 217, in publish
    self.document = self.reader.read(self.source, self.parser,
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/io.py", line 55, in read
    self.parse()
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/parsers/rst/__init__.py", line 191, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 170, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 393, in new_subsection
    newabsoffset = self.nested_parse(
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2096, in directive
    return self.run_directive(
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/autodoc.py", line 1659, in run
    documenter.generate(more_content=self.content)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/autodoc.py", line 1004, in generate
    self.document_members(all_members)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/autodoc.py", line 924, in document_members
    documenter.generate(
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/autodoc.py", line 1004, in generate
    self.document_members(all_members)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/autodoc.py", line 1379, in document_members
    ModuleLevelDocumenter.document_members(self, all_members)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/autodoc.py", line 924, in document_members
    documenter.generate(
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/autodoc.py", line 1001, in generate
    self.add_content(more_content)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/autodoc.py", line 1551, in add_content
    ClassLevelDocumenter.add_content(self, more_content, no_docstring)
  File "/tmpfs/src/github/python-datalabeling/.nox/docfx/lib/python3.8/site-packages/sphinx/ext/autodoc.py", line 727, in add_content
    for i, line in enumerate(self.process_doc(docstrings)):
RuntimeError: generator raised StopIteration

[tmatsuo]

According to my naive brute force attempts, if we remove undoc-members directive from the docs config, it succeeded. What is this directive for?

[busunkim96]

I added undoc-members to get enum members listed in the generate docs. I assume there are other ways to get the enums? googleapis/gapic-generator-python#625

[tmatsuo]

Now we know the reason. Considering that we are still not officially using the docfx yaml files, we can disable docfx builds. WDYT?

[busunkim96]

googleapis/synthtool#889 is almost ready and will remove docfx from the main Kokoro presubmit so it won't block merge. I think it's okay to let it keep running separately in docs-presubmit.

[busunkim96]

@Shabirmean found sphinx-doc/sphinx#5870 (comment) which says this was fixed in sphinx v1.7.6.

We cannot currently upgrade sphinx though because of docascode/sphinx-docfx-yaml#97

I suspect the strict sphinx==1.5.5 pin is not necessary and the tool would work with newer sphinx versions. https://github.com/docascode/sphinx-docfx-yaml/blob/23b4cd46c3e52d271be01647155e7164863fdafd/setup.py#L7 Might be worth trying to submit a PR that loosens the pin.

[busunkim96]

Seeing a consistently red check (even if it's optional) seems to be confusing folks, so I've opened cl/361161679 to turn off the Python docs-presubmit jobs (the jobs that run the docfx session).

[dandhlee]

Sounds good. Thanks for the update!

[dandhlee]

gcp-sphinx-docfx-yaml is live, and tested on the datalabeling repo with the new plugin to verify that it works. Will be updating it on Synthtool to resolve this!