Reference manual: class hierarchy, inherited members, underscore variables

[qed777]

In the Sage reference manual, it might be useful to

• List the base classes for a class.
• List inherited members but not their docstrings.
• Include docstrings for "private" variables (e.g., __init__, _foo).
• Include inheritance diagrams.

Sphinx extensions of interest: autodoc, inheritance_diagram.

See sage-devel for some discussions.

CC: @hivert @jhpalmieri @nthiery @sagetrac-mvngu @jasongrout

Component: documentation

Author: Mitesh Patel

Reviewer: John Palmieri

Merged: sage-4.4.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/7549

[qed777]

Attachment: trac_7549-doc_inheritance_underscore.patch.gz

Doc build options for inherited members, private variables. Inheritance diagrams. Apply to sage repo.

[qed777]

comment:1

The patch, which may depend weakly on #7473's "builder" patch,

• Adds a sage -docbuild option -i to include inherited members, mostly without docstrings.

• Adds a sage -docbuild option -u to include "private" variables.

• Uses Sphinx's :show-inheritance: autodoc option to list the base classes of a class.

• Sets up Sphinx's inheritance_diagram extension. See this attachment for an example. This requires Graphviz.

It's likely that the attached patch needs work. In particular, sage.doc.common.conf.process_inherited would benefit from expert intervention.

Note: To force a rebuild when toggling -u, add -S -aE to the command-line. For example,

sage -docbuild reference html -juv3 -S -aE

Toggling -i should automatically trigger a rebuild.

[qed777]

comment:3

Running

sage -docbuild reference html -juiv3

I encountered

reading sources... [ 27%] sage/combinat/lyndon_wordtor_weightedturesnssis_basis
Sphinx error:
'ascii' codec can't decode byte 0xc3 in position 811: ordinal not in range(128)

Are #6674 and #6682 relevant?

[qed777]

comment:4

Putting

# -*- coding: utf-8 -*-                                                         

at the beginning of sage.combinat.lyndon_word seems to placate Sphinx.

Should we do this with every Sage library .py and .pyx file?

[jhpalmieri]

comment:5

After adding the "utf-8" line to lyndon_word.py, I got this error:

reading sources... [ 51%] sage/interfaces/singular                 ismsm
reST markup error:
/Applications/sage/local/lib/python2.6/site-packages/sage/interfaces/singular.py:docstring of sage.interfaces.singular.SingularFunctionElement._sage_doc_:8: (SEVERE/4) Unexpected section title.

After fixing this (by ripping out a doctest), things built fine. The html version of the reference manual looks good, and seems to have all of the methods we want, with the inherited ones having no docstrings.

Here are some timings on my iMac. Between each build, I removed the directory doc/output:

sage -docbuild reference html -j        11 minutes
sage -docbuild reference html -ji       22 minutes
sage -docbuild reference html -ju      13 minutes
sage -docbuild reference html -jiu     63 minutes!

Also

sage -docbuild reference pdf -iu  

took too long on my iMac. On sage.math, it took about 2 hours and then bombed when trying to run pdflatex: there was one error (a missing "r" before a triple-quoted docstring with backslashes); once this was fixed, it ran for a while and then said

! TeX capacity exceeded, sorry [pool size=1165810].

I don't know if there is a way of fixing this, short of #6495. (I think that the tex installation on sage.math is somewhat old, which doesn't help.)

[jhpalmieri]

comment:6

I have yet to succeed making a pdf version of the reference manual. On the two machines I've tried, it takes ages and then crashes with "TeX capacity exceeded". This happened just with the "-i" option on my iMac. I haven't tried all of the possibilities on all of the machines.

So I wonder if it makes sense, at least until something like #6495 is in place, to disable the "-i" and "-u" flags for the pdf version of the manual. If not disable them, then at least print a warning saying it's going to take a long time and then might very well crash?

[qed777]

comment:7

Sure, I have no problem with either course. We could also make -i and -u "Advanced" options.

Aside: I can't even build the HTML manual with -iu on my local machine. It requires too much memory.

[qed777]

comment:8

Maybe we can use Sphinx's intersphinx extension to implement a HTML analogue of #6495?

[qed777]

comment:9

For links to Sage tickets, we could use Sphinx's extlinks extension.

[hivert]

comment:10

Also

sage -docbuild reference pdf -iu  

took too long on my iMac. On sage.math, it took about 2 hours and then bombed when trying to run pdflatex: there was one error (a missing "r" before a triple-quoted docstring with backslashes); once this was fixed, it ran for a while and then said

! TeX capacity exceeded, sorry [pool size=1165810].

I don't know if there is a way of fixing this, short of #6495. (I think that the tex installation on sage.math is somewhat old, which doesn't help.)

Did you try to launch lex with more memory ? I remember doing a

export extra_mem_bot=8000000; latex ...

in a similar situation.

[qed777]

Set up intersphinx for Python docs. Replaces previous patch. sage repo.

[qed777]

Author: Mitesh Patel

[qed777]

comment:11

Attachment: trac_7549-doc_inheritance_underscore_v2.patch.gz

Version 2, which replaces the previous patch,

• Enables intersphinx for the Python docs. This is easy to use, e.g.,

For more information, please consult :mod:`subprocess`.  Blah, blah, whatever...

• Adds # -*- coding: utf-8 -*- to the top of sage.combinat.lyndon_word.
• Drops a sage.interfaces.singular.SingularFunctionElement._sage_doc_ doctest.
• Prepares for extlinks. Sphinx 1.0 will include this new extension.

[qed777]

comment:12

It seems that :show-inheritance: shows just the immediate super classes, not the entire hierarchy. I do not know whether this is easy to change.

[qed777]

comment:13

Please see #6495 for progress on incrementally building the reference manual.

[qed777]

Rebased for 4.3.alpha1. Replaces previous patches.

[qed777]

comment:14

Attachment: trac_7549-doc_inheritance_underscore_v3.patch.gz

I should add that we can also link directly to matplotlib's docs from the Sage documentation. We just need to update doc.common.conf.intersphinx_mapping (the object inventory is here). Perhaps there are others of interest in this list?

[qed777]

comment:15

Against 4.3.2.alpha1:

applying trac_7549-doc_inheritance_underscore_v3.patch
patching file doc/common/conf.py
Hunk #5 succeeded at 378 with fuzz 2 (offset 5 lines).
now at: trac_7549-doc_inheritance_underscore_v3.patch

[jhpalmieri]

comment:16

Replying to @qed777:

I should add that we can also link directly to matplotlib's docs from the Sage documentation. We just need to update doc.common.conf.intersphinx_mapping (the object inventory is here). Perhaps there are others of interest in this list?

Cython, mpmath, and Sphinx are possibilities. Building Sage shouldn't rely on an internet connection, though, and it seems to me that intersphinx does. So should we also include the object inventories so that we can build the docs without an internet connection? This would rely on keeping those up to date. Can we have it check for an internet connection and fail gracefully if there isn't one?

Right now I don't know what I'm doing wrong, but I can't get private methods to be included in the docs. This is with 4.3.2. I put in a print statement in skip_underscore to make sure it was finding the correct methods, and it was. But I didn't see them in the documentation afterwards. I'll try again later.

[qed777]

Rebased vs. 4.3.3.alpha0. See comment. Apply only this patch. sage repo.

[qed777]

comment:17

Attachment: trac_7549-doc_inheritance_underscore_v4.patch.gz

V4

• Rebased vs. 4.3.3.alpha0.
• Combines the skip member handlers, since Sphinx takes the first answer that's not None. (I suppose we could keep them separate but return None when a test is inconclusive).
• Detects changes in -u, too. Toggling either new option should trigger a rebuild.

Should we move the new options to the "Advanced" section? Add "may use lots of memory"?

Note: I still haven't done a complete build in each configuration.

[qed777]

comment:18

I should add that I've disabled the intersphinx extension, for now. It can cache inventories, but I don't know whether it fails gracefully if the cache has "expired" but there's no connection.

[qed777]

comment:19

The HTML manual builds with -i, -u, and -iu.

Building the PDF manual with -u on sage.math yields

[4811] [4812] [4813] [4814] [4815] [4816]
! TeX capacity exceeded, sorry [pool size=1165811].
\Hy@pstringdef ...->\edef #1{\pdfescapestring {#2}
                                                  }
l.392012 ...matrix0.Matrix.multiplicative_order}{}
                                                  \begin{methoddesc}{multipl...

!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on reference.log.
make: *** [reference.pdf] Error 1

after I suppress problems with underscored methods in

• sagenb.notebook.worksheet
• structure.formal_sum
• sage.combinat.crystals.tensor_product
• sage.combinat.species.set_species
• sage.combinat.words.word_generators
• sage.rings.number_field.number_field
• sage.rings.number_field.number_field_ideal
• sage.rings.polynomial.multi_polynomial_libsingular
• sage.rings.polynomial.infinite_polynomial_ring
• sage.rings.polynomial.infinite_polynomial_element

I'll fix the first problem in a separate ticket and leave the others for someone else.

[jhpalmieri]

comment:20

I get the html manual to build, too, although with 1517 warnings when built with "-u", and more than 28,000 warnings with "-iu": either because the docstrings for underscore methods are sloppily formatted (not a surprise) or messages along the lines of those in #8244.

I noticed that after I built with "-j -iu", edited a few files and ran "sage -b", then when I built the manual again with "-j -iu", in the "reading sources" portion of the build, it only processes the changed files, but in the "writing output" phase, it rewrites all of the files. This could be a little annoying; can it be avoided?

I haven't had the time to try a pdf build yet, but I'll work on that later today.

[jhpalmieri]

comment:21

Now I've tried building the pdf, and I run into the same "TeX capacity exceeded" message. I've tried altering "extra_mem_top" and "extra_mem_bot", but it's not helping. (I may not be doing this right, though.)

I had to fix more problems to get the PDF to build: everywhere there was an email address, pdflatex was crashing on the @ sign, so I had to enclose the address in double backquotes. So I ended up changing these files (in addition to sagenb.notebook.worksheet: I applied the patch from #8265 to deal with this one):

• sage/combinat/species/set_species.py
• sage/combinat/words/word_generators.py
• sage/misc/hg.py
• sage/misc/lazy_attribute.py
• sage/misc/preparser.py
• sage/rings/number_field/number_field.py
• sage/rings/number_field/number_field_ideal.py
• sage/rings/padics/padic_generic.py
• sage/rings/padics/tutorial.py
• sage/rings/polynomial/infinite_polynomial_element.py
• sage/rings/polynomial/infinite_polynomial_ring.py
• sage/rings/polynomial/multi_polynomial_libsingular.pyx
• sage/rings/polynomial/pbori.pyx
• sage/rings/polynomial/symmetric_ideal.py
• sage/rings/polynomial/symmetric_reduction.pyx
• sage/structure/formal_sum.py
• sage/structure/sequence.py

I don't think that fixing these is a high priority, since the PDF manual doesn't build anyway with the "-u" option.

---

Aside from my question about why "-iu" rewrites all of the output when it shouldn't have to, I'm happy with this. (I haven't tested this with "-i" or "-u".) I agree that there should be a comment about "-i" and "-u" taking a lot of memory, and perhaps saying that they don't work with the pdf build. I don't think they need to be in the "advanced" options.

[jhpalmieri]

comment:25

Replying to @qed777:

Replying to @jhpalmieri:

I noticed that after I built with "-j -iu", edited a few files and ran "sage -b", then when I built the manual again with "-j -iu", in the "reading sources" portion of the build, it only processes the changed files, but in the "writing output" phase, it rewrites all of the files. This could be a little annoying; can it be avoided?

I've also seen this behavior, but I've been unable reproduce it with certainty --- at least, with a version of #6187's '"testreference" patch'. Are there particular types of changes that usually trigger the problem?

I'm not sure, but I just saw this message at the beginning of a docbuild:

WARNING: unsupported build info format in '/Applications/sage/devel/sage/doc/output/html/en/reference/.buildinfo', building all

[qed777]

comment:26

I've seen this message, too. It might be a Sphinx bug.

The HTML builder writes a hash of the html_* settings to .buildinfo in the output directory. It appears that StandaloneHTMLBuilder.get_outdated_docs contains nearly all of the relevant code.

If, for some reason, this get_outdated_docs doesn't get called, the builder writes empty strings for the hashes. For example, sage -docbuild reference html -j -S -a does this (-a tells Sphinx to write all files). The next build attempt triggers the warning.

But I don't know if this is responsible for what we see.

[qed777]

Attachment: trac_7549-doc_inheritance_underscore_v5.patch.gz

Rebase vs. 4.3.4.alpha1 + #8457. Apply only this patch.

[qed777]

comment:27

V5 is rebased for 4.3.4.alpha1 + #8457. I'm not sure what we should do about the many -iu warnings. Should we handle them in sage_getargspec or in conf?

[jhpalmieri]

comment:28

If I run sage -docbuild reference html -i, I get

reading sources... [  8%] sage/algebras/iwahori_hecke_algebra_element
Exception occurred:
  File "/Applications/sage/devel/sage/doc/common/conf.py", line 354, in skip_member
    if 'SAGE_CHECK_NESTED' in os.environ:
RuntimeError: maximum recursion depth exceeded
The full traceback has been saved in ...

I scanned the traceback but didn't see anything obviously helpful.

By the way, I think we can put back the doctest from interfaces/singular.py, by changing the triple quotes at the beginning of the docstring to r""". The attached patch does this.

Also, I used all of the extra error messages to figure out where the warnings

<autodoc>:0: (ERROR/3) Unexpected indentation.

were coming from, and the attached patch fixes the problems. So now, combined with #8457, I get no error messages when building the reference manual (without the -i or -u switches, of course).

For the new warnings when building with -i or -u or -iu, I say we deal with them on another ticket. It's a big job revising all of the docstrings for underscore functions so that they are in proper reST format. (Also, right now they're useful -- see the previous paragraph. :)

Finally, do we want some sort of warning for the -i switch, and possibly the -u switch? ("may be slow, may not be compatible with PDF output" as part of its help string, for example)?

[jhpalmieri]

comment:29

(You may need to apply #8511 to get rid of one of the "unexpected indentation" warnings.)

[jhpalmieri]

comment:30

On second thought, I moved the "unexpected indentation" fix to ticket #8511, so the "small fixes" patch just reinstates the interfaces/singular.py doctest.

[jhpalmieri]

apply on top of other patch

[qed777]

Attachment: trac_7549-small-fixes.patch.gz

Attachment: trac_7549-doc_inheritance_underscore_v6.patch.gz

Fix recursion problem. Apply only this combined patch.

[qed777]

Changed author from Mitesh Patel to Mitesh Patel, Florent Hivert

[qed777]

comment:31

I broke the module check in #7448's V4 when I made #7448's V5. I apologize for that. I've restored Florent's code and folded in the small fixes patch in this ticket's V6.

[qed777]

comment:32

The only change is in sage_autodoc.py.

[qed777]

Attachment: trac_7549-doc_inheritance_underscore_v7.patch.gz

Help notes. Replaces V6.

[hivert]

comment:33

Replying to @qed777:

I broke the module check in #7448's V4 when I made #7448's V5. I apologize for that. I've restored Florent's code and folded in the small fixes patch in this ticket's V6.

Hi Mitesh ! I don't see any code from me here except a few lines who where already in #7448, and which you decide to improve (thanks by the way). So I don't think I deserve to be in the author.

Also I'm Ok with reviewing this but they are three more tickets in combinatoric which I committed myself to review. So don't rely on me for reviewing this very soon... However, I'd like to see this merged soon, so if you imagine someone who is willing to review, please ask him. Anyway, thanks for this work. I really this sage's doc is going to greatly improve.

[hivert]

Changed author from Mitesh Patel, Florent Hivert to Mitesh Patel

[jhpalmieri]

comment:34

Sorry for not getting to this sooner. Positive review.

[jhpalmieri]

Reviewer: John Palmieri

[jhpalmieri]

comment:36

Merged in 4.4.alpha0:

• trac_7549-doc_inheritance_underscore_v7.patch

[jhpalmieri]

Merged: sage-4.4.alpha0