Considerable slowdown in writing autodoc pages with sphinx 1.6.x #4230
Labels
Milestone
Comments
|
Hmm, I feel this is a very difficult issue. We only provides Note: Sadly, I failed to build pandas document on my local. It is very difficult to prepare the environment to me. |
|
Yes, the pandas docs are quite huge and need a complicated environment to fully build. But I will make a smaller test docs version that should be easier to test (and to provide logs for). |
|
I made a test repo with a small subset of our docs: https://github.com/jorisvandenbossche/pandas-test-docs. This should be easier to test out. Running this now with different versions of sphinx, will post the logs shortly. |
|
Below the logs for building the rest repo with 1.5 vs 1.6 (for the rest they were run in exactly the same environment, only the sphinx version differed). There are some warnings, but those can be ignored (unrelated issues we are trying to solve). With sphinx 1.6.5 (log-sphinx165.txt): With sphinx 1.5.6 (log-sphinx156.txt): |
|
Looking at the difference in the logs in the writing phase, I notice that with sphinx 1.5.6 the following events are emitted: while with sphinx 1.6.5 it is: So without the 'html-page-context' event. I am not familiar with this event, but this might indicate it is not reusing a template page but each time building the full page from scratch with sphinx 1.6.5? Were there changes related to this? (do we need to update some settings?) |
|
Trying to figure out further what is happening, I patched With 1.5.6: With 1.6.5: So with sphinx 1.6.5 writing a single page consistently takes quite some more time. |
15 tasks
|
I just tested this with sphinx master, and apparently this is solved there. It actually is faster than 1.5.6 (the writing part went down from 20s in 1.5.6 to 4s with master, while it was 60s on 1.6.5). Is there a timeline for the 1.7 release? |
|
@jorisvandenbossche I can't speak as to plans for 1.7 nor what patches might have resolved this. However, could I suggest using |
|
I had a try at this as well, with https://github.com/jorisvandenbossche/pandas-test-docs but didn't see the improvement @jorisvandenbossche found. Writing the API doc pages was as slow in with 1.7b as with
I'll try to profile the slowdown next week. |
|
@TomAugspurger Are you sure you're starting from a fresh environment each time? You should be deleting the entire build directory including doctrees. If you do have time to profile this, it would be much appreciated. |
tk0miya
added a commit
to tk0miya/sphinx
that referenced
this issue
Jan 30, 2018
``docutils.utils.new_document()`` is much slower than an instantiation of document node. This adds a much faster alternative of the function which uses caches internally.
|
I investigated what happened in writing (in As a result, I got following profile: This result says Furthermore, I also tested how heavy the function is. I think this is a cause of slowdown. |
|
Note: This is my dockerfile for this issue: |
tk0miya
added a commit
to tk0miya/sphinx
that referenced
this issue
Jan 30, 2018
``docutils.utils.new_document()`` is much slower than an instantiation of document node. This adds a much faster alternative of the function which uses caches internally.
tk0miya
added a commit
that referenced
this issue
Jan 30, 2018
Fix #4230: slowdown in writing document with sphinx 1.6
|
@tk0miya Thanks a lot for looking into this. I will try it out. |
|
I tried this out, but I unfortunately cannot test it, as it appears that sphinx dev version is not building our docs correctly. Almost all API doc pages are empty (and thus do build faster .. but without being good comparison). The reason seems to be that in the docstring pages generated by autosummary, all occurences of |
|
So trying to look through the log output of sphinx 1.7. It clearly mentions 'method' in that last line, but, it writes a file with As a comparison, the equivalent bits of the log file for sphinx 1.5.6: |
|
It seems generated HTML files does not change between 1.7.0b1 and latest. |
|
This is my dockerfile for the investigation. And below is a result of diff between 1.7.0b1 and 1.7-release branch (HEAD). |
|
@tk0miya shall I open a new issue for the API docs problem? |
|
ah, sorry. I closed this unintentionally (on merging to master). Thanks, |
|
I re-did timings with 1.5.6, 1.6.6, and 1.7 (release branch):
So current 1.7 release branch is indeed a lot faster as 1.6, although not fully as fast as 1.5 (I think that will enable us to use 1.7 for building the pandas docs) |
|
Good to know. |
|
Thank you for tracking it down!
…On Sat, Feb 3, 2018 at 8:39 AM, Takeshi KOMIYA ***@***.***> wrote:
Good to know.
I think next tuning point is autodoc module. But it is too complicated to
me. So it might be in 2.0 or 3.0.
Anyway, thank you for reporting. Please let me know if you have another
trouble. I'll do my best :-)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4230 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABQHIlkua3-XA7vwrI7rDdksQYKpb3lWks5tRBslgaJpZM4QZaCZ>
.
|
4 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
For the pandas docs, we currenlty pin sphinx to 1.5.x because we previously had problems with 1.6. I was now investigating what the problem exactly is preventing us to upgrade (pandas-dev/pandas#16705), and the main problem is a considerable slowdown in the writing phase.
Problem
Building the pandas docs went from ca 20 min to > 50 min, when switching from sphinx 1.5.6 to 1.6.5.
It is especially our api pages (autodoc pages for methods/attributes generated by autosummary) that are very slow in writing. In the progress output I can clearly see that each generated docstring page takes a noticeble 0.5 à 1.5 seconds to write (and since we have many of those pages (> 1000), this takes a long time)
Procedure to reproduce the problem
Building the pandas docs, although this is not a small reproducible step.
Does anybody have an idea where to start looking or how to best debug this?
Environment info
The text was updated successfully, but these errors were encountered: