Improve build performance of docs site [HELP NEEDED] #44

rdwatters · 2017-03-18T18:14:19Z

@digitalcraftsman @moorereason @bep @budparr

What's the target build time for building the docs site per @bep 's comment that the docs site is THE representative site for Hugo? (This makes sense for dogfooding.)

Here's on my mid-2012 Macbook Pro, 500gb SSD, 2.6 GHz Intel Core i7, 8 GB 1600 MHz DDR3:

Ryans-MacBook-Pro-2:docs-concept ryanwatters$ hugo --stepAnalysis --renderToMemory
Started building sites ...
Go initialization:
	1.113346949s (1.113802966s)	  251.30 MB 	122794 Allocs
initialize:
	212.547µs (1.114093772s)	    0.17 MB 	278 Allocs
load data:
	21.08919ms (1.135273377s)	    4.41 MB 	139940 Allocs
load i18n:
	200ns (1.135339432s)	    0.00 MB 	0 Allocs
read pages from source:
	113.622354ms (1.249112786s)	   11.29 MB 	109481 Allocs
convert source:
	27.943266ms (1.277214141s)	   19.61 MB 	163911 Allocs
build Site meta:
	2.149305ms (1.279452647s)	    0.33 MB 	12103 Allocs
prepare pages:
	34.191505ms (1.313856577s)	   43.80 MB 	176257 Allocs
render and write aliases:
	5.693574ms (1.319670655s)	    0.90 MB 	16795 Allocs
render and write pages:
	2.889621211s (4.209774815s)	 1952.11 MB 	38189751 Allocs
render and write Sitemap:
	17.709624ms (4.227989034s)	    1.26 MB 	35400 Allocs
render and write robots.txt:
	11.819µs (4.228386929s)	    0.00 MB 	8 Allocs
render and write 404:
	23.416997ms (4.252246753s)	    3.88 MB 	67634 Allocs
Built site for language en:
0 of 7 drafts rendered
0 future content
0 expired content
295 regular pages created
259 other pages created
2 non-page files copied
0 paginator pages created
242 tags created
total in 3138 ms

My typical results running the local server:

0 of 7 drafts rendered
0 future content
0 expired content
295 regular pages created
259 other pages created
2 non-page files copied
0 paginator pages created
242 tags created
total in 3277 ms

And when I switch {{- partial "site-navigation.html" . -}} in baseof.html to {{- partialCached "site-navigation.html" . -}}:

Ryans-MacBook-Pro-2:docs-concept ryanwatters$ hugo --stepAnalysis --renderToMemory
Started building sites ...
Go initialization:
	174.639389ms (175.118835ms)	  251.31 MB 	122831 Allocs
initialize:
	195.392µs (175.403359ms)	    0.17 MB 	278 Allocs
load data:
	18.366801ms (193.893558ms)	    4.41 MB 	139937 Allocs
load i18n:
	241ns (193.974596ms)	    0.00 MB 	0 Allocs
read pages from source:
	21.789181ms (215.906723ms)	   11.41 MB 	109543 Allocs
convert source:
	27.771987ms (243.833173ms)	   19.75 MB 	163959 Allocs
build Site meta:
	2.178849ms (246.099917ms)	    0.32 MB 	12096 Allocs
prepare pages:
	34.853278ms (281.164449ms)	   43.67 MB 	176243 Allocs
render and write aliases:
	6.155326ms (287.483728ms)	    0.90 MB 	16801 Allocs
render and write pages:
	269.83084ms (557.55377ms)	  422.77 MB 	1681913 Allocs
render and write Sitemap:
	13.75197ms (571.515039ms)	    1.26 MB 	35401 Allocs
render and write robots.txt:
	36.122µs (571.749674ms)	    0.00 MB 	8 Allocs
render and write 404:
	1.080643ms (572.98897ms)	    0.55 MB 	1354 Allocs
Built site for language en:
0 of 7 drafts rendered
0 future content
0 expired content
295 regular pages created
259 other pages created
2 non-page files copied
0 paginator pages created
242 tags created
total in 398 ms

I take this to mean that the navigation is the biggest offender, but I'm not sure how to cache/variant this in a way that still gets the desired output w/r/t active classes, etc. I am not using Hugo menus and instead pulling this info from this data file.

The text was updated successfully, but these errors were encountered:

bep · 2017-03-18T18:46:34Z

I am not using Hugo menus and instead pulling this info from this data file.

This is about eating the dog food.

We have a menu system, so use it. If the menu system does not do what you want, let us fix it.

rdwatters · 2017-03-18T19:36:50Z

This is about eating the dog food.
We have a menu system, so use it. If the menu system does not do what you want, let us fix it.

Please see my comments about the original intention in #45. I think Hugo's system works fine, but I had different goals in mind. I agree though that dogfooding is important.

With the topic/title of this issue in mind, will this have a positive effect on the build?

bep · 2017-03-18T19:43:45Z

With the topic/title of this issue in mind, will this have a positive effect on the build?

Yes.

bep · 2017-03-18T19:45:53Z

The main performance issue with the nav partial is the "range where .Site.Pages ...". We should fix this in Hugo by adding a cache to this and similar template funcs, but even then I would say that the Hugo Docs site SHOULD use the built-in menu system, even if we loose some functionality.

moorereason · 2017-03-19T04:20:56Z

Did a quick hack to show (I think) timings for all partial templates.

These number are CPU time. Due to concurrency, we have multiple jobs and timers running at the same time. So, the site-nav clocks in at 37s, which is 3x the total build time of 11s (yes, my old machine is slow).

Started building sites ...
Go initialization:
        307.913249ms (308.852184ms)       235.86 MB     114574 Allocs
initialize:
        283.972µs (309.425923ms)            0.17 MB     280 Allocs
load data:
        34.587859ms (344.198022ms)          4.03 MB     139933 Allocs
load i18n:
        838ns (344.513491ms)        0.00 MB     0 Allocs
read pages from source:
        46.229286ms (391.159795ms)         11.45 MB     110487 Allocs
convert source:
        89.720896ms (481.560029ms)         19.39 MB     164057 Allocs
build Site meta:
        4.010658ms (486.110484ms)           0.33 MB     12139 Allocs
prepare pages:
        130.741201ms (617.466142ms)        45.26 MB     176074 Allocs
render and write aliases:
        10.241475ms (628.181205ms)          0.91 MB     17175 Allocs
render and write pages:
        10.665247545s (11.294332486s)    2018.03 MB     38316589 Allocs
render and write Sitemap:
        26.694281ms (11.322058659s)         1.26 MB     35459 Allocs
render and write robots.txt:
        20.324µs (11.322716906s)            0.00 MB     8 Allocs
render and write 404:
        33.123724ms (11.356740035s)         3.42 MB     67717 Allocs

   cumulative       average           max  count  template
  17.426388ms      56.033µs     110.348µs    311  partials/breadcrumb.html
  133.29855ms     429.995µs   20.477991ms    310  partials/content-footer.html
  35.544537ms     114.659µs   20.227264ms    310  partials/content-header-links.html
 273.131429ms    1.071103ms   32.086593ms    255  partials/contents-list.html
  29.573943ms   29.573943ms   29.573943ms      1  partials/functions-quick-reference.html
   1.251257ms        2.25µs      44.349µs    556  partials/head/metadata-favicons.html
  108.42969ms     195.017µs   20.240813ms    556  partials/head/metadata-opengraph.html
 137.126233ms     246.629µs   40.470452ms    556  partials/head/metadata-prefetch.html
  13.592142ms      24.446µs     105.669µs    556  partials/head/metadata-refreshredirect.html
  34.548345ms      62.137µs   20.197651ms    556  partials/head/metadata-rss.html
 101.714263ms     182.939µs   16.059325ms    556  partials/head/metadata-schemaorg.html
 110.214109ms     198.226µs    40.34383ms    556  partials/head/metadata-standard.html
  30.015338ms      53.984µs     685.135µs    556  partials/head/metadata-twitter.html
   5.923528ms      10.653µs      47.911µs    556  partials/head/noindex.html
 283.190581ms     509.335µs   30.531385ms    556  partials/head/site-style.html
    777.673µs     777.673µs     777.673µs      1  partials/homepage/homepage-features.html
     38.063µs      38.063µs      38.063µs      1  partials/homepage/homepage-focus.html
     77.803µs      77.803µs      77.803µs      1  partials/homepage/homepage-header.html
    1.75977ms     1.75977ms     1.75977ms      1  partials/homepage/homepage-hero.html
     262.53µs      262.53µs      262.53µs      1  partials/homepage/homepage-install.html
     75.358µs      75.358µs      75.358µs      1  partials/homepage/homepage-open-source.html
   1.006401ms    1.006401ms    1.006401ms      1  partials/homepage/homepage-tweets.html
  65.172886ms     220.178µs   34.478838ms    296  partials/page-title.html
  38.014923ms      68.372µs    1.735186ms    556  partials/site-footer.html
 1.068295033s    1.921393ms   41.210201ms    556  partials/site-head.html
  20.939787ms      37.729µs      87.371µs    555  partials/site-header.html
37.225429745s   66.952211ms  158.064466ms    556  partials/site-navigation.html
  13.346104ms      24.003µs    1.664718ms    556  partials/site-scripts.html
   6.145054ms      11.052µs      55.872µs    556  partials/site-search.html
  26.207012ms      47.134µs    3.801556ms    556  partials/style-embed.html
     359.32µs       2.681µs      90.234µs    134  partials/svg-icons/exclamation.svg
    138.844µs       8.167µs      88.767µs     17  partials/svg-icons/godoc-icon.html
    114.119µs     114.119µs     114.119µs      1  partials/svg-icons/gopher-homepage.svg
   1.038598ms    1.038598ms    1.038598ms      1  partials/svg-icons/hugo-logo-wide.svg
   4.559109ms      15.402µs      90.792µs    296  partials/table-of-contents.html
  31.405793ms     101.309µs   20.239975ms    310  partials/tags.html

Built site for language en:
0 of 7 drafts rendered
0 future content
0 expired content
296 regular pages created
259 other pages created
2 non-page files copied
0 paginator pages created
242 tags created
total in 11048 ms

rdwatters · 2017-03-19T15:32:58Z

Bothers me that you can't emoji-response on mobile GH. Yowzer, @moorereason Slow machine or not, no reason for this single partial to rebuild at almost 3 times the speed of the entire Smashing site. I will refactor to use menus.

The current Hugo site has next/prev set manually in the front matter, but I think this is better taken care of at the template level so that authors can focus just on the content piece. Thoughts?

bep · 2017-03-19T15:41:03Z

Thoughts?

Yes, that should not be in the content files.

moorereason · 2017-03-19T18:28:04Z

Ryan, this sounds similar to your desire to sort menu children by a field of the child pages. Hence, your use of the YAML data file.

Are you asking for the same feature for Next/Prev (or, more probably, NextInSection/PrevInSection)?

rdwatters · 2017-03-19T18:38:39Z

Are you asking for the same feature for Next/Prev (or, more probably, NextInSection/PrevInSection)?

Not entirely, although you're a bit of a mind-reader since I was thinking about a feature request for something a la .NextByTitle or .NextByDate or .NextByPublishDate, etc after my experience building the docs (different thread for sure). For example, the date in all the functions files is the same...

I'm referring to the front matter that is in the content files of the current docs site and whether you guys think this is worthwhile for me to consider as part of this refactoring. I think it asks a bit too much of the author to set next/prev manually, especially in longer sections, and eventuates in sloppy nav and tethers content metadata to presentation. But both of you also have much more experience with the PRs for the docs thus far.

Honestly, I've never built anything with Hugo's menu feature, so this is a good learning opportunity and also why I can't fully weigh in on any shortcomings/strengths of the feature. That said, if we think it will improve the perf on the docs build for me to change this to the menu, I'm all for it.

n10v · 2017-04-06T18:11:20Z

Why do you need so many markdownify functions?

$ grep -nr 'markdownify' .
./themes/hugo-docs-concept/layouts/_default/section.html:8:			{{- .Title | markdownify -}}</h1>
./themes/hugo-docs-concept/layouts/old.html:18:					<p>{{ $values.desc | markdownify}}</p>
./themes/hugo-docs-concept/layouts/partials/content-footer.html:9:			<h5 class="{{$section}}">{{if eq .Section "functions"}}{{- replace (.NextInSection.LinkTitle) "and" "<em>and</em>" | safeHTML -}}{{else}}{{.NextInSection.Title | markdownify}}{{end}}</h5>
./themes/hugo-docs-concept/layouts/partials/content-footer.html:18:				<h5 class="{{$section}}">{{if eq .Section "functions"}}{{- replace (.LinkTitle) "and" "<em>and</em>" | safeHTML -}}{{else}}{{.Title | markdownify}}{{end}}</h5>
./themes/hugo-docs-concept/layouts/partials/content-footer.html:27:			<h5 class="{{$section}}">{{if eq .Section "functions"}}{{- replace (.LinkTitle) "and" "<em>and</em>" | safeHTML -}}{{else}}{{.Title | markdownify}}{{end}}</h5>
./themes/hugo-docs-concept/layouts/partials/contents-list.html:8:				<h3 class="list-icon {{.Section}}">{{.Title | markdownify}}{{with .Params.sitelink }}&nbsp;<i class="icon-link-external" aria-hidden="true"></i>{{end}}</h3>
./themes/hugo-docs-concept/layouts/partials/contents-list.html:10:			<p>{{.Description | markdownify }}</p>
./themes/hugo-docs-concept/layouts/partials/functions-quick-reference.html:4:	<dd>{{ .Description | markdownify }}</dd>
./themes/hugo-docs-concept/layouts/partials/homepage/homepage-features.html:6:			{{ .description | markdownify }}
./themes/hugo-docs-concept/layouts/partials/page-title.html:5:		{{- .Title | markdownify -}}
./themes/hugo-docs-concept/layouts/partials/site-navigation.html:23:					<li class="submenu-item"><a class="submenu-item-link {{$section}}{{if eq ($.Scratch.Get "url") .URL}} active-page{{end}}" href="{{.Permalink}}">{{with .LinkTitle}}{{. | markdownify }}{{else}}{{.Title | markdownify}}{{end}}</a></li>
./themes/hugo-docs-concept/layouts/partials/site-navigation.html:27:					<li class="submenu-item"><a class="submenu-item-link {{$section}}{{if eq ($.Scratch.Get "url") .URL}} active-page{{end}}" href="{{with .Params.redirecturl}}{{.}}{{else}}{{.Permalink}}{{end}}">{{with .LinkTitle}}{{. | markdownify }}{{else}}{{.Title | markdownify}}{{end}}</a></li>
./themes/hugo-docs-concept/layouts/section/showcase.html:5:		<h1 class="page-title {{if eq .Kind "page"}}{{.Section}}{{end}}" id="{{with .LinkTitle}}{{. | urlize}}{{else}}{{ .Title | urlize }}{{end}}">{{with .LinkTitle}}{{.}}{{else}}{{ .Title | markdownify }}{{end}}</h1>
./themes/hugo-docs-concept/layouts/section/showcase.html:10:		{{$title := .Title | markdownify}}
./themes/hugo-docs-concept/layouts/shortcodes/datatable.html:12:		<td><a href="{{$art.url}}" target="_blank">{{$art.title | markdownify }}</a></td>
./themes/hugo-docs-concept/layouts/shortcodes/datatable.html:13:		<td>{{ $art.author | markdownify }}</td>
./themes/hugo-docs-concept/layouts/shortcodes/note.html:5:	<!-- <h2 id="{{if .Get 0}}{{.Get 0 | urlize}}{{else}}note{{end}}">{{if .Get 0}}{{.Get 0 | markdownify}}{{else}}Note{{end}}</h2> -->
./themes/hugo-docs-concept/layouts/shortcodes/readfile.html:3:{{- $file  | readFile | markdownify -}}
./themes/hugo-docs-concept/layouts/shortcodes/tip.html:5:	<!-- <h2 id="{{if .Get 0}}{{.Get 0 | urlize}}{{else}}tip{{end}}">{{if .Get 0}}{{.Get 0 | markdownify}}{{else}}Tip{{end}}</h2> -->
./themes/hugo-docs-concept/layouts/shortcodes/warning.html:5:	<!-- <h2 id="{{if .Get 0}}{{.Get 0 | urlize}}{{else}}warning{{end}}">{{if .Get 0}}{{.Get 0 | markdownify}}{{else}}Warning{{end}}</h2> -->
./themes/hugo-docs-concept/layouts/shortcodes/yt.html:10:<div class="video-description">{{ .Get "description" | markdownify }}</div>

budparr · 2017-04-06T18:23:13Z

I should report this here before anyone spends time on the docs code exist now. We're building a new theme for the docs. Here are the benchmarks for old and new:

Old (Hugo Docs Concept) Theme

Average time per operation: 6257ms
Average memory allocated per operation: 2147846kB
Average allocations per operation: 39609881

New Theme

Average time per operation: 1372ms
Average memory allocated per operation: 634227kB
Average allocations per operation: 8806877

rdwatters · 2017-04-06T18:39:19Z

Average time per operation: 1372ms

This just made my day. Thanks so much, @budparr !

Also, thanks @BoGeM for looking into the perf-related issues. I should have updated you sooner that Bud was working on a new theme and hope this didn't waste your time. Cheers.

rdwatters · 2017-04-30T22:24:00Z

Seeing as @budparr 's theme has shown considerable improvements for site build times, I'm going to go ahead and close this.

rdwatters mentioned this issue Mar 18, 2017

Homepage redesign #3

Closed

rdwatters mentioned this issue Mar 18, 2017

Refactor site-navigation.html partial to use Hugo's built-in menu system... #45

Closed

rdwatters mentioned this issue Mar 19, 2017

Docs Design Discussion #43

Closed

rdwatters added the help wanted label Apr 3, 2017

n10v mentioned this issue Apr 6, 2017

hugolib, tpl: Get rid of some redundant viper calls gohugoio/hugo#3292

Closed

rdwatters closed this as completed Apr 30, 2017

rdwatters mentioned this issue May 8, 2017

Use of definition list for functions Quick Reference... gohugoio/gohugoioTheme#19

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve build performance of docs site [HELP NEEDED] #44

Improve build performance of docs site [HELP NEEDED] #44

rdwatters commented Mar 18, 2017

bep commented Mar 18, 2017

rdwatters commented Mar 18, 2017 •

edited

Loading

bep commented Mar 18, 2017

bep commented Mar 18, 2017

moorereason commented Mar 19, 2017

rdwatters commented Mar 19, 2017

bep commented Mar 19, 2017

moorereason commented Mar 19, 2017

rdwatters commented Mar 19, 2017

n10v commented Apr 6, 2017 •

edited

Loading

budparr commented Apr 6, 2017 •

edited

Loading

rdwatters commented Apr 6, 2017

rdwatters commented Apr 30, 2017

Improve build performance of docs site [HELP NEEDED] #44

Improve build performance of docs site [HELP NEEDED] #44

Comments

rdwatters commented Mar 18, 2017

bep commented Mar 18, 2017

rdwatters commented Mar 18, 2017 • edited Loading

bep commented Mar 18, 2017

bep commented Mar 18, 2017

moorereason commented Mar 19, 2017

rdwatters commented Mar 19, 2017

bep commented Mar 19, 2017

moorereason commented Mar 19, 2017

rdwatters commented Mar 19, 2017

n10v commented Apr 6, 2017 • edited Loading

budparr commented Apr 6, 2017 • edited Loading

rdwatters commented Apr 6, 2017

rdwatters commented Apr 30, 2017

rdwatters commented Mar 18, 2017 •

edited

Loading

n10v commented Apr 6, 2017 •

edited

Loading

budparr commented Apr 6, 2017 •

edited

Loading