Performance Tuning #1080
-
|
Hi Razon & Community, how to tune build performance? I've read that "caching" for templates may help, and maybe some tuning for taxonomy parameters? I've read in may boost build time 4 times; I currently have site with 6000 pages at Netlify, and build takes 40 minutes with "--minify" option, with all default settings as in site template. Thank you! |
Beta Was this translation helpful? Give feedback.
Replies: 21 comments 25 replies
-
|
6000 pages tooks 40min sounds very very very slow... The HB framework docs site is about 6k pages, but it take less than 2min (took 3min to deploy completely), you can check the latest workflow log here.
There are some hints to improve build performance.
|
Beta Was this translation helpful? Give feedback.
-
|
Btw, I'd recommend creating a minimal repo for checking if the caching is working, otherwise you'll exceed the limit of your Netlify plan soon enough and generate costs. Or using GitHub Actions instead if your repo is public, GitHub Actions is free for public repo. |
Beta Was this translation helpful? Give feedback.
-
|
On my laptop it is slow too, Apple M1 Max 64Gb, 25+ minutes What I found is mentions about templates/layout caching; not about GitHub roundtrips. 6,000 primary pages; but with pagination etc. much more; measuring now |
Beta Was this translation helpful? Give feedback.
-
Anyway it's unusal, however I couldn't help if I can't see the source. |
Beta Was this translation helpful? Give feedback.
-
|
One improvement clue is to increase paginate number to reduce pagination pages, it helps more or less. |
Beta Was this translation helpful? Give feedback.
-
|
https://discourse.gohugo.io/t/hugo-build-takes-too-long/28787/5 |
Beta Was this translation helpful? Give feedback.
-
|
using |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, I'm confused, you said 6000 pages (6k), but there is 40k pages, and took 14minutes, but the time seems reasonable to me, 816947/(34388+5977)≈20ms / page Btw, you should combine --templateMetrics and --templateMetricsHints flag, this output is not helpful, and I couldn't provide much help on it, I didn't have source to reproduce and improve template, and see timely response... |
Beta Was this translation helpful? Give feedback.
-
|
I created 6000 pages, and Hugo created 32,000 more (which include pagination, taxonomies, etc.); I use "blogs" layout for pages |
Beta Was this translation helpful? Give feedback.
-
|
Will take a closer look into it tomorrow (too late today, time to sleep), if you can provide your repo to see how it improve, it would be great, don't worry if it's not possible, I'll simulate it on my site, but with few content. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Razon, here is new log, huge numbers, I'll try to disable "search" and rerun, |
Beta Was this translation helpful? Give feedback.
-
|
I'll provide repo access tomorrow; good night |
Beta Was this translation helpful? Give feedback.
-
|
"_default/term.html" in the logs above is "terms" generation, and as I found it is for Taxonomies. The issue is with taxonomies; it is also seen from logs; when I disable both, Categories, and Tags, it is 15 seconds (instead of 800 seconds). When I enable Categories only, it is 50 seconds. I have mess with tags, few forms of the same word (plural, singular, etc.); I am going to disable tags for my site; I need Categories, and Related Terms (and "related terms" is not really "tag"; "profit" is term related to "loss" for example) |
Beta Was this translation helpful? Give feedback.
-
|
I see, will check this weekend fully. Let me know if your repo is ready. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Razon, I added you to https://github.com/tokenizer-inc-canada/www.tokenizer.ca
This repo currently has "tags" enabled (if I disable it, Google Search will have issues with 404 errors); on my laptop, full build takes 20-30 minutes, on Netlify - 30 minutes (and if I enable "--minify" it will be 35-40 minutes) When I disable "tags" (but leave "categories" enabled) build takes 50 seconds on laptop!!! My plan is to write program which will do statistical analysis and remove "least frequent tags" and also normalize words: "Equities, Equity, equities, equity -> equity", and if I have category "Equity" I shouldn't have such tag. Automated tool can do that... 6000+ pages. Also, https://tokenizer.ca/tags/ page is extremely huge and Chrome simply throws error, I am searching a way to limit its' size. Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Made some improvements (not release yet), now it takes 15-16min to build site (minify enabled) on my WSL2 (half RAM, 16GiB). Will dive into it deeper.
It will list all tags in one page, no pagination, but I don't see any error on browser console. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for looking into it; yes, this site doesn't have errors with tags now. I had larger site before with messy tags and tags page was huge 6000+ tags and browser simply was throwing errors. On my Mac M1 Pro build with tags enabled takes abut 800-850 seconds (14 minutes); and with tags disabled it takes only 50 seconds (with "categories" taxonomy enabled). So, the issue is "cardinality", we should not have tags linked to single document only. Another issue is that that I have category names almost same as tags names. I can build app in Java which will make statistical analysis and remove unnecessary tags. So, it is issue from core Hugo I believe, and not from the framework, in the logs above we have "term.html" and "terms.html" files creation takes 12 minutes, I believe it is "terms dictionary" which Hugo needs to build tags. I tried to disable "Search" module, but it is not an issue, "Search" generation saves only minute, it is fast. Also, I have Quizzes attached on each page, it uses custom shortcode & JavaScript library which relies on strict HTML formatting to work properly, so that I disable "--minify" to make it work. I am wondering if we can have shortcode for that in HB, with a fix; I am not so good with Hugo, still learning... Thank you Razon! |
Beta Was this translation helpful? Give feedback.
-
Yes, since each term page is a section, which are similar to pagination page that list relevant pages. One taxonomy page will add
I think it probably is an issue of this framework, the flexible hooks system may cost too much build time, need time to debug, the
There is minify configuration on Hugo, you might need to take a look on official docs, I didn't heard about the Quizzes, couldn't help. Btw, those improvement are released, |
Beta Was this translation helpful? Give feedback.
-
|
I tried improved version with Netlify, with "minify" enabled, build takes 26 minutes now instead of previous 38, this is significant, thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Just made some significant improvement, now it takes 6min to generate 37k pages, build speed is good now, less than 10ms per page. Upgrade via
I'm closing this for now, no much things can be improved, ideas are welcome if you found any. You can remove me from your repo now. |
Beta Was this translation helpful? Give feedback.
-
|
Netlify full build (including publishing) dropped to 10 minutes 33 seconds! |
Beta Was this translation helpful? Give feedback.
-
|
Btw, you can use
|
Beta Was this translation helpful? Give feedback.
-
|
Made some changes, which may save 1min for your case.
This might be last reply on this topic, keep modules up to date is always recommended. |
Beta Was this translation helpful? Give feedback.
-
|
I am not sure why, but one of my sites has this error now:
|
Beta Was this translation helpful? Give feedback.
-
|
Upgrading now: Will publish result soon |
Beta Was this translation helpful? Give feedback.
-
|
Build failed after 40 minutes, I think it is "search" module so I am disabling it: |
Beta Was this translation helpful? Give feedback.
-
|
Seem weird, I couldn't reproduce it from any of my sites... And I didn't see any useful error from the log info.. |
Beta Was this translation helpful? Give feedback.
-
|
One possible reason is that there is some shortcodes/templates try to read page content from each other, for example,
Anyway it's hard to debug without seeing your source code to reproduce, but you can check the recent changes of your site if the site works perfectly in the past, since the search module doesn't change much things recently. |
Beta Was this translation helpful? Give feedback.
-
|
Yes I found what caused that, I found it after disabling "Search". It was "#" in keywords section of frontmatter, such as I'll try to enable search and to rerun, I think "search" was in a loop because of that. Netlify filesystem doesn't support "#" as a starting character of generated file. I didn't have this issue locally. |
Beta Was this translation helpful? Give feedback.
-
|
From the docs, I found this:
Trying it now... it works locally. |
Beta Was this translation helpful? Give feedback.
-
|
I doubt that it would help, since the build failed on executing the first line of search index template. Does it work after removing search module (you can simply remove If you have some custom shortcodes/templates, please check if those shortcodes/templates read some variables like |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it works now; I had Full build at Netlify with Search enabled and upgraded libraries took 45 minutes; it is large site. |
Beta Was this translation helpful? Give feedback.
-
|
Glad it work. Btw how much pages (including pagination pages and section pages) on your site (just for measuring the build speed)? |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
Seems there are much less aliases than previous (1371 vs19205), is that normal? Btw, I'm curious about why there are so many alias pages. Tags and categories are rendered by same template, shouldn't be much difference, but previous build speed was 38ms/page, the later is |
Beta Was this translation helpful? Give feedback.
-
|
If the aliases are neccessary, I'd recommend doing it on Netlify side (rewrite rules or something else, I remember there is such functionality, but forgot the name) as far as possible instead of using Hugo aliases, it would save build time for large site. |
Beta Was this translation helpful? Give feedback.
-
|
I don't use aliases; it comes from Theme I guess? I use only plain pages with taxonomies in front matter |
Beta Was this translation helpful? Give feedback.
-
|
yes, tags vs categories: if I have 1000 categories such as "Finance", "Investments" then it is fast; 1000 pages auto-generated for each category; but if I have tags such as [financial, finances, financing, finance, investing, invest, investment, ...] than I guess it could easily become very bad... 10x worse I think... |
Beta Was this translation helpful? Give feedback.
-
|
Hugo creates alias pages... I don't even know what is it ;) |
Beta Was this translation helpful? Give feedback.
-
|
I can try this, https://gohugo.io/getting-started/configuration/#disablealiases I think Hugo can monitor changes and generate aliases (301 redirects, etc) - Netlify specific; but on my laptop, clean build, I had zero files, then 13,000 iles, so it generates 13,000 aliases I can try to disable it to test |
Beta Was this translation helpful? Give feedback.
-
|
Just tested on my site, when I do a global search on the output (
Those aliases are created even disableAliases is set as Will check if there is change to improve. |
Beta Was this translation helpful? Give feedback.
-
|
Here is my largest site, built on Macbook Pro M1, "Categories" taxonomy enabled, and pagination is by 1000 pages. About 17ms/page. If I disable "category" taxonomy, it will become less than 5ms/page. |
Beta Was this translation helpful? Give feedback.
-
|
Will dive into the terms and term layout deeper if find some time. |
Beta Was this translation helpful? Give feedback.
-
|
Let's track this on hbstack/blog#884, since this topic has grown longer and longer, and it's not reverant to HugoMods. |
Beta Was this translation helpful? Give feedback.
Just made some significant improvement, now it takes 6min to generate 37k pages, build speed is good now, less than 10ms per page.
Upgrade via
hugo mod get -u.I'm closing this for now, no much things can be improved, ideas are welcome if you found any.
You can remove me from your repo now.