Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Very long parse time for simple template #44

Closed
larros opened this issue Sep 2, 2017 · 2 comments · Fixed by #45
Closed

Very long parse time for simple template #44

larros opened this issue Sep 2, 2017 · 2 comments · Fixed by #45

Comments

@larros
Copy link
Contributor

larros commented Sep 2, 2017

I've been using Askama for while and I think it's great.
However I've found one case where some of my templates take a very long time to compile. Now when I finally took some time to investigate it seems to be related to a specific construct in the templates that leads to very long time spent in the parser.

Compilation of the following code takes about a minute for me to compile before it fails.

#[macro_use]
extern crate askama;

use askama::Template;

#[derive(Template)]
#[template(source = "{{ a.b(c.d()) }}")]
struct MyTemplate {

}

fn main() {
    let template = MyTemplate {};
    println!("{}", template.render())
}

Profiling of similar case using 0.3.4 shows that most of the time is spent in the parser.

Total number in stack (recursive counted multiple, when >=5):

        87736       askama_derive::parser::expr_filtered  (in libaskama_derive-3ae4a4b851656feb.dylib) + 189  [0x118c74f7d]  parser.rs:189
        47349       askama_derive::parser::expr_attr  (in libaskama_derive-3ae4a4b851656feb.dylib) + 1711  [0x118c72a8f]  parser.rs:166
        47134       askama_derive::parser::attr  (in libaskama_derive-3ae4a4b851656feb.dylib) + 2242  [0x118c71e12]  parser.rs:159
        46629       askama_derive::parser::arguments  (in libaskama_derive-3ae4a4b851656feb.dylib) + 2276  [0x118c6add4]  parser.rs:126
        45844       askama_derive::parser::expr_addsub  (in libaskama_derive-3ae4a4b851656feb.dylib) + 248  [0x118c7ac18]  parser.rs:223```
@defyrlt
Copy link
Contributor

defyrlt commented Sep 2, 2017

Can confirm - your repo takes me 90 seconds to compile in dev mode & 9 in release mode. I don't know if parser itself can be optimized, but something like this will help (it's not implemented yet).
But I think it should be possible to optimize parser. 9 seconds in release mode is very long for such a small thing.

@larros
Copy link
Contributor Author

larros commented Sep 3, 2017

I've looked into this some more and it looks like a general problem with the expr_prec_layer macro in the parser. Every call to expr_any ends up in 512 calls to expr_filtered. This is usually not a big issue but in my case when I have two levels of method calls this grows quadratically.

I created some test cases and benchmarks during my investigation that can be found here
larros@b65246a

Entering the expr_prec_layer chain in different places gives huge differences.
test parser::tests::bench_expr_any ... bench: 37,446,586 ns/iter (+/- 2,987,389)
test parser::tests::bench_expr_bxor ... bench: 2,114,902 ns/iter (+/- 80,783)
test parser::tests::bench_expr_muldivmod ... bench: 130,346 ns/iter (+/- 8,045)

larros referenced this issue in larros/askama Sep 3, 2017
This implementation resolves djc/askama#44 by changing the precedence
implementation.
The previous solution was very slow because it had to try to parse
all combinations of precedence layers leading to 2^9 iterations for each
expr_any. This is solved by reusing the left operand instead of reparsing
it when the operator isn't found.

This implementation also solves another related issue that more expressions
with multiple operators couldn't be parsed. This is handled by using
expr_any for right instead only using higher level precedence layers.
larros referenced this issue in larros/askama Sep 3, 2017
This implementation resolves djc/askama#44 by changing the precedence
implementation.
The previous solution was very slow because it had to try to parse
all combinations of precedence layers leading to 2^9 iterations for each
expr_any. This is solved by reusing the left operand instead of reparsing
it when the operator isn't found.

This implementation also solves another related issue that expressions
with multiple operators couldn't be parsed, for example {{1 * 2 * 3}}.
This is handled by using expr_any for right instead only using higher
level precedence layers.
larros referenced this issue in larros/askama Sep 3, 2017
This implementation resolves djc/askama#44 by changing the precedence
implementation.
The previous solution was very slow because it had to try to parse
all combinations of precedence layers leading to 2^9 iterations for each
expr_any. This is solved by reusing the left operand instead of reparsing
it when the operator isn't found.

This implementation also solves another related issue that expressions
with multiple operators couldn't be parsed, for example {{1 * 2 * 3}}.
This is handled by using expr_any for the right operand instead of only
using higher level precedence layers.
@djc djc closed this as completed in #45 Sep 3, 2017
djc referenced this issue Sep 3, 2017
This implementation resolves djc/askama#44 by changing the precedence
implementation.
The previous solution was very slow because it had to try to parse
all combinations of precedence layers leading to 2^9 iterations for each
expr_any. This is solved by reusing the left operand instead of reparsing
it when the operator isn't found.

This implementation also solves another related issue that expressions
with multiple operators couldn't be parsed, for example {{1 * 2 * 3}}.
This is handled by using expr_any for the right operand instead of only
using higher level precedence layers.
busgaidw2 referenced this issue in busgaidw2/askama Aug 6, 2024
This implementation resolves djc/askama#44 by changing the precedence
implementation.
The previous solution was very slow because it had to try to parse
all combinations of precedence layers leading to 2^9 iterations for each
expr_any. This is solved by reusing the left operand instead of reparsing
it when the operator isn't found.

This implementation also solves another related issue that expressions
with multiple operators couldn't be parsed, for example {{1 * 2 * 3}}.
This is handled by using expr_any for the right operand instead of only
using higher level precedence layers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants