Dynamic templates

Dynamic replacement for quicktemplate template engine.

Retrospective

We've used for a long time quicktemplate for building JSON to interchange data between microservices on high-load project, and we were happy. Ерут we need to be able to change existing templates or add new templates on the fly. Unfortunately quicktemplates doesn't support this and this package was developed as a replacement.

It reproduces many of qtpl features and syntax.

How it works

Working in templates is divided into two phases - parsing and templating. The parsing phase builds from template a tree (like AST) and registers it in templates registry by unique name afterward. This phase isn't intended to be used in highload conditions due to high pressure to cpu/mem. The second phase - templating, against intended to use in highload.

Templating phase required a preparation to pass data to the template. There is a special object Ctx, that collects variables to use in template. Each variable must have three params:

• unique name
• data - anything you need to use in template
• inspector type

Inspector type must be explained.

The biggest problem during development was how to get data from an arbitrary structure without using reflection, since reflect package produces a lot of allocations by design and is extremely slowly in general.

To solve that problem was developed a code-generation framework inspector framework. It provides primitive methods to read data from struct's fields, iterating, etc. without using reflection and makes it fast. Framework generates for each required struct a special type with such methods. It isn't a pure dynamic like reflect provided, but works so fast and makes zero allocations.

You may check example of inspectors in package testobj_ins that represent testing structures in testobj.

Usage

The typical usage of dyntpl looks like this:

package main

import (
	"bytes"

	"github.com/koykov/dyntpl"
	"path/to/inspector_lib_ins"
	"path/to/test_struct"
)

var (
	// Fill up test struct with data.
	data = &test_struct.Data{
		// ...
	}

	// Template code.
	tplData = []byte(`{"id":"{%=data.Id%}","hist":[{%for _,v:=range data.History separator ,%}"{%=v.Datetime%}"{%endfor%}]}`)
)

func init() {
	// Parse the template and register it.
	tree, _ := dyntpl.Parse(tplData, false)
	dyntpl.RegisterTplKey("tplData", tree)
}

func main() {
	// Prepare output buffer
	buf := bytes.Buffer{}
	// Prepare dyntpl context.
	ctx := dyntpl.AcquireCtx()
	ctx.Set("data", data, inspector_lib_ins.DataInspector{})
	// Execute the template and write result to buf.
	_ = dyntpl.Write(&buf, "tplData", ctx)
	// Release context.
	dyntpl.ReleaseCtx(ctx)
	// buf.Bytes() or buf.String() contains the result.
}

Content of init() function should be executed once (or periodically on the fly from some source, eg DB).

Content of main() function is how to use dyntpl in a general way in highload. Of course, byte buffer should take from the pool.

Benchmarks

See bench.md for result of internal benchmarks.

Highly recommend to check *_test.go files in the project, since they contains a lot of typical language constructions that supports this engine.

See versus/dyntpl for comparison benchmarks with quicktemplate and native marshaler/template.

As you can see, dyntpl in ~3-4 times slower than quicktemplates. That is a cost for dynamics. There is no way to write template engine that will be faster than native Go code.

Syntax

Print

The most general syntax construction is printing a variable or structure field:

This is a simple statis variable: {%= var0 %}
This is a field of struct: {%= obj.Parent.Name %}

Construction {%= ... %} prints data as is without any type check, escaping or modification.

There are special escaping modifiers. They should use before =:

• h - HTML escape.
• a - HTML attribute escape.
• j - JSON escape.
• q - JSON quote.
• J - JS escape.
• u - URL encode.
• l - Link escape.
• c - CSS escape.
• f.<num> - float with precision, example: {%f.3= 3.1415 %} will output 3.141.
• F.<num> - ceil rounded float with precision, example: {%F.3= 3.1415 %} will output 3.142.

Note, that none of these directives doesn't apply by default. It's your responsibility to controls what and where you print.

All directives (except of f and F) supports multipliers, like {%jj= ... %}, {%uu= ... %}, {%uuu= ... %}, ...

For example, the following instruction {%uu= someUrl %} will print a double url-encoded value of someUrl. It may be helpful to build a chain of redirects:

https://domain.com?redirect0={%u= url0 %}{%uu= url1 %}{%uuu= url2 %}

Also, you may combine directives in any combinations ({%Ja= var1 %}, {%jachh= var1 %}, ...). Modifier will apply consecutive and each modifier will take to input the result of the previous modifier.

Bound tags

To apply escaping to some big block containing both text and printing variables, there are special bound tags:

• {% jsonquote %}...{% endjsonquote %} apply JSON escape to all contents.
• {% htmlescape %}...{% endhtmlescape %} apply HTML escape.
• {% urlencode %}...{% endurlencode %} URL encode all text data.

Example:

{"key": "{% jsonquote %}Lorem ipsum "dolor sit amet", {%= var0 %}.{%endjsonquote%}"}

Prefix/suffix

Print construction supports prefix and suffix attributes, it may be handy when you print HTML or XML:

<ul>
{%= var prefix <li> suffix </li> %}
</ul>

Prefix and suffix will print only if var isn't empty. Prefix/suffix has shorthands pfx and sfx.

Print modifiers

Alongside of short modifiers using before = engine supports user defined modifiers. You may use them after printing variable using | and looks lice function call:

Name: {%= obj.Name|default("anonymous") %}Welcome, {%= testNameOf(user, {"foo": "bar", "id": user.Id}, "qwe") %}
                  ^ simple example                     ^ call modifier without variable like simple function call
Chain of modifiers: {%= dateVariable|default("2022-10-04")|formatDate("%y-%m-%d") %}
                                    ^ first modifier      ^ second modifier

Modifiers may collect in chain with variadic length. In that case, each modifier will take to input the result of previous modifier. Each modifier may take an arbitrary count of arguments.

In general, modifier is a Go function with special signature:

type ModFn func(ctx *Ctx, buf *any, val any, args []any) error

, where:

• ctx - context of the template
• buf - pointer to return value
• val - value to pass to input (eg varName|modifier() value of varName)
• args - list of all arguments

After writing your function, you need to register it using one of the functions:

• RegisterModFn(name, alias string, mod ModFn)
• RegisterModFnNS(namespace, name, alias string, mod ModFn)

They are the same, but NS version allows to specify the namespace of the function. In that case, you should specify namespace in modifiers call:

Print using ns: {%= varName|namespaceName::modifier() %}

Conditions

dyntpl supports classic syntax of conditions:

{% if leftVar [==|!=|>|>=|<|<=] rightVar %}
    true branch
{% else %}
    false branch
{% endif %}

Examples: 1, 2, 3.

dyntpl can't handle complicated conditions containing more than one comparison, like:

{% if user.Id == 0 || user.Finance.Balance == 0 %}You're not able to buy!{% endif %}

In the grim darkness of the far future this problem will be solved, but now you can make nested conditions or use conditions helpers - functions with signature:

type CondFn func(ctx *Ctx, args []any) bool

, where you may pass an arbitrary amount of arguments and these functions will return bool to choose the right execution branch. These function are user-defined, like modifiers, and you may write your own and then register it using one of the functions:

func RegisterCondFn(name string, cond CondFn)
func RegisterCondFnNS(namespace, name string, cond CondFn) // namespace version

Then condition helper will be accessible inside templates and you may use it using the name:

{% if helperName(user.Id, user.Finance.Balance) %}You're not able to buy!{% endif %}

As an exception, there are two functions len() and cap() that works the same as built-in native Go functions. The result of their execution may be compared

{% if len(user.Name) > 0 %}...{% endif %}

, whereas user-defined helpers don't allow comparisons.

Ternary operator

dyntpl supports ternary operator for most primitive cases of printing the data. Conditions like this:

{% if x.a == 123 %}
    {%j= y.c %}
{% else %}
    {%j= z.d %}
{% endif %}

may be shortener using ternary operator:

{%j= x.a == 123 ? y.c : z.d %}

Condition helpers also supported:

{%= myConditionHelper(x.a, x.b, "foobar", 3.1415) ? y.c : z.d %}

switch

For multiple conditions, you can use switch statement, examples:

• classic switch
• no-condition switch
• no-condition switch with helpers

Loops

Dyntpl supports both types of loops:

• counter loops, like {% for i:=0; i<5; i++ %}...{% endfor %}
• range-loop, like {% for k, v := range obj.Items %}...{% endfor %}

Edge cases like for k < 2000 {...} or for ; i < 10 ; {...} isn't supported. Also, you can't make an infinite loop by using for {...}.

Separators

When separator between iterations is required, there is a special attribute separator that made special to build JSON output. Example of use:

[
  {% for _, a := range user.History separator , %}
    {
      "id": {%q= a.Id %},
      "date": {%q= a.Date %},
      "comment": {%q= a.Note %}
    }
  {% endfor %}
]

The output that will be produced:

[
  {"id":1, "date": "2020-01-01", "comment": "success"},
  {"id":2, "date": "2020-01-01", "comment": "failed"},
  {"id":3, "date": "2020-01-01", "comment": "rejected"}
]

As you see, commas between 2nd and last elements were added by dyntpl without any additional handling, like ...{% if i>0 %},{% endif %}{% endfor %}. Separator has shorthand variant sep.

loop-else

Separator isn't the last exclusive feature of loops. For loops allows else branch like for conditions:

<select name="type">
  {% for k, v := range user.historyTags %}
    <option value="{%= k %}">{%= v %}</option>
  {% else %}
    <option>N/D</option>
  {% endfor %}
</select>

If loop's source is empty and there aren't data to iterate, then else branch will execute without manual handling. In the example above, if user.historyTags is empty, the empty <option> will display.

Loop breaking

dyntpl supports default instructions break and continue to break loop/iteration, example:

{% for _, v := list %}
  {% if v.ID == 0 %}
    {% continue %}
  {% endif %}
  {% if v.Status == -1 %}
    {% break %}
  {% endif %}
{% endfor %}

These instructions works as intended, but they required condition wrapper and that's bulky. Therefore, dyntpl provide combined break if and continue if that works the same:

{% for _, v := list %}
  {% continue if v.ID == 0 %}
  {% break if v.Status == -1 %}
{% endfor %}

The both examples are equal, but the second is more compact.

Lazy breaks

Imagine the case - you've decided in the middle of iteration that loop requires a break, but the iteration must finish its work the end. Eg template printing some XML element and break inside it will produce an unclosed tag:

<?xml version="1.0" encoding="UTF-8"?>
<users>
  {% for _, u := range users %}
    <user>
        <name>{%= u.Name %}</name>
        {% if u.Blocked == 1 %}
          {% break %} {# <-- unclosed tag reson #}
        {% endif %}
        <balance>{%= u.Balance }</balance>
    </user>
  {% endfor %}
</users>

Obviously, an invalid XML document will build. For that case, dyntpl supports special instruction lazybreak. It breaks the loop but allows current iteration works till the end.

Nested loops break

In native Go to break nested loops you must use one of the instructions:

• goto <label>
• break <label>
• continue <label>

Well, supporting these things is too strange for template engine, isn't it? There is more handy break, provided by php. It allows to specify after a keyword how many loops must be touched by the instruction. dyntpl implements this case and provides instructions:

• break N
• lazybreak N

{% for i:=0; i<10; i++ %}
  bar
  {% for j:=0; i<10; i++ %}
    foo
    {% if j == 8 %}
      {% break 2 %}
    {% endif %}
    {% if j == 7 %}
      {% lazybreak 2 %}
    {% endif %}
    {%= j %}
  {% endfor %}
  {%= i %}
{% endfor %}

break/lazybreak N instructions supports conditional versions:

• break N if
• lazybreak N if

The example above may be changed to:

    ...
    {% break 2 if j == 8 %}
    {% lazybreak 2 if j == 7 %}
    ...

and you will give the same output.

Include sub-templates

To reuse templates exists instruction include that may be included directly from the template. Just call {% include subTplID %} (example {% include sidebar/right %}) to render and include output of that template inside current template. Sub-template will use the parent template's context to access the data.

Also, you may include sub-templates in bash-style using ..

Extensions

Dyntpl's features may be extended by including extension modules in the project. It's a Go packages that calls dyntpl's modifier registration functions (RegisterModFn, RegisterModFnNS) or condition registration functions (RegisterCondFn, RegisterCondFnNS) or any extending functions from dyntpl API. As result, inside templates will appear new modifiers and other helper functions.

Currently supported modules:

• dyntpl_vector provide support of vector parsers inside the templates.
• dyntpl_i18n provide support of i18n features.

To enable necessary module just import it to the project, eg:

import (
	_ "https://github.com/koykov/dyntpl_vector"
)

and vector's features will be available inside templates.

Feel free to develop your own extensions. Strongly recommend to register new modifiers using namespaces, like this.

Conclusion

Due to two phases (parsing and templating) in using dyntpl it isn't handy to use in simple cases, especially outside highload. The good condition to use it is a highload project and dynamic support requirement. Use dyntpl in proper conditions and it will solve your performance problems.