formating of dates, numbers and more

[phloe]

I've been discussing the viability of mustache as a templating solution with a colleague and he raised some questions about a lacking key feature; formating. And I tend to agree with him - having this as part of the mustache spec would be very helpful.

The string representation of a date or time, the numbers of decimals of a price or the number of characters shown shouldn't be part of your viewmodel - it should (in a perfect world) be handled by the templating language.

I'm thinking something along the lines of being able to define a "format" in some form immediately after the variable name:

{
    price: 2.7532
}

{{price:0.00}}USD

2.75USD

I know JSON doesn't support a native date representation - maybe just treat an ISO datestring as a date object and present it through a format like php's date function (other formats might be more fitting/standard)?

I'm sorry if this issue has been raised a thousand times (it seems so obvious somehow) - but I haven't been able to find an answer to this yet ;D

[janl]

Would this work for you?

{
price: 27532,
price_format: function() {
// implement format_price();
return format_price(this.price);
}
}

and: {{format_price}}

PS: don't use floats for decimals, especially money.

[phloe]

That would mean I would "pollute" the viewmodel with a lambda - of which I'm not too keen... :)

And if I'm using the same templates in multiple languages across server/clientside I would also have to have implementations of these lambdas in those too (and maintain them). Then the lustre of mustache quickly wears off :(

Mustache could be the silver bullet :)

Floats/decimals was just a bad example :D

[janl]

I like the idea of helpers, hence my mentioning them in http://writing.jan.io/mustache-2.0.html and my "helpers" branch here: https://github.com/janl/mustache.js/tree/helpers a helper would implement your number format, but you'd still implement them server-side and client-side.

fwiw, polluting view objects with lambdas is rather cool I came to appreciate even though it feels dirty.

[phloe]

Ah, I see.

I like the separation that helpers provides - definately more "separation of concerns" than lambdas ;D

Is it going in the spec anytime soon?

[groue]

This request is so common I had to provide with a solution for my Objective-C implementation. Three goals : no alteration of the Mustache syntax itself, no cooperation ("pollution") of the viewmodel, and provide an expressive-enough API so that users can hook in missing features we haven't thought of yet. https://github.com/groue/GRMustache/blob/master/Guides/sample_code/number_formatting.md

TL;DR : by providing carefully crafted hooks in the Mustache rendering engine, we can let users extend Mustache in many ways, and keep the "kernel" clean and simple.

[janl]

basically what @groue says, all we need to get is an update to the spec and convince the people involved with it :)

[davidsantiago]

Just for discussion, two other ways I've seen of doing this are in ctemplate's template modifiers and Liquid's filters. They're both pretty similar to each other.

[groue]

A clarification : as I said, my goal was to provide a solution which does not alter the Mustache syntax. My goal was to provide an expressive Mustache API. The spec and the syntax, as they are today, did not block me.

I think we should not focus on the syntax. For me, it cannot be the right level, since there will always be some users asking for missing features, and the syntax will always be late.

Hooks are the right level. Handlebars, on this topic, is particularly brilliant.

[groue]

A simple example of the Handlebars expressivity : https://gist.github.com/1048968

[phloe]

Yeah - spec could bloat if features get added without merit... I like the idea of helpers/hooks.

You could pick and choose the helpers - and hopefully write/port them as you need them for other languages.

I like the syntax of Liquid's filters though (not too dissimilar to janl's helper syntax). Imagine if you wrote a generic "date" helper that could get a format argument like so:

{{myDate|date:"%Y %h"}}

Obviously myDate would be an ISO datestring :)

[ahem]

I really like the idea of helpers as well. I feel that lambdas introduce logic into the viewmodel, and IMO there isn't really any difference between logic in the template and logic in the viewmodel - neither of which are all that desirable.

The filters or modifiers from ctemplate or liquid, as @janl suggests above looks very interesting.

@groue's handlebars way might be a good solution as well, but is the best tag really {{#somthing }}? I mean #-sections already have way too much functionality in them (if-statements, loops, scoping and so on). Also, it would mask the helpername so that you could no longer access properties with that name, wouldn't it?

[groue]

@groue handlebars way might be a good solution as well, but is the best tag really {{#somthing }}? I mean #-sections already have way too much functionality in them (if-statements, loops, scoping and so on).

I don't know if Mustache should jump in the Handlebars train. Yet we should consider its design with interest. Many mustache/spec issues do not exist in Handlebars, because Handlebars is expressive enough to let developers find a solution to their problems without asking for a language patch. And this really should ring a bell for the Mustache designers.

Its key design decision was to push the rendering out of the "kernel", and put it in "userland", with built-in implementations for the most common cases (bools, loops, scoping, mainly). Since rendering is done in userland, Handlebars users can use the public Handlebars APIs in order to build their own extensions to the language. They are just as powerful as the built-in helpers, which use public APIs as well. This really was a smart move.

Mustache does not totally prevent this expressivity, but unfortunately the only hooks that the Mustache spec provides are lambdas, which are so weak they can not help implementing basic stuff like number formatting or array indices, for instance. As a consequence, because we live in a world that favors pragmatic solutions over theoretical purity, Mustache implementors give lambdas extra powers, out of the specification guidelines, or even provide other hooks, not even thought by the current specification (my GRMustache, maybe others, is in this case), or, worse, play with the syntax (and that's really not smart, since compatibility is key to Mustache).

My opinion is that we should externalize in userland some Mustache features as well, and really avoid touching the syntax too much. Even if sections gain even more responsabilities. Finding unambiguous and non-collapsing names for "active" sections and viewmodel properties should not be too hard.

Just for instance, GRMustache hooks let you do the following rendering, without any built-in support, whatsoever, for number formatting:

raw: {{float}}
{{#percent_format}}percent: {{float}}{{/percent_format}}
{{#decimal_format}}decimal: {{float}}{{/decimal_format}}

gives:

raw: 0.5
percent: 50 %
decimal: 0,5

Everything is done in userland, in the code of the GRMustache user.

In conclusion: first start with userland, check the expressivity of your API, check if you can do anything useful with it. And then, after the API has been carefully tested, that its expressivity has been proven, then maybe, think about providing a Mustache standard library of helpers/hooks, like ctemplate or liquid. Thinking about this standard lib right now is premature.

[ahem]

I agree that this should be done in userland. All the spec should do is provide the hooks to make it possible to provide this form of extensions. I still stand by my objection to adding more features to sections, however. There is already several open issues here regarding that (#14, #22, #23 and others) so I am not the only one who feels like this. Also there is this:

<time datetime="{{#format_isodate}}{{myDate}}{{/format_iso}}">{{#format_readable_date}}{{myDate}}{{/format_readable_date}}/time>

It may just me, but I do feel that this is more readable (even with the strange PHP formatting letters):

<time datetime="{{myDate|date:"%c"}}">{{myDate|date:"%j. %F %Y"}}</time>

The arguments in the proposed filter syntax seems to be somewhat of a necessity, if the hook should be useful for formatters. Writing a number of decimal formatting tags like {{#2_decimals_format}} {{#3_decimals_format}} {{#4_decimals_format}} and so on seems slightly silly to me.

[groue]

I personally feel that literal number and date formats (%j, %Y and friends), right in the template, are smelly: template localization is going to be painful, and not everybody speaks English. Literal formats are not a necessity, and should even be avoided.

I'd rather have indirect formats {{myDate|date:iso8601}}, {{myDate|date:short}} {{myDate|date:long}}.

And now I can't see why {{#format_iso8601date}}{{myDate}}{{/format_iso8601date}} is really less readable, even if it looks more verbose.

Moreover, sections allow for formatting not only a single number, but all numbers in a whole section of the template. Think: ISO date across a whole XML document, with a single section. Think: currency format across the whole invoice HTML document, again with a single section.

[bobthecow]

How 'bout

{{#date.iso8601}}{{myDate}}{{/date.iso8601}}
{{#format.iso8601date}}{{myDate}}{{/format.iso8601date}}

? Then you can have hashes full of formatter lambdas, and indirect formats rather than literal formats.

[phloe]

I'm sold on the indirect formats - much better than the hard-to-read php/ruby formating :)

[ahem]

Okay, so the PHP dateformatting example was a poor one, you're right in that literal date formats should not be used. But wouldn't implentation be a problem with the section based approach? Maybe this is too early to discuss actual implemention, but I do feel that it should at least be considered.

{{myDate|format_iso8601date}} is easy, it maps to something like registeredHooks.format_iso8601date(value);. This is easy to implement and easy to understand.

With sections it's possible that more formatters could be active at one time, so some kind of protocol is required to select which formatter will be used. In GRMustache you have selected a pretty simple protocol - only numbers are formatted so the formatters only have to be called if what is outputted is a number, and you only have to call the most recently enabled formatter.

If this were a more generic setup, where hooks could be used to format anything, then either each hook would need to provide a callback where it could opt in or out, as to whether it would like to format a given value, or you would need some other way to decide.

Maybe all formatters are called in the order they were enabled, starting with the most recently enabled one, and then they would have to return either the original value, or a formatted version of it.This would mean that a template like this: {{#format_numbers}}{{#format_dates}}{{myDate}}{{/format_dates}}{{/format_numbers}} would turn into: registeredHooks.format_numbers( registeredHooks.format_dates(value) ).

Either way, it is definitely more complex behavior, harder to implement and harder to understand... and I still feel that it is less readable :-)

[bobthecow]

The new "helpers" feature of Mustache.php (2.0-dev) makes this super awesome:

Given a set of formatters:

<?php

class DateFormatters {
    const SHORT_DATE = 'd/m/Y';

    public function __isset($key) {
        return method_exists($this, '_'.$key);
    }

    public function __get($key) {
        return array($this, '_'.$key);
    }

    public function iso8601($date) {
        return $this->parse($date)->format(DATE_ISO8601);
    }

    public function atom($date) {
        return $this->parse($date)->format(DATE_ATOM);
    }

    public function short($date) {
        return $this->parse($date)->format(self::SHORT_DATE);
    }

    private function parse($date) {
        return new \DateTime($date);
    }
}

Register them as helpers:

$m = new Mustache;
$m->addHelper('date', new DateFormatters);

And use 'em in all your templates!

{{# date.short }}{{ item.createdAt }}{{/ date.short }}

(note that this is fully supported by the current spec: it doesn't require anything besides lambda support and dropping a bunch of helpers in the top-level context stack frame)

[ahem]

@bobthecow It doesn't fully follow the current spec, does it? Is the dotsyntax in the spec now? Also, dropping helpers into the top-level context stack frame isn't either, i believe, but that is sort of what I think most of us here is getting at anyways.

How does Mustache.php solve the issue of multiple formatters active at one time? Or how would the name be rendered in a template like this:

{{# date.short }}{{ item.Name }} created at {{item.createdAt}}{{/ date.short }}

Would item.Name be passed to the date formatter? Would the "created at" string?

[bobthecow]

1. It's fully spec-compliant. dot notation is part of the spec as of v1.1.0 (March 4, 2011).
2. There's room for the PHP implementation of helpers in the spec: how you get things into your rendering context is of no concern to the spec. It's very language and implementation specific. In Ruby, you could add helpers by monkeypatching your Mustache class to add the formatter properties.
3. The whole thing would be passed to the formatter, just like any other lambda. Instead, you would do: {{ item.Name }} created at {{# date.short }}{{item.createdAt}}{{/ date.short }}

[ahem]

Ah, okay. This is different from what @groue have implemented in GRMustache then. If you can't put anything between the tags, then it doesn't make sense to have this behavior implemented with sections - and I still stand by that {{ item.createdAt|date.short }} is much easier to read.

Also rendering the value, and then parsing it again with the helpers is not exactly best case is it? I would expect that in some cases information would be lost before the helper function got a chance to format it.

[bobthecow]

I was just pointing out that it's possible to implement without extending the spec.

That said, the pipe style does look a lot cleaner. I could get on board with a "pipe notation" that is (mostly) syntactic sugar for chaining lambda callbacks, the same way "dot notation" is (mostly) syntactic sugar for chaining nested section tags.

Where these:

{{ foo.bar.baz }}
{{ foo | bar | baz }}

are both a shorthand for

{{# foo }}{{# bar }}{{ baz }}{{/ bar }}{{/ foo }}

but in the case of dot notation, it does the current context stack restrictions, and in the case of pipe notation, it passes the previous value to the next lambda.

This would save the to string and back trip, and would clean up templates, especially when running multiple filters.

[groue]

Ah, okay. This is different from what @groue have implemented in GRMustache then. If you can't put anything between the tags...

Indeed GRMustache's proposal can deal with {{#format}}...{{date}}..{{#foo}}...{{otherDate}}...{{/foo}}...{{/format}}.

And this can not be acheived with lambdas. Some other hooks had to be introduced.

While I was still stuck with the lambda frame, the most I could do was to implement something like @bobthecow, that is to say a lambda that render and use their immediate content.

@bobthecow, what about checking GRMustache API ? It's not funny to read a documentation, but at least there is a documentation: https://github.com/groue/GRMustache/blob/master/Guides/sample_code/number_formatting.md

It's fully spec-compliant.

It looks that your helpers render raw rendered strings (that is to say, strings that you trust will not be touched by the Mustache engine) : unfortunately, this is not a spec-compliant behavior.

What you may have missed, and no one could blame you, is that the spec requires lambdas to output a template string that will be parsed by Mustache engine, no the rendered string itself : https://github.com/mustache/spec/blob/master/specs/~lambdas.yml#L28

[bobthecow]

What you may have missed, and no one could blame you, is that the spec requires lambdas to output a template string that will be parsed by Mustache engine, no the rendered string itself.

Sorry, yeah, I realized that after I wrote the example :)

Mustache.php is spec-compliant, and it works just like you say. You would have to pre-render the string, so the body of parse would handle that in a full (not off-the-cuff and typed into a <textarea>) implementation :)

[bobthecow]

@groue I have seen the GRMustache API, and I have fundamental problems with it :)

I'd much prefer the syntax to use pragmas rather than sections when it's changing the meaning of {{ }} inline like that.

raw: {{float}}

{{%FORMAT percent}}
percent: {{float}}
{{%FORMAT default}}

{{%FORMAT decimal}}
decimal: {{float}}
{{%FORMAT default}}

[groue]

@bobthecow You're right. Actually, I really don't like that lambdas can not output anything that will be rendered raw. You always have to make sure your lambda won't output any {{, in case you would trigger unintentional Mustache formatting. This means that you can not output data that comes from untrusted source.

[groue]

I have seen the GRMustache API, and I have fundamental problems with it :)

I'd be happy to know them.

changing the meaning of {{ }}

???

[bobthecow]

???

It's saying "Between {{#THIS_TAG}} and {{/THIS_TAG}} treat all {{ of_these }} differently". That's inconsistent with the way sections work elsewhere in mustache. It feels more like a compiler directive to me, which is why I'd prefer pragma instead of section syntax.

[ahem]

The spec does specify that "Lambdas used for sections should receive the raw section string" (here), so I guess, if one were to implement a lambda that would treat 'variable blocks whose value is a number' different from the rest, one could just do that.

This isn't exactly how GRMustache implements it, but I believe the effect is similar. The formatterstacks could be considered helpers for building lambdas like this, I guess. @groue should of course correct me on this :-)

As I read the spec, however, the context object isn't supposed to be accessible from inside the lambda. The lambda is supposed to receive only the text inside it's own section unrendered, so it would be something like {{ number }}.

Without access to the context object, it isn't possible to format anything, and even with access to the context object, it still needs to implement it's own Mustache parser (or have some sort of deep knowledge of it's parent parser, so it can reach into it like GRMustache does).

IMO this is not a viable option - the goal should be that a user could provide a formatter like this (JavaScript):

{ decimal: function (x) { return typeof(x) === 'number' ? x.toFixed(2) : x; } }

and, with that attached somehow, expect it to format a template like this:

{{% decimal }}{{ pi }}{{/decimal}}

or this:

{{ pi|decimal }}

into the expected "3.14".

[spullara]

How about something that is simply rewriting the current syntax in a more terse format. For example, you might write:

{{#format_isodate}}{{myDate}}{{/format_iso}}

but allow it to be rewritten, with no change to the viewmodel, as:

{{#format_isodate myDate/}}

And have it literally do the same exact thing, just with a more compact syntax.

[pvande]

@bobthecow I'll bet you a nickel it was the {{a}}{{a}}...{{/a}}{{/a}} test case that tripped you up. :)

[bobthecow]

@pvande Nope. It was whitespace edge cases. Those were the only spec fails.

[pvande]

@bobthecow Ah, well... Good on you then! (I've seen a number of Regex-based "parsers" bite it on that case.)

[bobthecow]

That said, I don't regret switching to a real parser. The new hotness is at least 20x faster — orders of magnitude faster with template caching enabled.

[groue]

Hi @pvande, @bobthecow, @defunkt. I wrote something about filters, that I wish you all read: https://github.com/groue/GRMustache/blob/filter_chain/WhyMustacheFilters.md.

[groue]

@pvande, @spullara, you'll find in pvande's wiki page a pragmatic interpretation of logiclessness that may move your positions on what's "forbidden" or not "realized".

[pvande]

So, here's a new angle.

Filters necessarily mean that we cannot prohibit this: {{ sql | query }} or {{ ruby | eval }}

Lambdas already give us some subset of this: {{#query}}DELETE FROM table;{{/query}} or {{#eval}}FileUtils.rm_rf '/'{{/eval}}

This seems (to me) to be directly counter to the Logic-Free principle, in addition to the "user-safe" notion. The latter might be considered an acceptable risk if you're not allowing user-written templates, but it's certainly unintentional behavior. But at least it's sandboxed to the data provided to the context stack, right?

data = {}

# { } -> Hash -> Object -> [Kernel] -> Kernel -> Kernel.eval
key = "class.superclass.included_modules.last.eval"

template = "{{# #{key} }}puts 'Hello'{{/ #{key} }}"
Mustache.render(template, data)

This only fails in today's implementation because of a fluke -- methods that don't have an arity of exactly 1 aren't passed arguments. Kernel.eval has an arity of -1 (it claims to take zero or more arguments in 1.8.7; actually, it has one or more optional arguments). If (more likely, when) the Ruby implementation sends content strings to methods with optional arguments ... game over.

Now, what can we do about this:

• We can refuse to dispatch to methods that weren't declared on the object type itself. (Hash.new() no longer responds to class, sandboxing is maintained)
• We can rescind the ability to execute code from templates. (Unfriendly, but there's no safer option.)
• We can rescind our claim of being "user-safe" and our claim to enforcing Logic-Free templates.
• We can reduce "user-safety" and "Logic-Free" to lifestyle choices; not something inherent to us, but something we choose to do.

I am not happy with these options.

[pvande]

@pvande, @spullara, you'll find in pvande's wiki page a pragmatic interpretation of logiclessness that may move your positions on what's "forbidden" or not "realized".

Your position has been noted.

[groue]

Lifestyle, for Christ's sake, or you'll never be able to look at yourself in a mirror, having contributed to such an evil library. Do you really want to ship a VM with Mustache, that would safely run the nasty code of users?

[groue]

GRMustache has updated its syntax for filters, so that it's consistent with what I think are good filters :

• composable
• compatible with "scoped" lookup.

As @pvande noticed, the best syntax known so far, which has those two properties, is f(x).y. Check out the doc!

[groue]

The GRMustache fitler documentation describes:

• what happens for missing filters or objects that are not callable (boom: a runtime error)
• "filter namespaces", for users who like {{ math.abs(x) }}

[groue]

@pvande it would be fair of you to acknowledge my propositions, or write a rebutal. This silence doesn't fit your position of maintainer of this repo. Could @defunkt share his thoughts with you on the subject?

[groue]

Anyway. I've shipped for good, updated my filters proposal with many elements from other Mustache implementations, and since some implementors may not feel at ease with parsing ’f(x).y’ expressions, provided a grammar and a state machine that's easy to implement.

[ceefour]

+1 for this!

[groue]

GRMustache has just shipped with support for "meta filters": filters that return filters.

Release notes: https://github.com/groue/GRMustache/blob/master/RELEASE_NOTES.md#v542

Sample code: https://gist.github.com/3803707

[groue]

GRMustache has just shipped with support for "variadic filters": filters that take several arguments.

Release notes: https://github.com/groue/GRMustache/blob/master/RELEASE_NOTES.md#v55

Sample code: https://gist.github.com/3803707

[oliverjanik]

What's the latest news?