How to use post-processing functions?

[filmaj]

The readme mentions it, but I'm not sure how to assemble it. pre.js seems to magically work based on the filename and export names? I tried copying a working pre.js I have and renaming it to post.js, and renaming the exports to post, but that didn't work.

[tripodsan]

like here: https://github.com/adobe/helix-pipeline#using-extension-points ?

or what do you want to do?

[trieloff]

@filmaj you can find a real-life example here: https://github.com/adobe/helix-cli/blob/9b1b05c80f9976cb83c0331278e0109af3944572/test/integration/src/html.pre.js#L98-L110

The list of extension points that you can hook in .before or .after are listed here: https://github.com/adobe/helix-pipeline#common-extension-points

[filmaj]

Thanks! I don't know how I missed the text in common extension points...

What I'm trying to do: Instead of loading all of spectrum-css's SVG icons in one massive go as part of our current global HTML template, I want to post-process HTML in the global HTML template, search through the HTML and find which spectrum-css icons are used, and inject only the ones being used into the DOM. We are thinking this would improve loading times.

[trieloff]

The optimal extension point for that might not yet be exposed. It would be here:

helix-pipeline/src/defaults/html.pipe.js

Line 69 in 17e9060

.after(tohast) // start HTML post-processing

(so that you can use the complete parsed HAST)

[ryanbetts]

@trieloff Any timeline for having that tohast extension point exposed to the wrapper functions? Or a way to access it before it is exposed? Sounds like exactly what we need.

I tried playing with the existing common extension points you mention above, yesterday. In our global template, they all seem to execute before the esi templates have been rendered, so not as useful.

[trieloff]

@ryanbetts as soon as #324 is merged ;)

[ryanbetts]

@trieloff Music to my ears! Thx.

[ryanbetts]

@trieloff Looking at the contents of the payload in the after:hast stage of the pipeline, I'm still not sure where to find the final HTML that I'm hoping to post-process. But still new to Helix so I'm probably missing something.

Here's where I've looked:

• context.content.document.body/context.content.body is empty.
• context.content.html is undefined
• context.response.body contains the template, with the <esi:include ...> tags still unrendered.

Am I just looking in the wrong spot?

[tripodsan]

I think, it's in context.response.hast :

helix-pipeline/src/html/html-to-hast.js

Line 20 in e9bb5ac

hast,

[ryanbetts]

@tripodsan That object is not empty, but seems to also be incomplete. I'm still unfamiliar with working with these hast objects, though. As far as I'm aware I turn them into a workable DOM with the helix VDOM utility, correct? Something like: new VDOM(context.response.hast)?

When I do that with the the hast object I don't end up getting what is rendered in the browser. I just get a bunch of empty <div> tags for the most part.

[tripodsan]

I don't think you go back to vdom again. I think the hast is meant to last-minute cleanup.

btw:

In our global template, they all seem to execute before the esi templates have been rendered, so not as useful.

hmm, I think there is a misconception. the response from the esi:includes are not incorporated into the response of the htl that includes them during the pipeline, this is an operation that happens on the (edge) server. the idea is, that the esi:included responses are cacheable independent. if you really need to perform an operation on the very final dom, then there is no other way as to do it clientside.

[ryanbetts]

Thanks for the speedy replies! I think I need to spend some more time in Helix school.

Okay, so if I'm following you correctly: .htl files are all run through the pipeline individually, and only combined at the moment of a request. Makes sense that you want all the components to be independently cacheable.

So, if we wanted to have something that was run for every template, we have 3 options:

1. Wait until it has been published to the edge and run the cleanup client side on the final DOM. This is the current approach that we are trying to move away from.
2. Run the cleanup in the pre function of each individual template it applies to. Would like to avoid this in order to keep things DRY.
3. Write our own pipeline that has that as a step to take care of that.

Am I correct that number (3) is an option?

[tripodsan]

yes, you can provide your own pipeline, via a *.pipe.js file. but you'd still need to have one for every template.
you can copy-paste it from here: https://github.com/adobe/helix-pipeline/blob/e9bb5ac292d0c56277c7b5d33e8053248648ebee/src/defaults/html.pipe.js

I would keep your cleanup-function in a helper, and then add it to every template. something like:

/src/cleanup.js

module.exports = (context, action) => { ... cleanup code ... }.

/src/footer.html.pre.js

module.after = { hast: require('./cleanup.js') };

[ryanbetts]

Gotcha. Thanks!

[ryanbetts]

@tripodsan Still quite confused as to what I should be doing with the HAST. Is there any documentation on how to work with that format?

[tripodsan]

I'm not too familiar myself. @trieloff might know more.
for for example: add-headers.js is removing the meta tags and populate the headers.

[trieloff]

@ryanbetts you can find a short introduction to the format here: https://github.com/syntax-tree/hast (it's a lightweight representation of the HTML syntax tree with a focus on ease of transformation, embedded into the unifiedjs ecosystem – i.e. the same stuff we use for Markdown handling)

There is a long list utilities specifically for HAST and another list of utilities for all unified syntax trees that make working with HAST really easy. The most useful are:

• https://github.com/syntax-tree/hast-util-select (select nodes using querySelector)
• https://github.com/syntax-tree/unist-util-visit (to visit each node and extract information)
• https://github.com/syntax-tree/unist-util-map (to translate nodes into something else)
• https://github.com/syntax-tree/unist-util-inspect (console.log(inspect(node)) is better than console.log(node))
• https://github.com/syntax-tree/unist-builder (convenient syntax to create new nodes)

[ryanbetts]

@trieloff This is great, thx! Between this list and examples in your source I think I'm off to the races.

[trieloff]

@ryanbetts please note that we made hast obsolete in v3.0.0: https://github.com/adobe/helix-pipeline/releases/tag/v3.0.0

[ryanbetts]

Duly noted. Will refactor to that now. Thx!

[rofe]

Can this be closed?