📦[WIP]: generate index from HTML meta

[hypervillain]

This PR replaces the currently generated searchable text with an array of weighted searchable items. It adds these keys to each item:

• title, extracted from tag
• description, extracted from tag
• keywords, extracted from tag and transformed to array
• headings, an array of headings found in document
• text, a lightweight version of previous html-to-text output
• image, image url extracted from og:image meta
• from, an array of folders and subfolders the file is stored in

These keys are then ordered by weight (see Fuse weighted search) and passed to generated function. Note that these are default values and should be overridable by the user.

Added config option: an ignore list of paths to be dismissed from search. I'm also planning to use the from key to filter out the results, in order for the user to build features like "search in [...]"

Directory read is now handled by globby.

About replacing html-to-text with retext: I mainly used it to keep the unifiedJS processor approach, but also because it could open the plugin to more features, eg. retext-keywords. I don't know exactly the pros html-to-text, so maybe it does things better.. Let me know!

Not everything has been carefully tested yet, but you can preview the result right away by running the (light) test suite 😊

[swyxio]

super fancy! i like it a lot.

is this good to merge or you still have some stuff todo? pls feel free to merge on your own but happy to give any opinions u want. its your project now as well as mine. you can also publish on npm now

[hypervillain]

Awesome, that's really nice of you 🙌
I'll clean things up and hit merge.

Also, 2 last small things:

• having some issue with 2FA atm, so I'll (kindly) ask you to publish on NPM
• do you know what's up with the Netlify build issue?

[swyxio]

ah. in order to get this to work i had to patch read-dir using patch-package

(i documented this in the readme i think, somewhere in the project)

read-dir was a dep of copy-template-dir. which isnt well maintained. i was already leaning towards ripping it out but it does exactly what i needed it to do. up to you on how you want to treat this dependency.. we might just need a better altnerative

[hypervillain]

Hey 👋

I think I'm done with the PR, @sw-yx!
Since your last read, I

• replaced read-dir with globby
• got rid of patch-package
• renamed ignore argument into exclude
• added an image field to each item created
• found a bug with Netlify build and reported it here

Feel free to review and merge if you will!
Hugo

[swyxio]

published as v0.1.0! havent tested myself