Encoded special characters should be decoded by the crawler

[tillprochaska]

Description

When websites contain encoded special characters (e.g. & instead of &), they aren’t displayed correctly in the DocSearch UI.

Steps to reproduce

1. Set up a website that uses encoded special characters (e.g. &) in a heading.
2. Use the docsearch helper in the record extractor to extract contents.
3. The encoded special characters are indexed.
4. When rendering the DocSearch UI, the record contents are encoded again which means that & is displayed rather than &.

You can test this on our documentation site: https://docs.aleph.occrp.org. The sidebar has a "Developers & Admins" section.

This is what it is rendered like in the DocSearch UI:

Expected behavior

Algolia Crawler or the docsearch helper should take care of decoding encoded characters, e.g. by using the respective Cheerio configuration when loading the HTML or by decoding after extracting contents from the HTML.

I’ve adjusted the Crawler configuration and manually decode & for now, but it would be nice if either Aloglia Crawler or the docsearch helper could do this automatically.

recordExtractor: ({ helpers }) => {
  const data = helpers.docsearch({ /* ... */ });

  data.forEach((record) => {
    Object.entries(record.hierarchy)
      .filter(([level, content]) => content)
      .forEach(([level, content]) => {
        record.hierarchy[level] = content.replace("&", "&");
      });
  });

  return data;
}

Environment

• OS: Any OS
• Browser: Any browser
• DocSearch version: 3.3.0