Store unprocessed page text content in page doc

[poltak]

• page text content goes through two main stages: 1) HTML preproc -> 2) plain-text preproc
• previously storing result of 2) at path content.fullText in page docs - pretty much the same content as will go into the terms index
• now storing result of 1) in page docs, while terms index source remains unchanged
• language, extracted from the DOM, is also used in 2), although only available from the source data of 1) - now simply adding that as an additional content.lang key in the page doc for access in 2)
• seems to be functional in both importing and page visit scenarios

[blackforestboi]

This can be merged.

Just a last question: What happens on revisit? The fullContent is entirely replaced, right? But the index is appended?

This is OK behaviour, so can be merged in. Not ideal yet, but ok to merge.

[poltak]

@oliversauter yes, that is how it works, although note that "index is appended" still means updating each of the existing terms' latest timestamp as well as that's how the scoring works in the terms indexes (#187 - unrelated to this PR)

What would be the ideal behaviour for the full content in pouch on a revisit? I think appending would be pretty difficult and probably need some smart way to understand the text content (so you're not just duplicating content each visit). Or there is the dumb route of uniqing the pouch text content by words - appending is then easy (dupe words ignored), but you would lose any possibility of deriving term frequency from that data.

[blackforestboi]

need some smart way to understand the text content

Yeah that is what we still need to figure out before we can do appending etc. as long as the document can be searched with old words, its ok. This is how people remember stuff again. Just in case we need to re-index/re-analyse, we would need the lastest version.

So all good for now.