docs: improve search for documentation

[tmair]

Description:
This PR ports the search web worker and its integration from angular.io to the RxJS documentation site. The main changes are outlined on the original commit in the angular repository: angular/angular@fccffc6

In Addition to the changes ported from the angular documentation the following changes where made:

• Adaption of the default lunr stop word filter to exclude from, of and every as a stop word (this allows for a search of the of function)
• disabling of the ignore word list for class and interface members (this allows to find results for next as next is a member of the Observable interface) This is now also fixed on angular.io

Note:
This PR needs to be rebased after the dependency issues in #6913 are merged.

Related issue (if exists):
This should fix #6500
The should be no regression for #4536

[jakovljevic-mladen]

@tmair, this is great PR, thank you for this one. I will review it as soon as I find some time - I should be able to do it in the next couple of days.

[jakovljevic-mladen]

This is a great PR, thank you a lot for making it. It took me a while until I got into this stuff so I could do the proper review. Changes look good to me, however, I've got some questions/comments.

One of them is related to the duplicated results (not related to this PR, but could be handy if we fixed the issue). For example, in this picture:

You can see that we have repeatWhen appearing two times (the on is the link to this path: /api/index/function/repeatWhen, and the other one to this path: /api/operators/repeatWhen). Is it possible to remove docs with /api/operators/ path (since this path is deprecated)?

Other questions/comments in files.

[jakovljevic-mladen]

It looks like this line is breaking existing functionality of the generated keywords. Please take a look at this gif (where I compared changes from this PR and official site):

However, removing this line won't make it much better:

Querying "from" will find 244 results instead of only one (like in the gif). So, I'm not sure what would be the best approach here...

[tmair]

Please see the explanation of this issue in my comment below.

[jakovljevic-mladen]

Searching for "when" will result in some results (because of repeatWhen and others).

How do these stop words work? If "when" is declared as a stop word, shouldn't there be no results in that case?

Anyways, I think that "when" should be removed from stop words as there are some operators that have this word in their name. "with" and "while" as well (e.g. withLatestFrom and skipWhile) and possibly others (I haven't checked the entire list).

[tmair]

Stop words will match only the stop word. So when the word when is included within a term that term is not filtered.

[tmair]

Hi @jakovljevic-mladen,

thanks for the review. I will look at your questions but right now I don't have the time for it. It will take me 3 to 4 weeks to get back to it, but I will continue the work on it

[tmair]

I rebased the branch on the current master and tried to incorporate your review feedback into my branch.

Since the search is somewhat difficult to understand I have made some notes on how the search works. I hope this helps to understand the code base better and should also serve as a reference for future work in this area.

How does the search work?

The search is split into a preprocessing step at build time and a index creation step (inside a web worker) on runtime.

The preprocessing happens in generateKeywords.js and performs the following steps:

1. calculate the set of pages that needs to be indexed for searching
2. For each non ignored property (ignoredProperties) of each page document, extract keywords used to build the search index (mainTokens). Here the ignore words list is used to filter out common english terminology (e.g. you, from, every, ... )
3. Extract all public members from the code and add those to the search index (memberTokens) ignoring the ignore words list.
4. Extract keywords from the headings of the documents (headingTokens) using the ignore word list
5. Generate the search data that will be downloaded by the service worker to build the search index. The search data is already stemmed (https://lunrjs.com/guides/core_concepts.html#stemming) to further reduce the amount of data needed to transfer to the service worker.

The service worker performs the following steps, before it can be used to answer any search results:

1. Load the search data computed by the preprocessing step described above
2. Create a processing pipeline by not including a stemming step, as this was already performed during preprocessing. However a stopWord processing step is performed.
3. Process the data through the processing pipeline to create a search index

During the preprocessing and indexing of the search data there are two places where words are ignored. The ignore word list, will be used to filter common english terms during the preprocessing phase (not used with member tokens). The stop word filter will perform the same operation when building the index on the service worker.

Improvements to the searching process done in this PR

Ignore words and stop words filtering

Because it is strange to have two ignore word lists and filtering steps, I have decided to remove the stop word filter from the index builder pipeline completly, as it is does the work twice and is quite confusing. However I have left all words (including from, every, of, with) included in the ignore word filter, as the search results are not great when those words are not ignored. For example from is used whenever there is an example included within a documentation page because of the import ... from ... syntax. The relevant pages are still found, because the title of the page is not filtered. Therefore from will still return the search results where from is included within the title of the page.

Searching for from

Regarding the different search results for from, here is what is happening during the search:

1. First search for a document that contains all of the search terms (divided by spaces) provided (in our case from)
2. If the search has at least one result, return that result set (and that is what is happening for from. Thus we only have one search result)
3. Search for a document that contains any of the search terms
4. If the search has at least one result, return that result set
5. Search for documents that contain any of the search terms and contain the first search term somewhere in the tile

So for from as there is only one result, we return that single result.

I modified the initial search in such a way, that it will add more weight to search terms within the title of the page. Maybe that will result in better search results.

Removing depreacted entrypoints from search results

I also removed the deprecated entrypoints from the search results (e.g. when searching for buffer). This will also reduce the size of the generated search data.

[benlesh]

@jakovljevic-mladen is this still something you're considering merging?

[jakovljevic-mladen]

@benlesh, yes, but I'm still waiting for #6913 to be merged. I should've wrote it in the comments earlier so that @tmair knows as well, I'm so sorry about that.

[benlesh]

@jakovljevic-mladen #6913 is approved and ready to merge. Sorry for the delay.

[jakovljevic-mladen]

@tmair, thank you a lot for making these changes. I read your exhaustive explanation of how search works and I agree with these changes. During my testing phase couple of months ago, I had issues when I was searching term from where my results were different to what you were describing. So, I assumed this is due to conflicting dependencies. Now that #6913 is merged, I rebased this PR against master, did npm install and tried search and it worked the way you described it.

Since I did rebase, I'd like you to review this PR to check if merge conflicts are resolved correctly. Also, we have issues with merging to master due to issue with Node 14 which is now removed, but pipelines still show this as a required step.

[tmair]

@jakovljevic-mladen I reviewed the PR and also did a short local test of the new search implementation. So far everything looks good.

The only thing that was different for me was a npm install will alter the package-lock.json. I don't know if that is an issue, since you have already commited the changes to the lock file.

[jakovljevic-mladen]

I don't know if that is an issue

It shouldn't be an issue. Doing an npm install changes package-lock.json almost every time, but we have the latest package-lock.json, so we should be fine. I'll merge as soon as we fix the Node 14 build issue.

[jakovljevic-mladen]

Thanks for the fix @tmair and sorry for waiting this long for the merge. It is cherry-picked to 7.x branch and deployed to the rxjs.dev.