Initial page visit indexing

[poltak]

• fixes Store urls and titles for all pages, even those which are closed quickly again #232
• description of new page visits process in 8cbda3d
• a lot of cleanup of modules related to the page visit process (log-page-visit, store-page, page-analysis)
• new index method for stage ii allowing updating just the terms for an existing page
• removal of unused pouch visits during page visit process
• cleanup of index pipeline code, and allowing it to work with no main terms content (stage i)
• update of terms search implementation to merge in title/URL terms results with content terms results
• implementation of new PausableTimer class: state wrapper around setTimeout (7b8c4c9)

TODO/enhancements:

• stage ii still does the old behaviour of "waiting" for at least 10s. This waiting includes when the tab is not active. Recent work on the new tabs state means we have access to each tab's accumulated active time. Maybe we could try to use this as a smarter way to decide when to trigger stage ii
• see if we can get the webNav API's transition data stored in tabs state. This is only available from webNav.onCommitted event (happens before the webNav.onCompleted event that we use for stage i), but it should be able to persist in tab state for later use
• the webNavigation API events used for stage i seem to both fire on certain pages - find a better way to ensure stage i only can ever happen per page

[blackforestboi]

I tried it with this article: newyorker.com/tech/elements/the-mission-to-decentralize-the-internet

It shows up in the list of results when no query is typed, and also in the index all the data is there. However, when searching for "decentralize" or "internet", no results come up.

[poltak]

@oliversauter looked into this, but seems like it is because of a logical bug in the existing terms search implementation which ignored title and URL terms if there wasn't a corresponding content term entry. Can reproduce on master. I've made an update to the search algo in here (as init visit is pretty useless without working title/URL search!) to fix that behaviour. Great find!

[blackforestboi]

Went over it again, and a few changes still necessary:

1. Renaming the text for indexing into:

• 'Indexing page title & url: url' for the first index
• 'Indexing Page Content (X Terms): url' for the content index
  Whereas 'url'is the current url.

2. Somehow this page now takes 18 seconds to index the content. Wonder if it is just an error in calculation in respect to the start of the index, or the actual index time. The latter would not be ideal :)

[poltak]

Forgot this work was still pending! @oliversauter added those requested dev console logs, although there's no access to the URL at the indexing terms stage, so logs the encoded ID version instead.

Can't reproduce 2 here. If reproducible for you, will have to get some info like # docs in DB. I think main thing is the size of your terms index. Terms indexing time is described here. Shouldn't be related to this work though.

You could paste this into background script dev console to check your terms index size, and I could try to reproduce with similar size:

((i = 0) => index.db.createReadStream({ gte: 'term/', lte: 'term/\uffff' }).on('data', () => (i += 1)).on('end', () => console.log(i)))()

[blackforestboi]

I could reproduce it :)

It was because I had Memex active twice (one prod, one testing). So it had to do the indexing on each. If only one is activated it takes 1.6 seconds.

I'll merge this one in.