Research findings

= Research on open questions =

== Storage ==

Open questions:

• Core architecture: cross-process storage use, message passing, etc.
• Interface and storage libraries: indexeddb, sqlite, dexie, mongo, …
• Encrypted storage.

Electron's ordinary configuration has one main process, and one renderer process per window. This is problematic:

• We want storage (e.g., history) to be shared between windows, and the lifetime of storage to match the lifetime of the app as a whole.
• Changes to storage can originate in any window, or indeed in the main process itself.
• Any part of the app might read large quantities of data from storage — history and bookmark UIs in renderer content, menus or sync in the main process, etc.

We have some options for changing our constraints:

• We can run all renderers in the same process.
• We can run a separate storage process. (This is necessary if we opt for MongoDB or a mainstream DB like Postgres.)

These suggest some broad patterns:

1. Have the main process own storage, and exclusively use IPC to get access from elsewhere. There is some evidence that message passing/IPC isn't adequate for high-volume storage: Chrome uses a pass-by-reference shared memory system for link highlighting across renderer processes.
2. Run all renderers in a single process, and do storage in the same process. The main process would not access storage. This is roughly equivalent, but gives renderers direct access.
3. Run storage in a separate process, requiring both renderers and the main process to do IPC.
4. Use a storage system that allows for concurrent writes from multiple processes.
5. Use a storage system that allows for concurrent reads from multiple processes, and do writes from renderers over IPC.

It's not clear whether web technologies like indexeddb behave correctly when concurrently accessed from multiple processes. Certainly supporting libraries like Lovefield are categorically not safe (and probably not efficient, either):

https://groups.google.com/forum/m/#!topic/lovefield-users/Xs97HL4tDfk

As noted in that thread, indexeddb itself has some painful limitations, including being "miserably slow".

https://github.com/arthurhsu/rdb/issues/2

https://github.com/arthurhsu/rdb/blob/master/FAQ.md

Running a separate storage process runs head on into IPC problems, but also eats the cost of yet another process. Limiting renderers to a single process seems like it might be a long-term dead end, and might not solve the background service access issue. So my feeling is that 1, 4, or 5 are avenues to explore.

=== Change notifications ===

It looks like the only sane way to do cross-process change notifications is to use IPC and/or Reacty patterns. Don't bother futzing with triggers.

== Prior art ==

Brave is very JS-ey: its approach can be summed up as "don't use storage at all".

All sites and all bookmarks are part of the immutable app-state bundle that's given to every renderer process for React to work with. All changes are messaged back as actions, and eventually the entire state bundle (as documented https://github.com/brave/browser-laptop/blob/master/docs/state.md[here]) is passed to JSON.stringify and written to a file.

I haven't benchmarked it, but it seems like that would be problematic with real-world history and bookmark volumes (thousands of bookmarks, tens of thousands of sites, hundreds of thousands of visits): either incremental changes are lost, or each change generates lots and lots of garbage strings and a big file write. Is this a Reacty thing to do? Do they really keep N copies of your entire profile as JS objects in memory at all times?

Update: importing my Firefox bookmarks (1700) bumps the Brave session store file to 350KB. They don't actually track history yet — just last visit timestamps. With those bookmarks imported, the browser takes about half a second to switch tabs and about 1.5 seconds to open a new window. Note that it doesn't write the session restore file if you simply add a bookmark or visit a page… which means that if it crashes, you're returned to the last state you were in when you quit the browser. Oops.