Data ingestion starved when in background

[emilk]

We currently ingest the incoming log data into the data store in a requestAnimationFrame callback, which won't get called at all when the web app is in a background tab. The same goes for the garbage collection.

The WebSocket ingestion however is done in a async callback, so log messages will be received and put on a channel that will grow until the user switches tabs again.

This means that if you start a web viewer, move it to a background tab and then keep sending it log events, it will eventually crash because it runs out of memory.

There are a few things we should consider doing here:

Do store ingestion and GC on a separate "task"

e.g. setInterval callback on web, and a background thread on native.

This will likely be a performance boost as well on native.

Apply back-pressure to halt new log messages when we don't ingest them fast enough

This has the downside of added latency though, which is bad for live views.

We have a --drop-at-latency flag that drops incoming log messages if the channel grows over some certain length of time. We may wanna change that to a maximum size in MB and allow the user to chose if this should apply backpressure (we get to see everything, eventually) or drop messages (we get a live view, but it may be lossy).

Related:

• Memory usage increase indefinitely after window minimized on Linux #7254

[jleibs]

This also applies to native on most OS's since ingestion and gc happen on the main UI thread which runs at a lower rate when the window is in the background.