Send incremental reports probe->app

[2opremio]

The uncompressed report sizes are very large, ~1MB each. This puts a lot of pressure on the codecs and causes a lot of traffic.

See #854 (comment)

We could send reports incrementally (i.e. full report + diffs) to improve this.

[peterbourgon]

To send an incremental report, the probe must know what the app "already knows" in order to calculate a diff. That means the current (stateless) POST method of getting reports to apps is insufficient, and we'll need to maintain a persistent connection. Given the Go HTTP server keeps the connection around between POSTs currently, this is probably fine from an ops POV. But for the cloud offering, for example, we will end up creating a persistent connection from every node in the customer infrastructure to whatever.weave.works. Do we think that's acceptable?

I suggest WebSockets, since we're already doing something exactly like this between the app and the web UI. Upon connection, send the complete report. Thereafter, send deltas. (Maybe every ~minute, dump the complete report again, as a sync.) If the connection ever dies, throw away your state and start again. Maybe can even reuse some code.

If you do this, don't remove the existing POST functionality. It's still very useful for debugging, and the various test and fix probes.

[tomwilkie]

This will also need some thought wrt multi tenancy. Some sort of
persistent connection (is a websocket) does indeed seem the way to go.

On Thursday, 18 February 2016, Peter Bourgon notifications@github.com
wrote:

To send an incremental report, the probe must know what the app "already
knows" in order to calculate a diff. That means the current (stateless)
POST method of getting reports to apps is insufficient, and we'll need to
maintain a persistent connection. Given the Go HTTP server keeps the
connection around between POSTs currently, this is probably fine. But for
the cloud offering, for example, we will end up creating a persistent
connection from every node in the customer infrastructure to
whatever.weave.works. Do we think that's acceptable?

I suggest WebSockets, since we're already doing something exactly like
this between the app and the web UI. Upon connection, send the complete
report. Thereafter, send deltas. (Maybe every ~minute, dump the complete
report again, as a sync.) If the connection ever dies, throw away your
state and start again. Maybe can even reuse some code.

If you do this, don't remove the existing POST functionality. It's still
very useful for debugging, and the various test and fix probes.

—
Reply to this email directly or view it on GitHub
#985 (comment).

[2opremio]

To send an incremental report, the probe must know what the app "already knows" in order to calculate a diff.

Not in general due to the CRDT semantics of reports (there are some exceptions to this like metrics we need to figure out, but we can decide not to send those incrementally for now).

We would though need to have strict guarantees on full reports being received periodically which is much easier to obtain than keeping state about what the app knows.

POST method of getting reports to apps is insufficient, and we'll need to maintain a persistent connection

As per the comment above I don't think that's true.

[bboreham]

Done in #3677

the probe must know what the app "already knows" in order to calculate a diff

As described in the PR, this turned out to be unnecessary.