Are we doing anything to prevent the same report being send to the same app twice?

[tomwilkie]

No description provided.

[tomwilkie]

Peter has some idea here. The prototype I hacked up shouldn't be merged.

This probably isn't a release blocker.

[peterbourgon]

The actual issue is that the same app instance may be reachable via several IPs, which all get picked up by the probe via weaveDNS. (Perhaps other methods, too.) Probes broadcast their reports to all upstreams, which is a fundamental error in design, but was something we decided to do in the short-term to help in ease of use.

The proper fix is to stop broadcasting reports to every apps. If we want to keep the current ease-of-use of being able to connect to any weavescope container app and see the same infrastructure, then something along the lines of:

1. Have some kind of peer daemon, ideally as a separate process in the weavescope container
2. The peer connects to all other weavescope peers via existing methods & does master election
3. The peer configures all probes to POST to the master app
4. The peer configures non-master apps to redirect to the master app
5. Details TBD

The quick and dirty fix is probably:

1. Change the app to reply to each POST with its unique ID add a process-lifetime unique ID to the output for /api
2. With each new app target, the probe should POST an empty report GET /api and receive the unique ID
3. Probe builds a map of unique ID to list of app targets (implies sorting)
4. With each publish, probe will send to the first app for a given unique ID
5. Probe will regularly (~60s?) re-resolve its known app targets to deal with changes

[tomwilkie]

Quick & dirty approach sounds fine. Only suggestion would be to not overload the report-posting endpoint, and instead stick the app id in /api (which also has the version, and should be useful in the future). As the resolver calls MultiPublisher.Add() repeatedly, you could fetch /api there and not need the extra timeout for the probe to re-check ids.

[peterbourgon]

Good ideas. Updated.

[tomwilkie]

@peterbourgon And this?

[peterbourgon]

Yep.