Reduce probe cpu usage

[tomwilkie]

2 issues - report generation is taking too long, and cpu usage is too high.

Reducing CPU usage (~40% to <20%), see #284

• WithNode -> AddNode, don't copy all the nodes every time we add a single one.
• Cache generated endpoint IDs to relieve some more GC pressure.
• Only gob serialise and gz compress reports once per publish cycle, not once per app.
• In weave tagger, stop iterating through all containers to find a container by prefix. Build an index instead.

Report generation:

• Log a warning when it takes too long
• Generate them in parallel
• Only fetch weave status json once per cycle (fixes Do fewer RPCs to Weave #397)

[peterbourgon]

Sorry about commenting on the commits rather than the diff view. That makes it a little harder to act on.

Back over to you.

[tomwilkie]

synchronisation

Is it needed? Tick and {Report, Tag} are never call concurrently. If it is, then we'll also need a pointer receiver I think?

what else does conntrack do?

It lists (or streams) connections; udp, tcp etc, and will list of stream nat mappings. I don't think it does anything else.

[tomwilkie]

@peterbourgon I think I addressed all the issues.

[peterbourgon]

Is it needed? Tick and {Report, Tag} are never call concurrently.

That's incidental and can easily and inadvertently change :)

If it is, then we'll also need a pointer receiver I think?

Yes.

[peterbourgon]

@tomwilkie I made the changes I proposed. Give me a 👍 and I'll merge.

[tomwilkie]

LGTM