Make statistics collection and aggregation distributed across all cluster nodes

[michaelklishin]

Follow-up to #41. Problem definition is the same and the solution there doesn't work for some workload because a single node collecting and aggregating all stats only can go so far.

So this issue is about making the collector distributed (stats are stored on every cluster node) for 3.6.x.

[sega-yarkin]

Is it make sense to append some parameters to choose which statistics I want to collect? For example, if I need statistics for exchanges and queues, but not need it for channels and connections, it will save some system resources.

[michaelklishin]

Some have asked for this. This may be a good chance to make that possible. Currently exchange and queue stats are emitted by channels so you cannot disable one without the others.

[michaelklishin]

@sega-yarkin let's not turn this issue into a support case. Please take this to the mailing list. Thanks.

[michaelklishin]

We are considering if we should try to target 3.6.x with this. This is easily over half of our support load right now.

Introducing another breaking management plugin version in 3.6.x isn't cool at all, however.

[noahhaon]

@michaelklishin What would the breaking changes be?

[michaelklishin]

@noahhaon mixed clusters will have to all run the new plugin, same as with 3.6.2. No breaking HTTP API changes planned.

[noahhaon]

Do users often upgrade rabbitmq-server without upgrading the plugins? I assume they are all packaged together now? I suppose the issue would be for rolling upgrades, but as long as those failures are handled gracefully ...

As you mentioned, this is a real pain point for large RMQ clusters and appears to be causing support issues for Pivotal. We would certainly love to see this feature, and it seems like it would be worth getting into 3.6.x, despite some pain during a rolling upgrade.

[michaelklishin]

It's not about plugins being out of sync with the server but rather mixed
patch version clusters. But I definitely see your point.

On Thu, Jun 30, 2016 at 7:56 PM, noahhaon notifications@github.com wrote:

Do users often upgrade rabbitmq-server without upgrading the plugins? I
assume they are all packaged together now? I suppose the issue would be for
rolling upgrades, but as long as those failures are handled gracefully ...

As you mentioned, this is a real pain point for large RMQ clusters and
appears to be causing support issues for Pivotal. We would certainly love
to see this feature, and it seems like it would be worth getting into
3.6.x, despite some pain during a rolling upgrade.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#236 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAAEQmrrfYHC-QOi0ncODG6TBSm_zu1Aks5qQ_UlgaJpZM4I9tBc
.

MK

Staff Software Engineer, Pivotal/RabbitMQ

[noahhaon]

Gotcha - well, maybe an out-of-cycle release of the plugin which would run on 3.6.x clusters? Then at least you're not violating least-surprise and breaking versioning semantics by bundling it with a 3.6.x release.

Not sure which is worse from a maintenance perspective, but I'd imagine many users with large clusters (including us) would be quite happy to install the plugin separately if it included this feature.

[michaelklishin]

@noahhaon we are leaning towards shipping it in 3.6.5 or so. Most users would rather upgrade all nodes to 3.6.5 than continue fighting the issues with the existing collector.

[michaelklishin]

A couple of updates:

• So far we intend to ship this in a 3.6.x release
• We will switch to Cowboy at the same time (3.7.0 already uses Cowboy) to reduce the delta between branches. The only user facing change is HTTP API response code changing from 201 to 204 in some cases — virtually no client libraries or users should be affected.

[michaelklishin]

It has been merged and will be dogfooded in stable before cutting a milestone release for the community.