Support multiple metrics libraries

[postwait]

Current the metrics subsystem doesn't support histogram reporting of request latencies. Can we make the metrics reporting interface such that the implementation can be changed out for "gometrics"-style recording that supports histograms?

I'd like to integrate circonus-gometrics, but layered above the go-metrics currently used, all good measurement data is destroyed.

[maier]

just discussed a shim approach which sounds like it will solve the issue. if there is a metric interface in fabio itself then swapping out backends which want/support full raw measurement values will be easier.

[magiconair]

@postwait and @maier I've moved the metrics library implementation behind an interface in the metrics/metricslib package where I've extracted the methods from the go-metrics library I'm currently using. This should make it possible to plug in any metrics library you want as long as it supports this interface. If the interface is not sufficient for what you are trying to do feel free to enhance it but keep it simple and generic if possible. Looking forward to the PRs :)

[magiconair]

I think I missed a spot. The current implementation distinguishes between two different namespaces for metrics: the service metrics and the general fabio metrics (like requests). Service metrics get purged when the service or an instance disappears. This was implemented by using two different registries but that abstraction got lost with this change. I need to add that back in or make the purge code aware of what is a service metric and what isn't. Just keep that in mind when implementing your change.

[maier]

Cool, thanks, I'm working off that branch to do an integration.

[magiconair]

@maier My previous approach was too complex. The whole notion of the metrics package providing two registries was somewhat broken which made the testing more complex than needed to be. I've dropped the metricslib and the gometrics package and rolled this all into the metrics package. There you can now create multiple registries each with a single name space. The main.go then sets up the two name spaces for global and service metrics. That makes it cleaner and simpler IMO but you need to change something in your code. Sorry about that.

Supporting multiple different providers could then be implemented by adding a MultiRegistry analogous to the io.MultiWriter

[maier]

I'll pull down the latest commit and check it out to see what the delta is between what I've got working and the updated method. Ok, I think this will still work. I'll reorganize what I have and kick the tires.

[magiconair]

Merged to master.

[magiconair]

@maier and @postwait please file follow-up tickets for issues with the circonus implementation. This is now on master. It would be great if you could test it and let me know whether it works.

[magiconair]

@maier Thanks for your help with this.

[maier]

I just pulled down master, rebuilt and tested. It works, I like the addition of the status code histograms.

[magiconair]

I've tried this myself with a free Circonus account and can see it working as well. However, I notice a lag of a couple of seconds during startup which I suspect is the authentication with Circonus. I'll check whether I can hide this somewhere.