Simplifying diagnosis of retries

[tjcelaya]

It is currently impossible to determine how frequently the client is retrying requests without turning on DEBUG logging. Since there are many other logs at this logging level it would be helpful for users if we provided some warning or notice of the fact that we're having trouble performing requests.

Options we've identified include the following:

• Expose a retry rate in the existing PoolStatsMBean (or in a new bean). We can track this from MantaHttpRequestRetryHandler.
• Using the same retry-rate-tracking, log a higher-visibility message (INFO/WARN/ERROR instead of DEBUG) when the retry rate exceeds a certain threshold. We can set a sane default for this if we want to it to be enabled by default but we should be careful to not log this message too frequently.
• Embed an HTTP header in retry requests that can be tracked by the server-side as an indicator of "unhealthy" client activity.

We can do any or all or none of these options. we just need to be clear about why we select our option.

[tjcelaya]

Since not all users track metrics through JMX, part of the requirement for this feature is the ability to expose metrics through logging. Since we're going with dropwizard's metrics library to satisfy option 1, we can leverage the variety of reporters available.

[tjcelaya]

As described in the final comment in #412, we'll need a total request count to provide context for the retry count. After some experimentation with the HttpClient metrics package, I've determined that we can't use that specific package since it relies on providing a custom implementation of HttpRequestExecutor and HttpClientConnectionManager classes from Apache HttpClient (which we already do ourselves). Thankfully, the metrics-httpclient package is relatively small, so it should be straightforward to subclass MantaHttpRequestExecutor to add an instrumented version.