PIP-271: Add broker health check status into prometheus metrics #20147

[vineeth1995]

Motivation:

Broker metrics don't have anything to indicate health of the broker (to indicate if broker is active). In Prometheus broker metrics which are used for monitoring, it will be useful if metrics also show the broker health. This way, Prometheus can automatically scrape the broker state and can be used for monitoring purposes. So we need such metric to capture broker health.

Goals:

This PIP adds support to include the broker health status in the broker operability metrics.

Sample:

When we hit "/metrics" endpoint, a part of the output looks like below. Notice the "pulsar_health" metric which is added as a result of this PIP. The status "1" says that broker is active, "0" is inactive, "-1" unknown.

Output

# TYPE pulsar_health gauge
pulsar_health{cluster="standalone", broker="localhost", metric="broker_state"} 1

Approach:

A new metric called "brk_health" is added into the BrokerOperabilityMetrics. This metric is updated at a fixed rate from the BrokerService.

BrokerOperabilityMetrics.java

 Metrics getStateMetrics() {
        Metrics rMetrics = Metrics.create(getDimensionMap("broker_state"));
        rMetrics.put("brk_health", healthCheckStatus);
        return rMetrics;
    }

We schedule a periodic health check job at a fixed rate in the BrokerService. This job updates the broker health check metric in the BrokerOperabilityMetrics stats at the frequency configured in the broker configs.

BrokerService.java

protected void initializeHealthChecker() {
        ServiceConfiguration config = pulsar().getConfiguration();
        if (config.getHealthCheckMetricsUpdateTimeInSeconds() > 0) {
            int interval = config.getHealthCheckMetricsUpdateTimeInSeconds();
            statsUpdater. scheduleAtFixedDelay(this::checkHealth,
                    interval, interval, TimeUnit.SECONDS);
        }
    }

    public CompletableFuture<Void> checkHealth() {
        return internalRunHealthCheck(TopicVersion.V2, pulsar(), null).thenAccept(__ -> {
            this.pulsarStats.getBrokerOperabilityMetrics().recordHealthCheckStatusSuccess();
        }).exceptionally(ex -> {
            this.pulsarStats.getBrokerOperabilityMetrics().recordHealthCheckStatusFail();
            return null;
        });
    }

No new API is needed as we already have a "healthCheck" API in Admin module which provides the necessary functionality. However we don't make a REST call to this API as it could be costly. Instead we add a helper function "internalRunHealthCheck" in the Admin module which piggy backs on the existing functionality in the Admin module.

Configuration Changes:

This PIP gives option to dynamically switch on/off the broker health check metric using "healthCheckMetricsUpdateTimeInSeconds" config. Setting it to -1 will disable the metric. We can also configure the frequency of the metric update using this config. By default it is set to value "-1" which effectively disables it.

broker.conf

healthCheckMetricsUpdateTimeInSeconds=-1 

[vineeth1995]

Link to vote: https://lists.apache.org/thread/c9y3b4z6jotdcbwt5qcvht4ngcpzrzzo

[asafm]

Few comments before you continue with implementation:

1. Please refactor out the core logic in BrokerBase.internalRunHealthCheck into it's own HealthCheckService.
2. Please refrain from adding more code to BrokerService - this class is 3600 lines - what good reason do you have adding more lines to this class? HealthCheckService sounds likes a reasonable place to host the scheduling part.
3. Please don't user scheduleAtFixedRate since you can overload the internal queue of the executor service. You can use scheduleAtFixedDelay.
4. You intend to place rMetrics.put("brk_health", healthCheckStatus); in getConnectionMetrics() but as far as I know, this metric as nothing to do with metrics of connections. Can you please find / create a more suitable place?

[rdhabalia]

healthcheck attribute is another attribute which gets refreshed by regular task same as other tasks like backlog-check, stats-check, inactive-topic, etc. also, this PIP doesn't introduce any new service or implementation rather it only calls existing admin-api on regular interval to refresh the attribute. so, it follows the existing task pattern and don't think it requires a new service class here.
I agree with point (3) and (4) , and I guess it has been taken care in the PIP.

[rdhabalia]

VOTE has passed and this PIP has been approved