kafka: Add fetch plan and execute latency metric

[StephanDollberg]

Adds a histogram metric to measure the time it takes to create the fetch
plan and execute it - aka a single fetch poll.

It's an approximation for the time it takes to process the data in a
fetch request once it is available.

I have separated two series one which is tracking empty fetches and one
that isn't.

Further the count of the histogram can be used to calculate the ratio of
fetch requests to polls like so:

sum(irate(vectorized_kafka_handler_requests_completed_total{...,
handler="fetch"}[$__rate_interval])) by ($aggr_criteria) /
sum(irate(vectorized_fetch_stats_plan_and_execute_latency_us_count{...}[$__rate_interval])) by
($aggr_criteria)

Looking at some scenarios we get the following values:

• 500MB/s, 4P/4C, 288P, ~110k batch, 1ms debounce: ~0.37
• 500MB/s, 4P/4C, 288P, ~110k batch, 10ms debounce: ~0.66
• 125MB/s, 8kP/8kC, 40k partitions, 1ms debounce: ~0.012
• 125MB/s, 8kP/8kC, 40k partitions, 10ms debounce: ~0.035
• 125MB/s, 8kP/8kC, 40k partitions, 100ms debounce: ~0.24

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.2.x
• v23.1.x
• v22.3.x

Release Notes

Improvements

• Adds a metric to track fetch plan and execute latency

[ballard26]

LGTM. Are we planning on removing or renaming the existing kafka_latency_fetch_latency_us metric which records per shards latency for each poll?

[StephanDollberg]

LGTM. Are we planning on removing or renaming the existing kafka_latency_fetch_latency_us metric which records per shards latency for each poll?

That is my longterm goal yes.

[vbotbuildovich]

ducktape was retried in job https://buildkite.com/redpanda/redpanda/builds/40383#018b9544-60c5-4e05-84b0-79caa19405da

[vbotbuildovich]

ducktape was retried in job https://buildkite.com/redpanda/redpanda/builds/40383#018b9554-c2f8-4a44-b0ac-850f74666cdb

[vbotbuildovich]

ducktape was retried in job https://buildkite.com/redpanda/redpanda/builds/40383#018b9554-c2f2-4968-a106-095df4014e20

[travisdowns]

LGTM and thanks for the example numbers in the patch description: very useful!

[StephanDollberg]

Failure is: #14254

[dotnwat]

as always, amazing commit messages @StephanDollberg

[vbotbuildovich]

/backport v23.2.x

[vbotbuildovich]

/backport v23.1.x

[vbotbuildovich]

Failed to create a backport PR to v23.1.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-13485-v23.1.x-543 remotes/upstream/v23.1.x
git cherry-pick -x 220b11ed449bc553cd4c69b830b976f8d01db646

Workflow run logs.

[vbotbuildovich]

Failed to create a backport PR to v23.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-13485-v23.2.x-8 remotes/upstream/v23.2.x
git cherry-pick -x 220b11ed449bc553cd4c69b830b976f8d01db646

Workflow run logs.