cli: add storage engine metrics to debug zip

[anish-shanbhag]

Metrics from the storage engine are already exposed in the /debug/lsm HTTP endpoint. These can be useful when debugging storage issues, and so this change adds these metrics to the debug zip under /nodes/$N/lsm.txt in the same text format as the HTTP route. The previously unused EngineStats status endpoint was repurposed to serve these metrics from each node.

Fixes: #79518
Epic: none
Release note: none

[cockroach-teamcity]

This change is 

[itsbilal]

Great work! Broadly I like the approach here

[itsbilal]

nit: maybe we should call this GetLSMStats instead

[anish-shanbhag]

Agreed, this is fixed

[itsbilal]

if we're changing the protobuf anyway, I'd have a slight preference for repeated string metrics [(gogoproto.nullable=false)], that way we have more infrastructure ready for per-store responses if we ever move in that direction. You can move the concatenation logic into a helper that can then be used by debug.GetEngineStats.

Also convention with changing protobufs is that the new field doens't use the same argument ID (the = 1 at the end) as the old field. You might need to make this 2, and you might also need a reserved 1 above it for the missing 1, but maybe that last part isn't necessary.

[anish-shanbhag]

Thanks for calling out the argument ID convention - I've updated to use = 2.

I also opted to use a store ID -> string map rather than a list of strings so that we can filter by the store ID if needed in the future.

[anish-shanbhag]

I did some more investigation into whether we can just make a request to /debug/lsm to avoid needing an EngineStats endpoint altogether, but I couldn't find anywhere in the zip creation flow where we directly make a request to the debug server to retrieve info. The reference to debug events which I think we saw here isn't actually a call to a debug endpoint, but rather just indicates that events should be stored in debug/events.json. The events themselves are retrieved using a call to the admin.Events gRPC endpoint.

Since the debug zip CLI command is likely to be run on a different host than the node itself, it seems that we would have to find the node's IP in order to make a request to /debug/lsm. I'm not sure how this would look, since it doesn't look like this pattern is used for the other files in the zip.

I also looked into whether we can just include these LSM stats as a field inside the StoreStatus object referenced in #100825, which would also eliminate the need for EngineStats. I think a possible approach could be adding a TODOEngine() storage.Engine field to the storeMetrics interface here which would let us generate the LSM stats via storeMetrics.TODOEngine().GetMetrics().String(). The downside is that these store statuses seem to only be updated once every few seconds rather than generated on demand. Given this, do we think it's best to stick with the current approach with EngineStats, or is it worth trying to consolidate these endpoints despite the slightly stale results?

[itsbilal]

@anish-shanbhag Thanks for looking into the debug endpoints! Given all of what you pointed out, especially the ability to dial other nodes and the ability to call the status server over grpc in addition to http, it makes sense to just use EngineStats for this.

I'm broadly in support of the approach you have here, and we can scope this change to just 1) updating EngineStats to return the LSM stats for each store, 2) pulling EngineStats and adding them to the debug zip. Once this change is merged, we should update #100825 to reflect the update to EngineStats, but otherwise leave that as a standalone issue.

[itsbilal]

LGTM, great work!

[anish-shanbhag]

TFTR!

bors r=itsbilal

[craig]

Build succeeded:

• linux_amd64_fips_build
• docker_image_amd64
• unit_tests
• macos_arm64_build
• check_generated_code
• linux_arm64_build
• lint
• acceptance
• macos_amd64_build
• linux_amd64_build
• local_roachtest
• examples_orms
• windows_build
• local_roachtest_fips