Elevated latency

[tbg]

Customer: Fakely, Inc
Deployment: Cloud (really roachprod but let's pretend)
Version: 22.2.0

One of our application teams has alerted us to recurring periods of elevated latency (>50ms p99) which cause them to miss their SLAs. We need assistance tracking down the source of the problem.

Grafana

http://35.231.99.87:3000/d/xJ2AWtF4z/new-dashboard?orgId=1&refresh=5s&from=now-15m&to=now

Admin UIs

http://35.185.112.74:26258/
http://34.139.230.17:26258/
http://34.148.94.36:26258/
http://35.227.42.232:26258/
http://34.148.61.172:26258/
http://35.231.99.87:26258/

[tbg]

This is roachprod, export GCE_PROJECT=andrei-jepsen

debug.zip: roachprod get tobias-p99:1 debug.zip