kv: investigate brown-out while scaling up under load

[tbg]

Describe the problem

It has been reported here (internal link) that scaling up under load results in a period of 0 qps.

To Reproduce

Try to reproduce this in a roachtest. Apparently this has happened under multiple workloads, so that kv50 seems like a good starting point.

Expected behavior

We expect to see a reproduction! QPS dropping to zero for a time is the symptom that was reported.

Additional data / screenshots

The slack link above contains a link to recording where this occurs in a customer environment.

The slack thread also mentions a possible connection to #37906. This is unsubstantiated (since we're purely scaling up) but it should be kept in mind while investigating. #37904, at the time of writing, is not backported to 20.2 (or rather, it was un-backported due to a subtle bug).

Environment:

20.2.x (probably .4)

Additional context

[blathers-crl]

Hi @tbg, I've guessed the C-ategory of your issue and suitably labeled it. Please re-label if inaccurate.

While you're here, please consider adding an A- label to help keep our repository tidy.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[mikeczabator]

[mikeczabator]

Note, this is occurring in 20.2.5, but we also saw it in 20.2.3 and 20.2.4.

[mikeczabator]

More food for thought. When we scale the cluster up, you see the Leaseholders per Store start shuffling all over the place. Leases immediately start shuffling around across many nodes. Perhaps this could be correlated as all of this happens so quickly.

[ajwerner]

This lease transfer behavior feels intimately related to #51867.

[ajwerner]

The related PR was not backported to 20.2. I wonder if it would help. I suspect it might.

[mwang1026]

closing in favor of #67740