bulk: increased split-after size can result in replica imbalance

[nicktrav]

Describe the problem

Spin-off from #68303 - provided a distilled version. Context on original, here.

It appears as though e12c9e6 causes replica imbalance which doesn't play nicely with large imports.

With that commit, on clearrange/checks=true, the import fails as a number of nodes run out of disk due to the imbalance:

Without that commit the import succeeds:

Jira issue: CRDB-12773

[nicktrav]

cc: @dt

[blathers-crl]

cc @cockroachdb/bulk-io

[dt]

Reverting in #75882 for now, will try to find some time next week to poke at this, and see why the closer-to-default split size does so badly / what the extremely low split size was doing that was better for the allocator.

Maybe the bigger range counts were covering for scatter's known padding issues? Might need @aayushshah15 to help me poke at this when I get to it.

[aayushshah15]

Maybe the bigger range counts were covering for scatter's known padding issues? Might need @aayushshah15 to help me poke at this when I get to it.

I'd still expect the allocator to fix straight-up replica count divergence though. Very curious why this was happening. Feel free to put something on my calendar if you want to look together, but I can also just try repro-ing on my own with the split size bumped.

[aayushshah15]

I am playing with this a little bit (i.e. running with 384mb with our patch for AdminScatter) on a tpcc import. Repro steps are identical to what I've described in that PR, except for the increased split-after size.

The cluster is at http://104.196.120.245:26258/#/metrics/replication/cluster

The main things that I've noticed so far are that, with 384mb, we essentially stop scattering after the very beginning of the import. My sense is that this runs contrary to our understanding (also something that wasn't happening with 48mb)? Yellow is rebalances.

Additionally, as far as I can tell, the allocator is continuously rebalancing. The rate at which we're converging is just slower than the rate at which newer splits are being created. After the import is done, I'd expect this to fully converge.

        {
          "time": "2022-02-03T15:12:34.421667145Z",
          "message": "kv/kvserver/allocator_scorer.go:186 [n7,status] s7: should-rebalance(ranges-overfull): rangeCount=1909, mean=1291.00, overfull-threshold=1356"
        },

[dt]

fixed by #77588