storage: splitPostApply fails with *roachpb.RaftGroupDeletedError

[tbg]

https://sentry.io/cockroach-labs/cockroachdb/issues/298639766/

*errors.errorString: store.go:1821 *roachpb.RaftGroupDeletedError
  File "github.com/cockroachdb/cockroach/pkg/storage/store.go", line 1821, in splitPostApply
  File "github.com/cockroachdb/cockroach/pkg/storage/replica_proposal.go", line 582, in handleReplicatedEvalResult
  File "github.com/cockroachdb/cockroach/pkg/storage/replica_proposal.go", line 784, in handleEvalResult
  File "github.com/cockroachdb/cockroach/pkg/storage/replica.go", line 3757, in processRaftCommand
  File "github.com/cockroachdb/cockroach/pkg/storage/replica.go", line 2888, in handleRaftReadyRaftMuLocked
...
(5 additional frame(s) were not displayed)

store.go:1821 *roachpb.RaftGroupDeletedError

[bdarnell]

Based on the location of the error, it looks like something like this happened:

1. Node N1 (which has a replica of range R1) crashes (or just falls behind)
2. Range R1 splits, creating R2
3. R2 is rebalanced away from N1 (probably concurrently with the next couple of steps)
4. N1 wakes up and starts catching up
5. N1 learns of the existence of R2 from a raft message, creating a placeholder Replica
6. The rebalance in step 3 completes. N1 learns about it and GC's its replica
7. N1 reaches the split in R1's raft log. It tries to create or update a Replica object for R2, but fails because we have a tombstone.

But step 6 doesn't quite work that way, because R2 could never have become initialized while the split was pending, and we don't currently GC uninitialized replicas. I'm not sure whether there's some path I'm overlooking or if I've gone wrong in one of the other steps.

[cuongdo]

This hasn't happened in the last 3 months, so clearing milestone.

[nvanbenschoten]

This may be related to #21146.

[bdarnell]

Yes, this is almost certainly a duplicate of #21146. Closing in favor of that one which has more detailed analysis.