[xCluster] Handle Pausing/Resuming Test Clusters with Replication

[nspiegelberg]

Jira Link: DB-4612
During local SDET testing, we ran into issues with Pause/Resume of a Cluster that was setup for xCluster Replication. The Dev Cluster had been paused for 2 days and the resume failed (https://yugabyte.slack.com/archives/C4141D60H/p1632120259192700). I was pulled in because of log spew. It seems to be a race between [1] the Tablet Log GC, which would kick in on startup and remove all logs and [2] the Consumer, which would call GetChanges on the next OpID after the pause occurred.

The xCluster GC logic is in LogReader::GetSegmentPrefixNotIncluding, which provides the candidate files for GetSegmentsToGCUnlocked. 4 variables control GC, as far as we care (precedence):

1. MinSpacePolicy. CDC Policy. Include a file for GC if our space is too low, log_stop_retaining_min_disk_mb (100GB)
2. MaxTimePolicy. CDC Policy. Include all files for GC that are older than a GFLAG, log_max_seconds_to_retain. (24hr)
3. cdc_max_replicated_index. CDC Policy. Include all files for GC that CDC has read past.
4. wal_retention_secs. Global Policy. This is used at the higher layer. Skip candidate files newer than log_min_seconds_to_retain (15min). xCluster overrides this to cdc_wal_retention_time_secs (4 hr).

The above pause triggered MaxTimePolicy, it would’ve been GC’d regardless of cdc_max_replicated_index. It probably didn’t seem like a big deal when we initially wrote it and we likely weren’t thinking of the pause cluster use cases. MinSpacePolicy seems like the important heuristic to keep here & we should get rid of MaxTimePolicy to support this use case. Probably should keep it under a GFLAG in case there are some downsides we're not seeing.

Additionally: cdc_wal_retention_time_secs seems to be necessary because there’s no synchronization in setting cdc_max_replicated_index on the TServers from Master right now during Bootstrap. We raise the limit with an AlterTable, then we write the cdc_state table with the Replicated OpIds. We should be able to consolidate.

[nspiegelberg]

When looking into this, we should also audit the design of FLAGS_cdc_min_replicated_index_considered_stale_secs. This looks like it could accidentally GC valid logs if we're not careful.

[nspiegelberg]

One thought on how to support this use case:

1. Size policy kicks in immediately.
2. Time policy isn't used for the first N minutes after the process is started.
3. Do not collect on Time policy if CDC is updating it's checkpoint and the overall retention size is decreasing.