storage: GC system.rangelog

[bdarnell]

We currently allow the system.rangelog table to grow without bound, on the assumption that its rate of growth will be negligible, but this isn't true. On long-lived clusters of many nodes, the rangelog table can grow to a significant fraction of the total database size. We need some sort of garbage collection here (and we should also take the opportunity to evaluate the usefulness of this log and whether we could be writing less data to it).

[a-robinson]

@BramGruneir, is there any legacy reason that we need to keep printing the RangeLogEvent_Info protos the way that we do? I don't see any existing readers of it other than tests. It would break existing readers, but we could save a ton of space if we printed them in a more compact format.

I'm separately going to shrink down the allocator details, because I'm most familiar with what's safe to change there, but there'll still be a bunch of space used in the first chunk of the info column. For example, this:

"UpdatedDesc":{"range_id":32,"start_key":"uw==","end_key":"u4mAoBdEMtxE94Q=","replicas":[{"node_id":1,"store_id":1,"replica_id":1},{"node_id":5,"store_id":5,"replica_id":2},{"node_id":2,"store_id":2,"replica_id":3},{"node_id":6,"store_id":6,"replica_id":4}],"next_replica_id":5},"AddReplica":{"node_id":6,"store_id":6,"replica_id":4}

Could just as easily be printed using the more compact RangeDescriptor.String() method, making it:

"UpdatedDesc":"r32:/Table/51{-/1/-6910…} [(n1,s1):1, (n5,s5):2, (n2,s2):3, (n6,s6):4,
 next=5]","AddReplica":"(n6,s6):4"

[a-robinson]

I looked into the above questions and found that the RangeLog admin endpoint relies on being able to parse the aforementioned fields, so we'd have to only start writing the more compact versions after all nodes have updated to v2.0 or later, and we'd have to still support parsing the old version. So it'd be a bit of work, perhaps more than it's worth. There aren't any other dependencies in our code base, though, so the only question mark is whether anyone externally is parsing these entries, which seems unlikely

[BramGruneir]

I think the best of way doing this would be with a proper migration. It would be great to consider normalizing the table a bit, or changing the string columns to json and adding an index or two.

For now, it might be a lot easier to just delete all tail entires after some limit to stop the table from getting too big. Even better would be to limit the number of entries per range, but without an index, i worry about how slow that transaction might be.

[kaavee315]

Is someone working on it actively? If not, we were planning to take this up. I have some ideas and questions regarding the same:-

1. I see the various implementations of replicaQueues like GCQueue, raftLogQueue etc. which scan all the replicas and process the required parts. We can implement this as a similar rangeLog queue as part of the store. I think the issue over here is that this framework is at KV layer level and not at SQL level, so reading the data and finding what to be deleted might be a trouble.
2. Is there a similar framework of processing data periodically at SQL layer level? I couldn't find any.
3. What should be the criteria to decide if we want to delete a particular entry? Should it just be a limit per rangeId?

[bdarnell]

No one is currently working on this.

I don't think the replica queues are a great fit for this because they work at the wrong level. I think we'd need something new at the SQL level to handle the GC of this data. Ideally it would be flexible enough to handle TTLs for any table instead of something specific to system.rangelog. This is a commonly-requested feature (#20239).

What should be the criteria to decide if we want to delete a particular entry? Should it just be a limit per rangeId?

Ranges are an implementation detail and should not be used in deciding when to GC a row. Either make it based on the timestamp in the row or the total number of rows in the table. (I'd generally prefer to make it time-based).

[tbg]

I don't really see this in the CF&S area until we add row-level TTLs (#20239). The more likely option right now is some periodic job that deletes from the table (via SQL). @jordanlewis feel free to move this elsewhere. I'm leaving in CF&S mostly because I don't know where to put it.