Dynamic retention

[grahamking]

Currently the default retention period is 1 hour. That can easily grow to more than the max grpc message size (4 MiB), at which point tokio-console won't connect. Examples of people hitting this:

• fix(console): fix the issue of continuous reconnection caused by decode failure #500
• Continuous memory leak with console_subscriber #184
• Stuck in RECONNECTING #360

We could make the default less than 1 hour, but there's really no minimum safe duration for retention. It depends on how busy the app is and on how much trace data runtime and tokio log.

Instead, we could make the retention period dynamic by shrinking it if the message is too big. I am thinking of a utility function called from add_instrument_subscription and publish that divides retention in half until the update fits in a message. Pseudo-code:

while update.encoded_len() > MAX_MESSAGE_SIZE {
   self.retention /= 2;
   self.cleanup_closed();
   update = make_update_again();
}
// now publish it

Would you be open to a PR that makes the retention period dynamic like this?

[grahamking]

Moved to #503