Tablet Throttler: v18 features and objectives

[shlomi-noach]

v17 is about to be released, and its features are frozen. The following are changes I'd like to build in the tablet throttler for v18:

• Removal of old command line flag based throttler configuration. v17 enables throttler-config-via-topo by default. In v18 it will be the only option, and in fact the very flag throttler-config-via-topo will be deprecated. Work already submitted in Deprecating and removing tablet throttler CLI flags and tests #13246
• Formalize throttled apps in topo. Today, throttling apps is in vttablet memory, and only for the PRIMARY tablet. Throttled apps should follow suit with the rest of throttler configuration to be topo-based (v18 will be a transition version where both are supported). This will either be in UpdateThrottlerConfig or in a different command, to be seen.
  The suggestion was made by the community in the June 23 edition of Vitess Monthly Meeting.
  Tablet throttler: throttled app configuration via vtctl UpdateThrottlerConfig #13351
• Support SQL interface for throttling apps. Today there's a specific dialect for throttling Online DDL migrations, but we'd like to have a general syntax.
  Reroute 'ALTER VITESS_MIGRATION ... THROTTLE ...' through topo #13511 for Online DDL specifically.
• Be able to exempt apps from being throttled. Foundations for this have been laid by Tablet throttler: be explicit about client app name, exempt some apps from checks and heartbeat renewals #13195. Much like we can request specific apps to be more throttled, we want to be able to completely exempt (with timeout) some apps from being throttled, giving them a free pass to work. This has the risk of starving other apps, but it's a decision the user will make at their discretion.
• Either:
  • Change inter-throttler communication away from HTTP and into gRPC
    Tablet throttler: inter-checks via gRPC  #13514
  • or, use table heath realtime stats subscriptions. See Tablet throttler: (feature flagged) get remote tablets metrics from Realtime Stats #13018 for WIP. We'd need to see if we can have different realtime stats streams/subscriptions as we don't want to flood all subscribers (e.g. vtgates) with throttler-specific information.

[shlomi-noach]

This was all done.