[proposal] Support for Mid tier

[hormes]

In the resource model chapter, we mentioned the type of Mid tier, which is used to support long-running jobs. Such jobs are typically found in scenarios such as big data computing, real-time computing, and AI training. This issue is used to track work related to Mid tier support.

On the current framework, things that need to be done to support Mid tier include:

• Long-period peak prediction. In order not to increase excessive resource overhead (including third-party dependencies), this peak prediction is best done on the node side.
  • a histogram statistics tool
  • collecte metrics and store in the histogram
  • checkpoint histogram and restore
  • peak prediction algorithm
• The slo-controller needs to be aware of this prediction, at least at the Priority band level, to calculate the amount of schedulable resources.
  • API of nodemetrics
  • report the prediction metrics
• The koord-scheduler needs to be aware of the amount of schedulable resources, similar to how batch obtains by extended resources.
  • API of node allocable resources
  • calculate the total schedulable resources and update the extended resources

[saintube]

/area koord-scheduler
/area koord-manager
/area koordlet