[proposal] Support the resource management for Mid-tier pods.

[saintube]

What is your proposal:

Support the resource management for Mid-tier pods in the koordlet:

• Identify the extended resources of Mid-tier.
  • In the runtime hook, Set the necessary cgroups for Mid+BE pods/containers, just like the BatchResource hook.
  • In the metrics module, export the node allocatable and pod requests/limits about Mid resources. koordlet: export mid resource metrics #1874
• Distinguish pods with priority classes (non-Prod) apart from the QoS. koordlet: fix scope of prodReclaimablePredictor resource reclaim #1469

Why is this needed:

The #1361 adds the Mid-tier resource, allowing a pod with Mid priority class to allocate kubernetes.io/mid-cpu and kubernetes.io/mid-memory resources. However, the resource management of Mid-tier is not fully supported in the koordlet. For example, the koordlet does not set the related cgroups like cpu.cfs_quota_us for a Mid+BE pod.

[saintube]

/area koordlet

[stale]

This issue has been automatically marked as stale because it has not had recent activity.
This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the issue is closed
  You can:
• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close
  Thank you for your contributions.

[saintube]

/remove-lifecycle stale

[stale]

This issue has been automatically marked as stale because it has not had recent activity.
This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the issue is closed
  You can:
• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close
  Thank you for your contributions.

[saintube]

/remove-lifecycle stale

[j4ckstraw]

#1874 export mid resource metric