metrics: seperater metrics as internal and external for slo-controller and koordlet

[zwzhang0107]

Ⅰ. Describe what this PR does

seperate koordlet/slo-controller metrics as internal and external
ExternalHTTPPath: for end users, such as batch resource util (also compatible with old URL /metircs for koordlet）
InternalHTTPPath: for cluster admin, such as performance of components.
DefaultHTTPPath: merge all internal, external and controller runtime metrics.

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it

Ⅳ. Special notes for reviews

V. Checklist

• I have written necessary docs and comments
• I have added necessary unit tests and integration tests
• All checks passed in make test

[codecov]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (da6a05c) 67.23% compared to head (f021958) 67.26%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
pkg/util/metrics/merged_gather.go	72.72%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1807      +/-   ##
==========================================
+ Coverage   67.23%   67.26%   +0.03%     
==========================================
  Files         407      410       +3     
  Lines       45644    45662      +18     
==========================================
+ Hits        30687    30715      +28     
+ Misses      12741    12731      -10     
  Partials     2216     2216              

Flag	Coverage Δ
unittests	67.26% <90.32%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[saintube]

Need to update the deploy templates under the /config since the metrics paths are going to change.

[ZiMengSheng]

/milestone v1.5

[zwzhang0107]

Need to update the deploy templates under the /config since the metrics paths are going to change.

the url path has not defined yet both in config and charts

[eahydra]

Is it really necessary to split it into two metrics urls? Users also need to configure two prometheus configurations for slo-controller, and this indicator must be distinguished when adding indicators in future code. Moreover, only two components, koord-manager and koordlet, are involved here. Do scheduler/descheduler also need to be followed? I don't think such a distinction should be made, and it is not a common or recommended practice.

[zwzhang0107]

Is it really necessary to split it into two metrics urls? Users also need to configure two prometheus configurations for slo-controller, and this indicator must be distinguished when adding indicators in future code. Moreover, only two components, koord-manager and koordlet, are involved here. Do scheduler/descheduler also need to be followed? I don't think such a distinction should be made, and it is not a common or recommended practice.

@eahydra Yes, it is necessary. any extended metric will be charged(ref https://help.aliyun.com/zh/ack/ack-managed-and-ack-dedicated/user-guide/dynamic-resource-overcommitment?spm=a2c4g.11186623.0.0.74d118ab3vDxaM#ef961016a2uds), and for end users they only need care about external metrics, this will cost a lot if we do not separate internal and external after we add new metrics about performance.

scheduler/descheduler should also split if they have the same senario in future.

[eahydra]

Is it really necessary to split it into two metrics urls? Users also need to configure two prometheus configurations for slo-controller, and this indicator must be distinguished when adding indicators in future code. Moreover, only two components, koord-manager and koordlet, are involved here. Do scheduler/descheduler also need to be followed? I don't think such a distinction should be made, and it is not a common or recommended practice.

@eahydra Yes, it is necessary. any extended metric will be charged(ref https://help.aliyun.com/zh/ack/ack-managed-and-ack-dedicated/user-guide/dynamic-resource-overcommitment?spm=a2c4g.11186623.0.0.74d118ab3vDxaM#ef961016a2uds), and for end users they only need care about external metrics, this will cost a lot if we do not separate internal and external after we add new metrics about performance.

scheduler/descheduler should also split if they have the same senario in future.

This seems to be solved through the reable_config mechanism of prometheus. Only need to configure the metrics regex and add action=keep.

[zwzhang0107]

reable_config

this is not supported for

Is it really necessary to split it into two metrics urls? Users also need to configure two prometheus configurations for slo-controller, and this indicator must be distinguished when adding indicators in future code. Moreover, only two components, koord-manager and koordlet, are involved here. Do scheduler/descheduler also need to be followed? I don't think such a distinction should be made, and it is not a common or recommended practice.

@eahydra Yes, it is necessary. any extended metric will be charged(ref https://help.aliyun.com/zh/ack/ack-managed-and-ack-dedicated/user-guide/dynamic-resource-overcommitment?spm=a2c4g.11186623.0.0.74d118ab3vDxaM#ef961016a2uds), and for end users they only need care about external metrics, this will cost a lot if we do not separate internal and external after we add new metrics about performance.
scheduler/descheduler should also split if they have the same senario in future.

This seems to be solved through the reable_config mechanism of prometheus. Only need to configure the metrics regex and add action=keep.

prometheus.yaml is not configurable for end user by default in product like ACK

[saintube]

Ⅰ. Describe what this PR does

seperate koordlet/slo-controller metrics as internal and external /external-metrics: for end users, such as batch resource util /metrics: for cluster admin, such as performance of components.

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it

Ⅳ. Special notes for reviews

V. Checklist

• I have written necessary docs and comments
• I have added necessary unit tests and integration tests
• All checks passed in make test

It seems the external metrics are using the path /metrics.

[zwzhang0107]

Ⅰ. Describe what this PR does

seperate koordlet/slo-controller metrics as internal and external /external-metrics: for end users, such as batch resource util /metrics: for cluster admin, such as performance of components.

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it

Ⅳ. Special notes for reviews

V. Checklist

• I have written necessary docs and comments
• I have added necessary unit tests and integration tests
• All checks passed in make test

It seems the external metrics are using the path /metrics.

updated

[saintube]

/lgtm

[FillZpp]

/lgtm

[zwzhang0107]

/approve

[koordinator-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: FillZpp, saintube, zwzhang0107

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/OWNERS [FillZpp]
• pkg/koordlet/OWNERS [saintube,zwzhang0107]
• pkg/slo-controller/OWNERS [zwzhang0107]
• pkg/util/OWNERS [FillZpp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment