[Improvement] Enable set ServerLoadProtection fot Master/Worker #15439

ruanwenjun · 2024-01-07T15:18:04Z

Purpose of the pull request

Refactor the master/ worker server overload logic.
Enable closing the overload protection
Support set jvmMemory/SystemMemory/cpuUsage/diskUsage protection.
Get the metrics from actuator.

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

...meter/src/main/java/org/apache/dolphinscheduler/meter/metrics/PrometheusMetricsProvider.java

codecov-commenter · 2024-01-08T02:29:55Z

Codecov Report

Attention: 119 lines in your changes are missing coverage. Please review.

Comparison is base (a405abe) 38.11% compared to head (246a274) 38.00%.

❗ Current head 246a274 differs from pull request most recent head 704e6fc. Consider uploading reports for the commit 704e6fc to get more accurate results

Files	Patch %	Lines
...rver/master/config/MasterServerLoadProtection.java	30.00%	13 Missing and 1 partial ⚠️
...ler/server/master/metrics/MasterServerMetrics.java	0.00%	12 Missing ⚠️
...rver/worker/config/WorkerServerLoadProtection.java	40.00%	9 Missing and 3 partials ⚠️
...eduler/server/master/task/MasterHeartBeatTask.java	0.00%	11 Missing ⚠️
...e/dolphinscheduler/server/master/MasterServer.java	0.00%	10 Missing ⚠️
...e/dolphinscheduler/server/worker/WorkerServer.java	0.00%	9 Missing ⚠️
...lphinscheduler/common/model/BaseHeartBeatTask.java	0.00%	8 Missing ⚠️
...inscheduler/alert/registry/AlertHeartbeatTask.java	0.00%	7 Missing ⚠️
...r/server/master/registry/MasterRegistryClient.java	14.28%	5 Missing and 1 partial ⚠️
...r/master/dispatch/host/LowerWeightHostManager.java	0.00%	5 Missing ⚠️
... and 12 more

Additional details and impacted files

@@             Coverage Diff              @@
##                dev   #15439      +/-   ##
============================================
- Coverage     38.11%   38.00%   -0.11%     
+ Complexity     4697     4686      -11     
============================================
  Files          1299     1304       +5     
  Lines         44777    44769       -8     
  Branches       4797     4798       +1     
============================================
- Hits          17067    17015      -52     
- Misses        25863    25904      +41     
- Partials       1847     1850       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

deploy/kubernetes/dolphinscheduler/values.yaml

...uler-common/src/main/java/org/apache/dolphinscheduler/common/model/AlertServerHeartBeat.java

...master/src/main/java/org/apache/dolphinscheduler/server/master/task/MasterHeartBeatTask.java

                               @NonNull RegistryClient registryClient) {
-        super("MasterHeartBeatTask", masterConfig.getHeartbeatInterval().toMillis());
+        super("MasterHeartBeatTask", masterConfig.getMaxHeartbeatInterval().toMillis());


...worker/src/main/java/org/apache/dolphinscheduler/server/worker/task/WorkerHeartBeatTask.java

-                               @NonNull Supplier<Integer> workerWaitingTaskCount) {
-        super("WorkerHeartBeatTask", workerConfig.getHeartbeatInterval().toMillis());
+                               @NonNull WorkerTaskExecutorThreadPool workerTaskExecutorThreadPool) {
+        super("WorkerHeartBeatTask", workerConfig.getMaxHeartbeatInterval().toMillis());


sonarcloud · 2024-01-08T11:14:53Z

Please retry analysis of this Pull-Request directly on SonarCloud

Gallardot · 2024-01-09T06:40:39Z

...c/main/java/org/apache/dolphinscheduler/server/master/config/MasterServerLoadProtection.java

+        }
+        if (systemMetrics.getSystemMemoryUsedPercentage() > maxSystemMemoryUsagePercentageThresholds) {
+            log.info(
+                    "Worker OverLoad: the SystemMemoryUsedPercentage: {} is over then the MaxSystemMemoryUsagePercentageThresholds {}",


...c/main/java/org/apache/dolphinscheduler/server/master/config/MasterServerLoadProtection.java

caishunfeng · 2024-01-10T05:45:23Z

...st/java/org/apache/dolphinscheduler/server/master/config/MasterServerLoadProtectionTest.java

+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+class MasterServerLoadProtectionTest {


caishunfeng · 2024-01-10T05:45:48Z

...st/java/org/apache/dolphinscheduler/server/master/config/MasterServerLoadProtectionTest.java

+                .jvmMemoryUsedPercentage(0.71)
+                .systemMemoryUsedPercentage(0.71)
+                .totalCpuUsedPercentage(0.71)
+                .diskUsedPercentage(0.71)


Suggested change

.jvmMemoryUsedPercentage(0.71)

.systemMemoryUsedPercentage(0.71)

.totalCpuUsedPercentage(0.71)

.diskUsedPercentage(0.71)

.jvmMemoryUsedPercentage(0.81)

.systemMemoryUsedPercentage(0.81)

.totalCpuUsedPercentage(0.81)

.diskUsedPercentage(0.81)

I change the default configuration to 0.7, so this test case doesn't need to change.

caishunfeng

LGTM

sonarcloud · 2024-01-13T04:07:08Z

Quality Gate failed

Failed conditions

29.7% Coverage on New Code (required ≥ 60%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

ruanwenjun requested review from zhongjiajie, EricGao888, caishunfeng and SbloodyS as code owners January 7, 2024 15:18

github-actions bot added backend document labels Jan 7, 2024

ruanwenjun force-pushed the dev_wenjun_addServerProtection branch 2 times, most recently from f40cc59 to 0d5b555 Compare January 7, 2024 15:45

github-advanced-security bot found potential problems Jan 7, 2024

View reviewed changes

...meter/src/main/java/org/apache/dolphinscheduler/meter/metrics/PrometheusMetricsProvider.java Fixed Show fixed Hide fixed

ruanwenjun force-pushed the dev_wenjun_addServerProtection branch 4 times, most recently from a2f0c99 to 9f46edb Compare January 8, 2024 02:11

ruanwenjun force-pushed the dev_wenjun_addServerProtection branch 3 times, most recently from d11ef47 to d18c963 Compare January 8, 2024 05:48

Gallardot reviewed Jan 8, 2024

View reviewed changes

deploy/kubernetes/dolphinscheduler/values.yaml Outdated Show resolved Hide resolved

deploy/kubernetes/dolphinscheduler/values.yaml Outdated Show resolved Hide resolved

deploy/kubernetes/dolphinscheduler/values.yaml Outdated Show resolved Hide resolved

ruanwenjun force-pushed the dev_wenjun_addServerProtection branch from d18c963 to d6d30cc Compare January 8, 2024 06:53

Gallardot reviewed Jan 8, 2024

View reviewed changes

deploy/kubernetes/dolphinscheduler/values.yaml Outdated Show resolved Hide resolved

ruanwenjun force-pushed the dev_wenjun_addServerProtection branch 3 times, most recently from 6408e09 to 0b51840 Compare January 8, 2024 08:58

github-advanced-security bot found potential problems Jan 8, 2024

View reviewed changes

ruanwenjun force-pushed the dev_wenjun_addServerProtection branch 3 times, most recently from 8f66358 to 91b0eca Compare January 8, 2024 10:50

github-actions bot added the test label Jan 8, 2024

ruanwenjun force-pushed the dev_wenjun_addServerProtection branch 4 times, most recently from 364fae9 to 306f4ca Compare January 9, 2024 02:25

ruanwenjun requested a review from Gallardot January 9, 2024 03:16

ruanwenjun force-pushed the dev_wenjun_addServerProtection branch from 306f4ca to 73270ee Compare January 9, 2024 06:11

Gallardot reviewed Jan 9, 2024

View reviewed changes

ruanwenjun added the improvement make more easy to user or prompt friendly label Jan 10, 2024

ruanwenjun changed the title ~~[Feature] Enable set ServerLoadProtection fot Master/Worker~~ [Improvement] Enable set ServerLoadProtection fot Master/Worker Jan 10, 2024

caishunfeng reviewed Jan 10, 2024

View reviewed changes

...c/main/java/org/apache/dolphinscheduler/server/master/config/MasterServerLoadProtection.java Outdated Show resolved Hide resolved

caishunfeng reviewed Jan 10, 2024

View reviewed changes

...c/main/java/org/apache/dolphinscheduler/server/master/config/MasterServerLoadProtection.java Outdated Show resolved Hide resolved

caishunfeng reviewed Jan 10, 2024

View reviewed changes

...c/main/java/org/apache/dolphinscheduler/server/master/config/MasterServerLoadProtection.java Show resolved Hide resolved

caishunfeng reviewed Jan 10, 2024

View reviewed changes

ruanwenjun force-pushed the dev_wenjun_addServerProtection branch from 73270ee to c99f1ea Compare January 10, 2024 05:48

ruanwenjun requested a review from caishunfeng January 11, 2024 02:00

ruanwenjun force-pushed the dev_wenjun_addServerProtection branch from c99f1ea to 2cadae0 Compare January 11, 2024 16:08

ruanwenjun requested a review from Gallardot January 11, 2024 16:08

caishunfeng assigned ruanwenjun Jan 12, 2024

caishunfeng approved these changes Jan 12, 2024

View reviewed changes

caishunfeng added the ready-to-merge label Jan 12, 2024

caishunfeng added this to the 3.3.0 milestone Jan 12, 2024

Enable set ServerLoadProtection fot Master/Worker

704e6fc

ruanwenjun force-pushed the dev_wenjun_addServerProtection branch from bf49597 to 704e6fc Compare January 13, 2024 03:40

ruanwenjun merged commit 7b9c9e0 into apache:dev Jan 14, 2024
58 of 87 checks passed

ruanwenjun deleted the dev_wenjun_addServerProtection branch January 14, 2024 14:18

Gallardot mentioned this pull request Jan 29, 2024

[Improvement][K8S] Remove overload protection when deploying on k8s #15066

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Improvement] Enable set ServerLoadProtection fot Master/Worker #15439

[Improvement] Enable set ServerLoadProtection fot Master/Worker #15439

ruanwenjun commented Jan 7, 2024

codecov-commenter commented Jan 8, 2024 •

edited

Loading

sonarcloud bot commented Jan 8, 2024

Gallardot Jan 9, 2024

ruanwenjun Jan 10, 2024

caishunfeng Jan 10, 2024

caishunfeng Jan 10, 2024

ruanwenjun Jan 10, 2024

caishunfeng left a comment

sonarcloud bot commented Jan 13, 2024

[Improvement] Enable set ServerLoadProtection fot Master/Worker #15439

[Improvement] Enable set ServerLoadProtection fot Master/Worker #15439

Conversation

ruanwenjun commented Jan 7, 2024

Purpose of the pull request

Brief change log

Verify this pull request

codecov-commenter commented Jan 8, 2024 • edited Loading

Codecov Report

sonarcloud bot commented Jan 8, 2024

Gallardot Jan 9, 2024

Choose a reason for hiding this comment

ruanwenjun Jan 10, 2024

Choose a reason for hiding this comment

caishunfeng Jan 10, 2024

Choose a reason for hiding this comment

caishunfeng Jan 10, 2024

Choose a reason for hiding this comment

ruanwenjun Jan 10, 2024

Choose a reason for hiding this comment

caishunfeng left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Jan 13, 2024

Quality Gate failed

codecov-commenter commented Jan 8, 2024 •

edited

Loading