forked from ray-project/ray
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add perf metrics for 2.36.0 (ray-project#47574)
``` REGRESSION 12.66%: single_client_get_object_containing_10k_refs (THROUGHPUT) regresses from 13.204885454613315 to 11.533423619760748 in microbenchmark.json REGRESSION 9.50%: client__1_1_actor_calls_sync (THROUGHPUT) regresses from 523.3469473257671 to 473.62862729568997 in microbenchmark.json REGRESSION 6.76%: multi_client_put_gigabytes (THROUGHPUT) regresses from 45.440179854469804 to 42.368678421213005 in microbenchmark.json REGRESSION 4.92%: 1_n_actor_calls_async (THROUGHPUT) regresses from 8803.178389859915 to 8370.014425096557 in microbenchmark.json REGRESSION 3.89%: n_n_actor_calls_with_arg_async (THROUGHPUT) regresses from 2748.863962184806 to 2641.837605625889 in microbenchmark.json REGRESSION 3.45%: client__1_1_actor_calls_async (THROUGHPUT) regresses from 1019.3028285821217 to 984.156036006501 in microbenchmark.json REGRESSION 3.06%: client__1_1_actor_calls_concurrent (THROUGHPUT) regresses from 1007.6444648899972 to 976.8103650114274 in microbenchmark.json REGRESSION 0.65%: placement_group_create/removal (THROUGHPUT) regresses from 805.1759941825478 to 799.9345402492929 in microbenchmark.json REGRESSION 0.33%: single_client_put_calls_Plasma_Store (THROUGHPUT) regresses from 5273.203424794718 to 5255.898134426729 in microbenchmark.json REGRESSION 0.02%: 1_1_actor_calls_async (THROUGHPUT) regresses from 9012.880467992636 to 9011.034048587637 in microbenchmark.json REGRESSION 0.01%: client__put_gigabytes (THROUGHPUT) regresses from 0.13947664668408546 to 0.13945791828216536 in microbenchmark.json REGRESSION 0.00%: client__put_calls (THROUGHPUT) regresses from 806.1974515278531 to 806.172478450918 in microbenchmark.json REGRESSION 70.55%: dashboard_p50_latency_ms (LATENCY) regresses from 104.211 to 177.731 in benchmarks/many_actors.json REGRESSION 13.13%: time_to_broadcast_1073741824_bytes_to_50_nodes (LATENCY) regresses from 18.961532712000007 to 21.451945214000006 in scalability/object_store.json REGRESSION 4.50%: 3000_returns_time (LATENCY) regresses from 5.680022101000006 to 5.935367576000004 in scalability/single_node.json REGRESSION 3.96%: avg_iteration_time (LATENCY) regresses from 0.9740754842758179 to 1.012664566040039 in stress_tests/stress_test_dead_actors.json REGRESSION 2.75%: stage_2_avg_iteration_time (LATENCY) regresses from 63.694758081436156 to 65.44879236221314 in stress_tests/stress_test_many_tasks.json REGRESSION 1.66%: 10000_args_time (LATENCY) regresses from 17.328640389999997 to 17.61703060299999 in scalability/single_node.json REGRESSION 1.40%: stage_4_spread (LATENCY) regresses from 0.45063567085147194 to 0.4569625792772166 in stress_tests/stress_test_many_tasks.json REGRESSION 0.69%: dashboard_p50_latency_ms (LATENCY) regresses from 3.347 to 3.37 in benchmarks/many_pgs.json REGRESSION 0.19%: 10000_get_time (LATENCY) regresses from 23.896780481999997 to 23.942006032999984 in scalability/single_node.json ``` Signed-off-by: kevin <kevin@anyscale.com>
- Loading branch information
1 parent
b52a38f
commit d5f1a01
Showing
11 changed files
with
169 additions
and
169 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,32 +1,32 @@ | ||
| { | ||
| "_dashboard_memory_usage_mb": 485.167104, | ||
| "_dashboard_memory_usage_mb": 429.867008, | ||
| "_dashboard_test_success": true, | ||
| "_peak_memory": 3.74, | ||
| "_peak_process_memory": "PID\tMEM\tCOMMAND\n1215\t8.81GiB\t/app/product/go/infra/anyscaled/anyscaled_/anyscaled startv2 --control_plane_url=https://console.any\n6550\t1.74GiB\t/home/ray/anaconda3/lib/python3.9/site-packages/ray/core/src/ray/gcs/gcs_server --log_dir=/tmp/ray/s\n7868\t0.92GiB\tpython distributed/test_many_actors.py\n2138\t0.36GiB\tvector --watch-config --log-format json --config-yaml /etc/vector/vector.yaml\n6668\t0.35GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.9/site-packages/ray/dashboard/dashboa\n6842\t0.09GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.9/site-packages/ray/dashboard/agen\n2403\t0.07GiB\t/usr/bin/python3 /app/infra/dataplane/webterminal/webterminal_sidecar_image.binary.runfiles/product/\n6844\t0.07GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.9/site-packages/ray/_private/runti\n7668\t0.07GiB\tray::JobSupervisor\n6868\t0.07GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/jupyter-lab --allow-root --ip=127.0.0.1 --no-", | ||
| "actors_per_second": 603.6854672610009, | ||
| "_peak_memory": 3.8, | ||
| "_peak_process_memory": "PID\tMEM\tCOMMAND\n1173\t7.02GiB\t/app/product/go/infra/anyscaled/anyscaled_/anyscaled startv2 --control_plane_url=https://console.any\n3408\t1.75GiB\t/home/ray/anaconda3/lib/python3.9/site-packages/ray/core/src/ray/gcs/gcs_server --log_dir=/tmp/ray/s\n4371\t0.93GiB\tpython distributed/test_many_actors.py\n1954\t0.46GiB\tvector --watch-config --log-format json --config-yaml /etc/vector/vector.yaml\n3524\t0.37GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.9/site-packages/ray/dashboard/dashboa\n3687\t0.09GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.9/site-packages/ray/dashboard/agen\n2361\t0.08GiB\t/usr/bin/python3 /app/infra/dataplane/webterminal/webterminal_sidecar_image.binary.runfiles/product/\n3689\t0.07GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.9/site-packages/ray/_private/runti\n4167\t0.07GiB\tray::JobSupervisor\n3712\t0.07GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/jupyter-lab --allow-root --ip=127.0.0.1 --no-", | ||
| "actors_per_second": 637.7871696983945, | ||
| "num_actors": 10000, | ||
| "perf_metrics": [ | ||
| { | ||
| "perf_metric_name": "actors_per_second", | ||
| "perf_metric_type": "THROUGHPUT", | ||
| "perf_metric_value": 603.6854672610009 | ||
| "perf_metric_value": 637.7871696983945 | ||
| }, | ||
| { | ||
| "perf_metric_name": "dashboard_p50_latency_ms", | ||
| "perf_metric_type": "LATENCY", | ||
| "perf_metric_value": 104.211 | ||
| "perf_metric_value": 177.731 | ||
| }, | ||
| { | ||
| "perf_metric_name": "dashboard_p95_latency_ms", | ||
| "perf_metric_type": "LATENCY", | ||
| "perf_metric_value": 3768.817 | ||
| "perf_metric_value": 3350.577 | ||
| }, | ||
| { | ||
| "perf_metric_name": "dashboard_p99_latency_ms", | ||
| "perf_metric_type": "LATENCY", | ||
| "perf_metric_value": 3786.709 | ||
| "perf_metric_value": 3361.16 | ||
| } | ||
| ], | ||
| "success": "1", | ||
| "time": 16.56491756439209 | ||
| "time": 15.679211616516113 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,32 +1,32 @@ | ||
| { | ||
| "_dashboard_memory_usage_mb": 127.606784, | ||
| "_dashboard_memory_usage_mb": 196.333568, | ||
| "_dashboard_test_success": true, | ||
| "_peak_memory": 2.17, | ||
| "_peak_process_memory": "PID\tMEM\tCOMMAND\n1219\t7.63GiB\t/app/product/go/infra/anyscaled/anyscaled_/anyscaled startv2 --control_plane_url=https://console.any\n3423\t0.89GiB\t/home/ray/anaconda3/lib/python3.9/site-packages/ray/core/src/ray/gcs/gcs_server --log_dir=/tmp/ray/s\n4478\t0.41GiB\tpython distributed/test_many_pgs.py\n2007\t0.34GiB\tvector --watch-config --log-format json --config-yaml /etc/vector/vector.yaml\n3539\t0.11GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.9/site-packages/ray/dashboard/dashboa\n3704\t0.09GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.9/site-packages/ray/dashboard/agen\n3706\t0.07GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.9/site-packages/ray/_private/runti\n2349\t0.07GiB\t/usr/bin/python3 /app/infra/dataplane/webterminal/webterminal_sidecar_image.binary.runfiles/product/\n4272\t0.07GiB\tray::JobSupervisor\n3731\t0.07GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/jupyter-lab --allow-root --ip=127.0.0.1 --no-", | ||
| "_peak_memory": 2.13, | ||
| "_peak_process_memory": "PID\tMEM\tCOMMAND\n1135\t7.03GiB\t/app/product/go/infra/anyscaled/anyscaled_/anyscaled startv2 --control_plane_url=https://console.any\n3382\t0.89GiB\t/home/ray/anaconda3/lib/python3.9/site-packages/ray/core/src/ray/gcs/gcs_server --log_dir=/tmp/ray/s\n4327\t0.41GiB\tpython distributed/test_many_pgs.py\n2463\t0.28GiB\tvector --watch-config --log-format json --config-yaml /etc/vector/vector.yaml\n3498\t0.11GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.9/site-packages/ray/dashboard/dashboa\n3663\t0.09GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.9/site-packages/ray/dashboard/agen\n2634\t0.08GiB\t/usr/bin/python3 /app/infra/dataplane/webterminal/webterminal_sidecar_image.binary.runfiles/product/\n3665\t0.07GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.9/site-packages/ray/_private/runti\n4107\t0.07GiB\tray::JobSupervisor\n3700\t0.07GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/jupyter-lab --allow-root --ip=127.0.0.1 --no-", | ||
| "num_pgs": 1000, | ||
| "perf_metrics": [ | ||
| { | ||
| "perf_metric_name": "pgs_per_second", | ||
| "perf_metric_type": "THROUGHPUT", | ||
| "perf_metric_value": 22.037557767422825 | ||
| "perf_metric_value": 22.13026931060565 | ||
| }, | ||
| { | ||
| "perf_metric_name": "dashboard_p50_latency_ms", | ||
| "perf_metric_type": "LATENCY", | ||
| "perf_metric_value": 3.347 | ||
| "perf_metric_value": 3.37 | ||
| }, | ||
| { | ||
| "perf_metric_name": "dashboard_p95_latency_ms", | ||
| "perf_metric_type": "LATENCY", | ||
| "perf_metric_value": 12.126 | ||
| "perf_metric_value": 7.238 | ||
| }, | ||
| { | ||
| "perf_metric_name": "dashboard_p99_latency_ms", | ||
| "perf_metric_type": "LATENCY", | ||
| "perf_metric_value": 376.468 | ||
| "perf_metric_value": 276.52 | ||
| } | ||
| ], | ||
| "pgs_per_second": 22.037557767422825, | ||
| "pgs_per_second": 22.13026931060565, | ||
| "success": "1", | ||
| "time": 45.377079010009766 | ||
| "time": 45.186978340148926 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| {"release_version": "2.35.0"} | ||
| {"release_version": "2.36.0"} |
Oops, something went wrong.