Auto detect backend metrics not defined in metrics configuration #2769

namannandan · 2023-11-06T19:29:33Z

Description

Enable auto detection of backend metrics that are not defined in the metrics configuration file.

To enable backwards compatibility with version <=0.6.0, set request ID for model load reqeusts so that metrics updated in the initialize method of the handler include both ModelName and Level dimensions.

Fixes: #2747, #2772

Type of change

New feature (non-breaking change which adds functionality)
This change requires a documentation update

Feature/Issue validation/testing

Unit and integration tests included in this PR
Manual testing with custom metrics example updated in this PR

$ cd examples/custom_metrics
$ cat metrics.yaml
dimensions:
  - &model_name "ModelName"
  - &worker_name "WorkerName"
  - &level "Level"
  - &device_id "DeviceId"
  - &hostname "Hostname"

ts_metrics:
  counter:
    - name: Requests2XX
      unit: Count
      dimensions: [*level, *hostname]
    - name: Requests4XX
      unit: Count
      dimensions: [*level, *hostname]
    - name: Requests5XX
      unit: Count
      dimensions: [*level, *hostname]
    - name: ts_inference_requests_total
      unit: Count
      dimensions: ["model_name", "model_version", "hostname"]
    - name: ts_inference_latency_microseconds
      unit: Microseconds
      dimensions: ["model_name", "model_version", "hostname"]
    - name: ts_queue_latency_microseconds
      unit: Microseconds
      dimensions: ["model_name", "model_version", "hostname"]
  gauge:
    - name: QueueTime
      unit: Milliseconds
      dimensions: [*level, *hostname]
    - name: WorkerThreadTime
      unit: Milliseconds
      dimensions: [*level, *hostname]
    - name: WorkerLoadTime
      unit: Milliseconds
      dimensions: [*worker_name, *level, *hostname]
    - name: CPUUtilization
      unit: Percent
      dimensions: [*level, *hostname]
    - name: MemoryUsed
      unit: Megabytes
      dimensions: [*level, *hostname]
    - name: MemoryAvailable
      unit: Megabytes
      dimensions: [*level, *hostname]
    - name: MemoryUtilization
      unit: Percent
      dimensions: [*level, *hostname]
    - name: DiskUsage
      unit: Gigabytes
      dimensions: [*level, *hostname]
    - name: DiskUtilization
      unit: Percent
      dimensions: [*level, *hostname]
    - name: DiskAvailable
      unit: Gigabytes
      dimensions: [*level, *hostname]
    - name: GPUMemoryUtilization
      unit: Percent
      dimensions: [*level, *device_id, *hostname]
    - name: GPUMemoryUsed
      unit: Megabytes
      dimensions: [*level, *device_id, *hostname]
    - name: GPUUtilization
      unit: Percent
      dimensions: [*level, *device_id, *hostname]

model_metrics:
  # Dimension "Hostname" is automatically added for model metrics in the backend
  # "HandlerMethodTime" and "ExamplePercentMetric" metrics are not defined here to show auto-detection of backend metrics
  counter:
    - name: InferenceRequestCount
      unit: count
      dimensions: []
    - name: InitializeCallCount
      unit: count
      dimensions: [*model_name, *level]
    - name: PreprocessCallCount
      unit: count
      dimensions: [*model_name]
    - name: PostprocessCallCount
      unit: CallCount
      dimensions: [*model_name, *level]
  gauge:
    - name: HandlerTime
      unit: ms
      dimensions: [*model_name, *level]
    - name: PredictionTime
      unit: ms
      dimensions: [*model_name, *level]
    - name: RequestBatchSize
      unit: count
      dimensions: ["ModelName"]
    - name: SizeOfImage
      unit: kB
      dimensions: [*model_name, *level]

Note that HandlerMethodTime and ExamplePercentMetric are not defined in the metrics configuration file above.

$ torchserve --ncs --start --model-store model_store --ts-config examples/custom_metrics/config.properties --models mnist=mnist.mar
$ curl http://127.0.0.1:8080/predictions/mnist -T examples/image_classifier/mnist/test_data/0.png
$ curl http://127.0.0.1:8082/metrics
.....
.....
# HELP HandlerMethodTime Torchserve prometheus gauge metric with unit: Milliseconds
# TYPE HandlerMethodTime gauge
HandlerMethodTime{MethodName="preprocess",ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",} 1.291036605834961
.....
.....
# HELP ExamplePercentMetric Torchserve prometheus histogram metric with unit: Percent
# TYPE ExamplePercentMetric histogram
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="0.005",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="0.01",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="0.025",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="0.05",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="0.075",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="0.1",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="0.25",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="0.5",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="0.75",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="1.0",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="2.5",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="5.0",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="7.5",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="10.0",} 0.0
ExamplePercentMetric_bucket{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",le="+Inf",} 3.0
ExamplePercentMetric_count{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",} 3.0
ExamplePercentMetric_sum{ModelName="mnist",Level="Model",Hostname="88665a372f4b.ant.amazon.com",} 150.0
.....
.....

As can be seen above, HandlerMethodTime and ExamplePercentMetric are auto detected, registered to the prometheus client and updated.

docs/metrics.md

frontend/server/src/main/java/org/pytorch/serve/metrics/MetricCache.java

lxning · 2023-11-09T18:56:48Z

frontend/server/src/main/java/org/pytorch/serve/metrics/MetricCache.java

+        for (Dimension dimension : parsedMetric.getDimensions()) {
+            dimensionNames.add(dimension.getName());
+        }
+


why need recopy to a new list? is the return value of getDimensions immutable?

parsedMetric is an instance of Metric object but while creating a new IMetric object using MetricBuilder.build, we need to provide a list of dimension names, hence the new list.

No, the return value of getDimensions is not immutable.

Then, why parsedMetric.getDimensions().add("Hostname") can not work?

parsedMetric is an instance of Metric. parsedMetric.getDimensions() returns a list of Dimension objects.

When registering a metric to cache, we need to provide a list of dimension names. So, we need to extract the dimension names from the Dimension objects.

When updating a metric, we need to provide a list of dimension values. So, we need to extract the dimension values from the Dimension objects.

Therefore the new lists are being created above.

Also, the Metric class API returns references to private objects which should ideally be immutable. So, I believe it is better to create a new list like we are doing here.

it takes time to deepcopy. Metric object will be GC eventually if it is just used once at here and not referenced any more. The shallow copy of dimension name is fine and faster if Metric is only used at here for building IMetrics and it will not be GC.

Added getDimensionNames and getDimensionValues utility functions to Metric class.

ts/metrics/caching_metric.py

…information

…tection

docs/metrics.md

lxning · 2023-11-18T04:29:12Z

frontend/server/src/main/java/org/pytorch/serve/metrics/MetricCache.java

+        for (Dimension dimension : parsedMetric.getDimensions()) {
+            dimensionNames.add(dimension.getName());
+        }
+


Then, why parsedMetric.getDimensions().add("Hostname") can not work?

lxning · 2023-11-18T04:43:38Z

frontend/server/src/main/java/org/pytorch/serve/wlm/WorkerLifeCycle.java

+                            List<String> dimensionValues = new ArrayList<String>();
+                            for (Dimension dimension : parsedMetric.getDimensions()) {
+                                dimensionValues.add(dimension.getValue());
+                            }
+                            // Hostname is added as a dimension by default to backend metrics
+                            dimensionValues.add(parsedMetric.getHostName());
+
+                            this.metricCache


Added details in this comment: https://github.com/pytorch/serve/pull/2769/files#r1399626654

…ject

lxning · 2023-11-21T18:40:54Z

ts/model_service_worker.py

+            # Backwards Compatibility with releases <=0.6.0
+            # Request ID is not set for model load requests
+            # TODO: UUID serves as a temporary request ID for model load requests
+            self.metrics_cache.set_request_ids(str(uuid.uuid4()))
+


Here is just initialize metrics_cache in service, not real model load metrics. uuid should be set within func load. So far there is no model loading metrics as i know.

Moved logic for request id assignment for model load requests from model_service_worker.py to model_loader.py.

That's correct, we don't have any default metrics that we emit during model load time, but a custom handler can emit metrics in the initialize method. If the request id is not set, it can cause the metric to only have Level:Error as the default dimension that gets added instead of ModelName:<model_name> and Level:Model. Described the issue here: #2772

Setting uuid for model loading should be at func load, not here.

Agreed. I've already moved it to the load method: https://github.com/pytorch/serve/pull/2769/files#diff-bd9cbbce71a9f862d4edafb574d20bd4dcdf5ae8f80bf2aa8ad3b73e80fb5807R93-R99

lxning · 2023-11-21T18:54:14Z

frontend/server/src/main/java/org/pytorch/serve/metrics/Metric.java

        this.hostName = hostName;
-        this.dimensions = Arrays.asList(dimensions);
+        this.setDimensions(Arrays.asList(dimensions));


Why original "this.dimensions = Arrays.asList(dimensions);" needs to be replaced with setDimensions?

This is because setDimensions now takes care of setting up the values of dimensionNames and dimensionsValues as well to support the getDimensionNames and getDimensionValues methods.

got it. Thanks

…_worker.py to model_loader.py

namannandan added 4 commits November 6, 2023 11:24

Auto detect backend metrics not defined in metrics configuration

237b26b

Merge branch 'master' into naman-dynamic-metrics

d6e613f

Add tests for metric auto detection

ce7a13b

Update custom metrics example to show metrics auto detection

b6b085f

namannandan marked this pull request as ready for review November 7, 2023 21:53

namannandan requested review from lxning, mreso, msaroufim and agunapal November 7, 2023 21:53

namannandan added 3 commits November 7, 2023 16:11

Update metrics documentation

69574b7

Fix unit test failure

7565730

Merge branch 'master' into naman-dynamic-metrics

1c32f1c

lxning reviewed Nov 9, 2023

View reviewed changes

namannandan added 6 commits November 9, 2023 16:40

Set request ID for load model requests

1c9f42a

Update metrics integration test to use custom metrics example

66c6ca7

Make auto updates to frontend metrics cache thread safe

0147477

Add auto detect metrics is always called on metric log that has type …

93c5f79

…information

Merge branch 'master' into naman-dynamic-metrics

77b0a2b

Fix auto detect backend metrics unit test

d65e312

namannandan mentioned this pull request Nov 13, 2023

Set request ID for load model requests #2788

Closed

1 task

namannandan added 6 commits November 15, 2023 16:36

Update documentation about performance impact of metrics auto detection

e90e22c

Merge branch 'master' into naman-dynamic-metrics

17d842c

fix linter error

06778f3

Disable metrics auto detection by default

8dc2a41

fix integration tests

76190e2

Update documentation and custom metrics example about metrics auto de…

2c40c9f

…tection

namannandan requested a review from lxning November 18, 2023 01:20

lxning reviewed Nov 18, 2023

View reviewed changes

Update metrics auto detection documentation

e18dc55

namannandan force-pushed the naman-dynamic-metrics branch from 04724d2 to e18dc55 Compare November 20, 2023 18:46

Merge branch 'master' into naman-dynamic-metrics

de67f45

namannandan requested a review from lxning November 20, 2023 19:07

namannandan added 3 commits November 20, 2023 16:48

Add helper functions to get dimension names and values from Metric ob…

329a0f2

…ject

Merge branch 'master' into naman-dynamic-metrics

8b91887

Fix java formatting

abe5b07

lxning reviewed Nov 21, 2023

View reviewed changes

Move request id assignment for model load requests from model_service…

a4d0b87

…_worker.py to model_loader.py

namannandan requested a review from lxning November 21, 2023 21:48

lxning approved these changes Nov 21, 2023

View reviewed changes

namannandan added this pull request to the merge queue Nov 21, 2023

Merged via the queue into pytorch:master with commit 98c19bc Nov 21, 2023
13 checks passed

chauhang added this to the v0.10.0 milestone Feb 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto detect backend metrics not defined in metrics configuration #2769

Auto detect backend metrics not defined in metrics configuration #2769

namannandan commented Nov 6, 2023 •

edited

Loading

lxning Nov 9, 2023

namannandan Nov 11, 2023 •

edited

Loading

lxning Nov 18, 2023

namannandan Nov 20, 2023

lxning Nov 20, 2023

namannandan Nov 21, 2023

lxning Nov 18, 2023

lxning Nov 18, 2023

namannandan Nov 20, 2023

lxning Nov 21, 2023

namannandan Nov 21, 2023

lxning Nov 21, 2023

namannandan Nov 21, 2023

lxning Nov 21, 2023

namannandan Nov 21, 2023

lxning Nov 21, 2023

Auto detect backend metrics not defined in metrics configuration #2769

Auto detect backend metrics not defined in metrics configuration #2769

Conversation

namannandan commented Nov 6, 2023 • edited Loading

Description

Type of change

Feature/Issue validation/testing

Choose a reason for hiding this comment

namannandan Nov 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

namannandan commented Nov 6, 2023 •

edited

Loading

namannandan Nov 11, 2023 •

edited

Loading