Add support for metrics legacy mode

[namannandan]

Description

Add support for metrics legacy mode, where:

• ts_inference_requests_total, ts_inference_latency_microseconds and ts_queue_latency_microseconds are only available via the metrics API endpoint in prometheus format.
• All other metrics inclusive of auto detected backend metrics are only logged to ts_metrics.log and model_metrics.log

Fixes #2795

Type of change

• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Feature/Issue validation/testing

• Unit and integration tests included in this PR
• Manual testing using the custom metrics example

$ cat examples/custom_metrics/config.properties 
metrics_mode=legacy
metrics_config=examples/custom_metrics/metrics.yaml
model_metrics_auto_detect=true
models={\
  "mnist": {\
    "1.0": {\
        "defaultVersion": true,\
        "marName": "mnist.mar",\
        "minWorkers": 1,\
        "maxWorkers": 1\
    }\
  }\
}

$ curl http://127.0.0.1:8080/predictions/mnist -T examples/image_classifier/mnist/test_data/0.png
0                                                                                                                                                                                                                    

$ curl http://127.0.0.1:8082/metrics
# HELP ts_inference_latency_microseconds Torchserve prometheus counter metric with unit: Microseconds
# TYPE ts_inference_latency_microseconds counter
ts_inference_latency_microseconds{model_name="mnist",model_version="default",hostname="88665a372f4b.ant.amazon.com",} 76988.864
# HELP ts_inference_requests_total Torchserve prometheus counter metric with unit: Count
# TYPE ts_inference_requests_total counter
ts_inference_requests_total{model_name="mnist",model_version="default",hostname="88665a372f4b.ant.amazon.com",} 1.0
# HELP ts_queue_latency_microseconds Torchserve prometheus counter metric with unit: Microseconds
# TYPE ts_queue_latency_microseconds counter
ts_queue_latency_microseconds{model_name="mnist",model_version="default",hostname="88665a372f4b.ant.amazon.com",} 144.148

$ cat logs/model_metrics.log
2023-12-01T08:37:25,446 - InitializeCallCount.count:1.0|#ModelName:mnist,Level:Model|#hostname:88665a372f4b.ant.amazon.com,requestID:b430fc75-6b7c-4c5b-bb20-c4ddcec8af1c,timestamp:1701400045
2023-12-01T08:38:12,530 - InferenceRequestCount.count:1.0|#|#hostname:88665a372f4b.ant.amazon.com,requestID:null,timestamp:1701400092
2023-12-01T08:38:12,530 - PreprocessCallCount.count:1.0|#ModelName:mnist|#hostname:88665a372f4b.ant.amazon.com,requestID:null,timestamp:1701400092
2023-12-01T08:38:12,530 - RequestBatchSize.count:1.0|#ModelName:mnist|#hostname:88665a372f4b.ant.amazon.com,requestID:null,timestamp:1701400092
2023-12-01T08:38:12,531 - SizeOfImage.kB:0.265625|#ModelName:mnist,Level:Model|#hostname:88665a372f4b.ant.amazon.com,requestID:f75a9e93-622a-416a-b727-622f6ecb34f6,timestamp:1701400092
2023-12-01T08:38:12,551 - HandlerMethodTime.Milliseconds:20.53213119506836|#MethodName:preprocess,ModelName:mnist,Level:Model|#hostname:88665a372f4b.ant.amazon.com,requestID:f75a9e93-622a-416a-b727-622f6ecb34f6,timestamp:1701400092
2023-12-01T08:38:12,602 - PostprocessCallCount.CallCount:1.0|#ModelName:mnist,Level:Model|#hostname:88665a372f4b.ant.amazon.com,requestID:f75a9e93-622a-416a-b727-622f6ecb34f6,timestamp:1701400092
2023-12-01T08:38:12,603 - ExamplePercentMetric.Percent:50.0|#ModelName:mnist,Level:Model|#hostname:88665a372f4b.ant.amazon.com,requestID:f75a9e93-622a-416a-b727-622f6ecb34f6,timestamp:1701400092
2023-12-01T08:38:12,604 - HandlerTime.ms:74.73|#ModelName:mnist,Level:Model|#hostname:88665a372f4b.ant.amazon.com,requestID:f75a9e93-622a-416a-b727-622f6ecb34f6,timestamp:1701400092
2023-12-01T08:38:12,605 - PredictionTime.ms:74.87|#ModelName:mnist,Level:Model|#hostname:88665a372f4b.ant.amazon.com,requestID:f75a9e93-622a-416a-b727-622f6ecb34f6,timestamp:1701400092

[agunapal]

Will try it out with the nightly