Add metric cache and migrate existing metrics to cache implementation

[lxning]

Description

Implementation of Metric Cache and migration of existing metrics to cache implementation
Design: #1492

Fixes #2140 #2141

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Feature/Issue validation/testing

• Unit tests
  Included in PR and run as part of sanity test in CI

• Manual end to end tests to verify metrics frontend implementation
  Verified both log and prometheus modes

• Regression tests

  • CPU: cpu_regression_log.txt
    • Relevant passing tests:
      • test_metrics.py::test_metrics_log_mode
      • test_metrics.py::test_metrics_prometheus_mode
    • Unrelated failing tests:
      • test_handler.py::test_echo_stream_inference
  • GPU: gpu_regression_log.txt
    • Relevant passing tests:
      • test_metrics.py::test_metrics_log_mode
      • test_metrics.py::test_metrics_prometheus_mode
    • Unrelated failing tests:
      • test_distributed_inference_handler.py::test_large_model_inference
      • test_handler.py::test_echo_stream_inference
      • test_sm_mme_requirements.py::test_oom_on_model_load
      • test_sm_mme_requirements.py::test_oom_on_invoke

• Benchmark

  • CPU: cpu_ab_report.csv
  • GPU: gpu_ab_report.csv

Checklist:

• Did you have fun?
• Have you added tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

[lxning]

why not directly parse the backend log to generate IMetric?

[codecov]

Codecov Report

Merging #2199 (57ee9d3) into master (92bd04b) will decrease coverage by 0.09%.
The diff coverage is n/a.

❗ Current head 57ee9d3 differs from pull request most recent head 63a61ef. Consider uploading reports for the commit 63a61ef to get more accurate results

@@            Coverage Diff             @@
##           master    #2199      +/-   ##
==========================================
- Coverage   69.82%   69.73%   -0.09%     
==========================================
  Files          77       77              
  Lines        3420     3420              
  Branches       57       57              
==========================================
- Hits         2388     2385       -3     
- Misses       1029     1032       +3     
  Partials        3        3              

see 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[lxning]

testModelMetrics should be updated and reflected in integration test ModelServerTest.java

[msaroufim]

Approving after chatting about this PR offline, I would still like to see some more docs around how a user goes about creating a custom python metric and how everything pipes back into a prometheus dashboard, critical to mention this in release notes as well and a tutorial as well although for the tutorial I'm fine if we do it in a followup PR