Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: reflect model config in config.properties #2382

Merged
merged 3 commits into from
Jun 12, 2023
Merged

fix: reflect model config in config.properties #2382

merged 3 commits into from
Jun 12, 2023

Conversation

lxning
Copy link
Collaborator

@lxning lxning commented Jun 1, 2023

Description

Please read our CONTRIBUTING.md prior to creating your first pull request.

Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #(issue)
#2365

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

  • Test A: without model config yaml file.
inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8082
metrics_address=http://0.0.0.0:8081
number_of_netty_threads=32
job_queue_size=1000
model_store=/home/model-server/model-store
workflow_store=/home/model-server/wf-store
enable_envvars_config=true
metrics_mode=prometheus
load_models=mnist_scripted_v2.mar
models={\
  "mnist_scripted_v2": {\
    "2.0": {\
        "defaultVersion": true,\
        "marName": "mnist_scripted_v2.mar",\
        "minWorkers": 1,\
        "maxWorkers": 3,\
        "batchSize": 5,\
        "maxBatchDelay": 200,\
        "responseTimeout": 120\
    }\
  }\
}

Result:
curl http://localhost:8082/models/mnist_scripted_v2
[
  {
    "modelName": "mnist_scripted_v2",
    "modelVersion": "2.0",
    "modelUrl": "mnist_scripted_v2.mar",
    "runtime": "python",
    "minWorkers": 1,
    "maxWorkers": 3,
    "batchSize": 5,
    "maxBatchDelay": 200,
    "loadedAtStartup": true,
    "workers": [
      {
        "id": "9000",
        "startTime": "2023-06-01T19:08:27.108Z",
        "status": "READY",
        "memoryUsage": 0,
        "pid": 18309,
        "gpu": true,
        "gpuUsage": "gpuId::1 utilization.gpu [%]::0 % utilization.memory [%]::0 % memory.used [MiB]::810 MiB"
      }
    ]
  }
]
  • Test B: With model config yaml
cat model-config.yaml
# TorchServe frontend parameters
minWorkers: 1
maxWorkers: 1
batchSize: 8
maxBatchDelay: 400

Result:
curl http://localhost:8082/models/mnist_scripted_v2
[
  {
    "modelName": "mnist_scripted_v2",
    "modelVersion": "2.0",
    "modelUrl": "mnist_scripted_v2.mar",
    "runtime": "python",
    "minWorkers": 1,
    "maxWorkers": 1,
    "batchSize": 8,
    "maxBatchDelay": 400,
    "loadedAtStartup": true,
    "workers": [
      {
        "id": "9000",
        "startTime": "2023-06-01T19:55:11.720Z",
        "status": "READY",
        "memoryUsage": 0,
        "pid": 21593,
        "gpu": true,
        "gpuUsage": "gpuId::1 utilization.gpu [%]::0 % utilization.memory [%]::0 % memory.used [MiB]::810 MiB"
      }
    ]
  }
]

Checklist:

  • Did you have fun?
  • Have you added tests that prove your fix is effective or that this feature works?
  • Has code been commented, particularly in hard-to-understand areas?
  • Have you made corresponding changes to the documentation?

@lxning lxning requested review from HamidShojanazeri and mreso and removed request for HamidShojanazeri June 1, 2023 19:57
@lxning lxning self-assigned this Jun 1, 2023
@lxning lxning added the bug Something isn't working label Jun 1, 2023
@lxning lxning added this to the v0.8.1 milestone Jun 1, 2023
@codecov
Copy link

codecov bot commented Jun 1, 2023

Codecov Report

Merging #2382 (3a0322b) into master (08a9093) will decrease coverage by 0.11%.
The diff coverage is n/a.

❗ Current head 3a0322b differs from pull request most recent head b0d0afc. Consider uploading reports for the commit b0d0afc to get more accurate results

@@            Coverage Diff             @@
##           master    #2382      +/-   ##
==========================================
- Coverage   72.01%   71.90%   -0.11%     
==========================================
  Files          78       78              
  Lines        3648     3648              
  Branches       58       58              
==========================================
- Hits         2627     2623       -4     
- Misses       1017     1021       +4     
  Partials        4        4              

see 2 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Collaborator

@mreso mreso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants