Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix describe worker state wrong #1680

Merged
merged 2 commits into from
Jun 10, 2022
Merged

fix describe worker state wrong #1680

merged 2 commits into from
Jun 10, 2022

Conversation

lxning
Copy link
Collaborator

@lxning lxning commented Jun 10, 2022 •
edited
Loading

Description

Please read our CONTRIBUTING.md prior to creating your first pull request.

Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #(issue)
#1679

Type of change

Please delete options that are not relevant.

  • [X ] Bug fix (non-breaking change which fixes an issue)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

  • Before
    Logs for Before: get "ready" even model is still in loading mode
    curl http://localhost:8081/models/deeplabv3_resnet101_scripted
    [
    {
    "modelName": "deeplabv3_resnet101_scripted",
    "modelVersion": "1.0",
    "modelUrl": "deeplabv3_resnet_101_scripted.mar",
    "runtime": "python",
    "minWorkers": 12,
    "maxWorkers": 12,
    "batchSize": 1,
    "maxBatchDelay": 100,
    "loadedAtStartup": true,
    "workers": [
    {
    "id": "9000",
    "startTime": "2022-06-09T19:44:57.371Z",
    "status": "READY",
    "memoryUsage": 0,
    "pid": -1,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9001",
    "startTime": "2022-06-09T19:44:57.372Z",
    "status": "READY",
    "memoryUsage": 0,
    "pid": -1,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9002",
    "startTime": "2022-06-09T19:44:57.372Z",
    "status": "READY",
    "memoryUsage": 0,
    "pid": -1,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9003",
    "startTime": "2022-06-09T19:44:57.373Z",
    "status": "READY",
    "memoryUsage": 0,
    "pid": -1,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9004",
    "startTime": "2022-06-09T19:44:57.373Z",
    "status": "READY",
    "memoryUsage": 0,
    "pid": -1,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9005",
    "startTime": "2022-06-09T19:44:57.373Z",
    "status": "READY",
    "memoryUsage": 0,
    "pid": -1,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9006",
    "startTime": "2022-06-09T19:44:57.373Z",
    "status": "READY",
    "memoryUsage": 0,
    "pid": -1,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9007",
    {
    "id": "9008",
    "startTime": "2022-06-09T19:44:57.374Z",
    "status": "READY",
    "memoryUsage": 0,
    "pid": -1,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9009",
    "startTime": "2022-06-09T19:44:57.375Z",
    "status": "READY",
    "memoryUsage": 0,
    "pid": -1,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9010",
    "startTime": "2022-06-09T19:44:57.375Z",
    "status": "READY",
    "memoryUsage": 0,
    "pid": -1,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9011",
    "startTime": "2022-06-09T19:44:57.375Z",
    "status": "READY",
    "memoryUsage": 0,
    "pid": -1,
    "gpu": false,
    "gpuUsage": "N/A"
    }
    ]
    }
    ]

  • fixing
    Logs for fixing: status changing from "unloading" to "ready"
    curl http://localhost:8081/models/deeplabv3_resnet101_scripted
    [
    {
    "modelName": "deeplabv3_resnet101_scripted",
    "modelVersion": "1.0",
    "modelUrl": "deeplabv3_resnet_101_scripted.mar",
    "runtime": "python",
    "minWorkers": 12,
    "maxWorkers": 12,
    "batchSize": 1,
    "maxBatchDelay": 100,
    "loadedAtStartup": true,
    "workers": [
    {
    "id": "9000",
    "startTime": "2022-06-09T19:52:52.438Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3885,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9001",
    "startTime": "2022-06-09T19:52:52.440Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3886,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9002",
    "startTime": "2022-06-09T19:52:52.440Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3882,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9003",
    "startTime": "2022-06-09T19:52:52.441Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3884,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9004",
    "startTime": "2022-06-09T19:52:52.441Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3881,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9005",
    "startTime": "2022-06-09T19:52:52.441Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3880,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9006",
    "startTime": "2022-06-09T19:52:52.441Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3879,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9007",
    "startTime": "2022-06-09T19:52:52.441Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3887,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9008",
    "startTime": "2022-06-09T19:52:52.442Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3888,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9009",
    "startTime": "2022-06-09T19:52:52.442Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3889,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9010",
    "startTime": "2022-06-09T19:52:52.442Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3890,
    "gpu": false,
    "gpuUsage": "N/A"
    },
    {
    "id": "9011",
    "startTime": "2022-06-09T19:52:52.443Z",
    "status": "UNLOADING",
    "memoryUsage": 0,
    "pid": 3883,
    "gpu": false,
    "gpuUsage": "N/A"
    }
    ]
    }
    ]

curl http://localhost:8081/models/deeplabv3_resnet101_scripted
[
{
"modelName": "deeplabv3_resnet101_scripted",
"modelVersion": "1.0",
"modelUrl": "deeplabv3_resnet_101_scripted.mar",
"runtime": "python",
"minWorkers": 12,
"maxWorkers": 12,
"batchSize": 1,
"maxBatchDelay": 100,
"loadedAtStartup": true,
"workers": [
{
"id": "9000",
"startTime": "2022-06-09T19:52:52.438Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3885,
"gpu": false,
"gpuUsage": "N/A"
},
{
"id": "9001",
"startTime": "2022-06-09T19:52:52.440Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3886,
"gpu": false,
"gpuUsage": "N/A"
},
{
"id": "9002",
"startTime": "2022-06-09T19:52:52.440Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3882,
"gpu": false,
"gpuUsage": "N/A"
},
{
"id": "9003",
"startTime": "2022-06-09T19:52:52.441Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3884,
"gpu": false,
"gpuUsage": "N/A"
},
{
"id": "9004",
"startTime": "2022-06-09T19:52:52.441Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3881,
"gpu": false,
"gpuUsage": "N/A"
},
{
"id": "9005",
"startTime": "2022-06-09T19:52:52.441Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3880,
"gpu": false,
"gpuUsage": "N/A"
},
{
"id": "9006",
"startTime": "2022-06-09T19:52:52.441Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3879,
"gpu": false,
"gpuUsage": "N/A"
},
{
"id": "9007",
"startTime": "2022-06-09T19:52:52.441Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3887,
"gpu": false,
"gpuUsage": "N/A"
},
{
"id": "9008",
"startTime": "2022-06-09T19:52:52.442Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3888,
"gpu": false,
"gpuUsage": "N/A"
},
{
"id": "9009",
"startTime": "2022-06-09T19:52:52.442Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3889,
"gpu": false,
"gpuUsage": "N/A"
},
{
"id": "9010",
"startTime": "2022-06-09T19:52:52.442Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3890,
"gpu": false,
"gpuUsage": "N/A"
},
{
"id": "9011",
"startTime": "2022-06-09T19:52:52.443Z",
"status": "READY",
"memoryUsage": 0,
"pid": 3883,
"gpu": false,
"gpuUsage": "N/A"
}
]
}
]

Checklist:

  • Did you have fun?
  • Have you added tests that prove your fix is effective or that this feature works?
  • Has code been commented, particularly in hard-to-understand areas?
  • Have you made corresponding changes to the documentation?

@lxning lxning requested review from maaquib and xyang16 June 10, 2022 02:56
@lxning lxning self-assigned this Jun 10, 2022
@lxning lxning added the bug Something isn't working label Jun 10, 2022
@lxning lxning added this to the v0.6.1 milestone Jun 10, 2022
@lxning lxning added the p0 high priority label Jun 10, 2022
@lxning lxning merged commit a4d5090 into master Jun 10, 2022
@lxning lxning deleted the issue_1679 branch June 10, 2022 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xyang16 xyang16 xyang16 approved these changes

@maaquib maaquib maaquib approved these changes

Assignees

@lxning lxning

Labels
bug Something isn't working p0 high priority
Projects
None yet
Milestone
v0.6.1
Development

Successfully merging this pull request may close these issues.
3 participants
@lxning @maaquib @xyang16