Processing /Volumes/workplace/python_env/serve Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting Pillow Using cached Pillow-9.4.0-2-cp38-cp38-macosx_10_10_x86_64.whl (3.3 MB) Collecting psutil Using cached psutil-5.9.4-cp36-abi3-macosx_10_9_x86_64.whl (243 kB) Collecting packaging Using cached packaging-23.0-py3-none-any.whl (42 kB) Collecting wheel Using cached wheel-0.40.0-py3-none-any.whl (64 kB) Building wheels for collected packages: torchserve Building wheel for torchserve (setup.py): started Building wheel for torchserve (setup.py): finished with status 'done' Created wheel for torchserve: filename=torchserve-0.7.0b20230322-py3-none-any.whl size=67550780 sha256=6ad31a2e413e775293dc7a095bef6e000b824ab45eae9ffc3e0b03c6810d7d5a Stored in directory: /private/var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/pip-ephem-wheel-cache-bzf1cao3/wheels/24/42/60/ce01bb072bdf069a0689515be64114c26b8dc772df749bad0d Successfully built torchserve Installing collected packages: wheel, psutil, Pillow, packaging, torchserve Attempting uninstall: wheel Found existing installation: wheel 0.40.0 Uninstalling wheel-0.40.0: Successfully uninstalled wheel-0.40.0 Attempting uninstall: psutil Found existing installation: psutil 5.9.4 Uninstalling psutil-5.9.4: Successfully uninstalled psutil-5.9.4 Attempting uninstall: Pillow Found existing installation: Pillow 9.4.0 Uninstalling Pillow-9.4.0: Successfully uninstalled Pillow-9.4.0 Attempting uninstall: packaging Found existing installation: packaging 23.0 Uninstalling packaging-23.0: Successfully uninstalled packaging-23.0 Attempting uninstall: torchserve Found existing installation: torchserve 0.7.0b20230322 Uninstalling torchserve-0.7.0b20230322: Successfully uninstalled torchserve-0.7.0b20230322 Successfully installed Pillow-9.4.0 packaging-23.0 psutil-5.9.4 torchserve-0.7.0b20230322 wheel-0.40.0 Processing ./model-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting enum-compat Using cached enum_compat-0.0.3-py3-none-any.whl (1.3 kB) Building wheels for collected packages: torch-model-archiver Building wheel for torch-model-archiver (setup.py): started Building wheel for torch-model-archiver (setup.py): finished with status 'done' Created wheel for torch-model-archiver: filename=torch_model_archiver-0.7.0b20230322-py3-none-any.whl size=14695 sha256=8e8c079f8b1dce0b7e58f7c39993ab9b019a8f89cdc33a6e4d0c0e77b66078a3 Stored in directory: /Users/lninga/Library/Caches/pip/wheels/40/d0/f5/5019474ce9b424781ec8c156be2dabd8d7526bfba45adc4d1e Successfully built torch-model-archiver Installing collected packages: enum-compat, torch-model-archiver Attempting uninstall: enum-compat Found existing installation: enum-compat 0.0.3 Uninstalling enum-compat-0.0.3: Successfully uninstalled enum-compat-0.0.3 Attempting uninstall: torch-model-archiver Found existing installation: torch-model-archiver 0.7.0b20230322 Uninstalling torch-model-archiver-0.7.0b20230322: Successfully uninstalled torch-model-archiver-0.7.0b20230322 Successfully installed enum-compat-0.0.3 torch-model-archiver-0.7.0b20230322 Processing ./workflow-archiver Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Building wheels for collected packages: torch-workflow-archiver Building wheel for torch-workflow-archiver (setup.py): started Building wheel for torch-workflow-archiver (setup.py): finished with status 'done' Created wheel for torch-workflow-archiver: filename=torch_workflow_archiver-0.2.6b20230322-py3-none-any.whl size=12701 sha256=0ef4081e728aae14e6554eb5ea308de816fd0f5f59050909b238296d633edc7a Stored in directory: /Users/lninga/Library/Caches/pip/wheels/6a/08/97/e89e9bf08db09591cf7092e9dbfc31fa105e0423bc3c8da6c1 Successfully built torch-workflow-archiver Installing collected packages: torch-workflow-archiver Attempting uninstall: torch-workflow-archiver Found existing installation: torch-workflow-archiver 0.2.6b20230322 Uninstalling torch-workflow-archiver-0.2.6b20230322: Successfully uninstalled torch-workflow-archiver-0.2.6b20230322 Successfully installed torch-workflow-archiver-0.2.6b20230322 WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar ... WARNING - Overwriting /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar ... newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 267ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 34ms 5ms 1ms 291µs 251ms 7ms 401µs 301ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 119ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 625µs (cache) (cache) 116ms 1ms 54µs 120ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 2.2s time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 267µs (cache) (cache) 2.2s 1ms 51µs 2.2s ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 5ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 240µs (cache) (cache) 2ms 1ms 54µs 5ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 18ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 726µs 269µs 320µs 14ms 1ms 63µs 19ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 4ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 292µs (cache) (cache) 2ms 1ms 43µs 5ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 18ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 598µs 171µs 241µs 15ms 1ms 43µs 19ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 104ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 256µs (cache) (cache) 102ms 1ms 33µs 105ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 6ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 182µs (cache) (cache) 3ms 1ms 31µs 6ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 101ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 191µs (cache) (cache) 98ms 1ms 27µs 101ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 1350ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 215µs (cache) (cache) 1347ms 1ms 38µs 1350ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 19ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 592µs 174µs 218µs 16ms 1ms 35µs 19ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 6ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 226µs (cache) (cache) 3ms 1ms 37µs 6ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 4ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 302µs (cache) (cache) 1ms 1ms 44µs 4ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 246B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 235µs (cache) (cache) 1ms 1ms 35µs 4ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 7ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 215µs (cache) (cache) 4ms 1ms 40µs 7ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 4ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 212µs (cache) (cache) 1ms 1ms 33µs 4ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 15ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 687µs 169µs 279µs 12ms 1ms 66µs 16ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 112ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 229µs (cache) (cache) 110ms 1ms 24µs 112ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 4ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 215µs (cache) (cache) 1ms 1ms 29µs 4ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 4ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 180µs (cache) (cache) 1ms 1ms 30µs 4ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 4ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 294µs (cache) (cache) 1ms 1ms 37µs 5ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 3ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 195µs (cache) (cache) 1ms 1ms 26µs 3ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 13ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 510µs 137µs 210µs 10ms 1ms 26µs 13ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 4ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 212µs (cache) (cache) 1ms 1ms 59µs 4ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 3ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 983µs 195µs (cache) (cache) 1ms 1ms 31µs 3ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 21ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 993µs 532µs 141µs 207µs 18ms 1ms 27µs 21ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 29ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 990µs 583µs 151µs 251µs 26ms 1ms 28µs 29ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 107ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 254µs (cache) (cache) 105ms 970µs 22µs 107ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 175µs (cache) (cache) 4ms 1ms 35µs 6ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1371ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 217µs (cache) (cache) 1369ms 1ms 33µs 1372ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 151ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 612µs 164µs 256µs 148ms 1ms 29µs 152ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 911µs 174µs (cache) (cache) 3ms 1ms 29µs 5ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 4.6s time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 237µs (cache) (cache) 4.6s 2ms 44µs 4.6s ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 7ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 401µs (cache) (cache) 4ms 1ms 35µs 8ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 10.3s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 225µs (cache) (cache) 10.3s 1ms 32µs 10.3s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 21ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 219µs (cache) (cache) 18ms 1ms 42µs 22ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 7.3s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 327µs (cache) (cache) 7.3s 1ms 31µs 7.3s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 20ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 188µs (cache) (cache) 17ms 2ms 47µs 21ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 112ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 197µs (cache) (cache) 109ms 1ms 22µs 112ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 197µs (cache) (cache) 3ms 1ms 29µs 6ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 847ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 168µs (cache) (cache) 845ms 927µs 23µs 847ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 5ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 174µs (cache) (cache) 3ms 1ms 32µs 5ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 860ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 164µs (cache) (cache) 858ms 1ms 28µs 860ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 5ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 147µs (cache) (cache) 2ms 1ms 27µs 5ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 843ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 157µs (cache) (cache) 841ms 934µs 28µs 844ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 3ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 933µs 146µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 3ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 957µs 409µs (cache) (cache) 1ms 964µs 28µs 3ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 3ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 171µs (cache) (cache) 1ms 1ms 31µs 3ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 3ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 175µs (cache) (cache) 1ms 1ms 31µs 4ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 4ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 228µs (cache) (cache) 1ms 1ms 33µs 4ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 164µs (cache) (cache) 1ms 1ms 29µs 3ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 3ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 168µs (cache) (cache) 1ms 1ms 31µs 4ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 3ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 985µs 149µs (cache) (cache) 1ms 1ms 28µs 3ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 4ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 959µs 151µs (cache) (cache) 2ms 1ms 27µs 4ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 3ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 969µs 159µs (cache) (cache) 1ms 1ms 28µs 3ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 3ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 194µs (cache) (cache) 1ms 975µs 28µs 3ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 879ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 661µs 205µs 237µs 876ms 959µs 32µs 879ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 14ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 874µs 431µs 120µs 189µs 11ms 1ms 30µs 14ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 12ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 850µs 363µs 124µs 194µs 9ms 1ms 28µs 12ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 12ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 956µs 973µs 132µs 174µs 8ms 947µs 24µs 12ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 3ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 157µs (cache) (cache) 1ms 904µs 24µs 3ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 3ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 153µs (cache) (cache) 1ms 955µs 26µs 3ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 4ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 212µs (cache) (cache) 1ms 999µs 26µs 4ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 3ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 909µs 159µs (cache) (cache) 1ms 1ms 42µs 3ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 14ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 514µs 133µs 199µs 11ms 1ms 27µs 15ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 12ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 904µs 430µs 123µs 186µs 9ms 941µs 27µs 12ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 12ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 976µs 151µs (cache) (cache) 10ms 903µs 24µs 12ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 4ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 957µs 164µs (cache) (cache) 2ms 1ms 48µs 4ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 15ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 485µs 146µs 205µs 11ms 1ms 32µs 15ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 17ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 477µs 151µs 209µs 12ms 3ms 35µs 17ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 8ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 508µs 149µs 214µs 5ms 1ms 34µs 8ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 3ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 186µs (cache) (cache) 1ms 1ms 35µs 4ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 111ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 566µs 190µs 254µs 108ms 903µs 23µs 112ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 5ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 166µs (cache) (cache) 3ms 1ms 34µs 6ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 100ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 162µs (cache) (cache) 98ms 801µs 21µs 100ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 5ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 158µs (cache) (cache) 3ms 972µs 28µs 5ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 104ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 936µs 150µs (cache) (cache) 102ms 862µs 27µs 104ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 6ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 185µs (cache) (cache) 3ms 1ms 35µs 6ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 109ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 185µs (cache) (cache) 107ms 827µs 25µs 110ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 5ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 159µs (cache) (cache) 3ms 959µs 27µs 5ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 3ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 952µs 156µs (cache) (cache) 1ms 1ms 29µs 3ms ✓ Successful request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 34.2s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 7.75kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 398ms [min: 3ms, max: 10.3s, s.d.: 1489ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 742µs [min: 120µs, max: 1ms, s.d.: 637µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 396ms [min: 1ms, max: 10.3s, s.d.: 1489ms] │ └─────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman inference Iteration 1/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_v4.mar&model_name=my_text_classifier&initial_workers=1&synchronous=true 200 OK ★ 38.2s time ★ 388B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"my_text_classifier\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 33ms 5ms 1ms 284µs 38.2s 7ms 453µs 38.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier 200 OK ★ 66ms time ★ 353B↑ 372B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 137B │ { │ "World": 0.02911965176463127, │ "Sports": 2.943157232948579e-05, │ "Business": 0.9074352383613586, │ "Sci/Tec": 0.06341569125652313 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 7ms 1ms 635µs 231µs 61ms 2ms 55µs 73ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier 200 OK ★ 21ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"my_text_classifier\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 560µs (cache) (cache) 16ms 2ms 102µs 22ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 19ms time ★ 233B↑ 1.09kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 794B │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ ",model_version="default",} 39558.154 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="my_text_classifier",mo │ del_version="default",} 131.854 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 658µs 179µs 261µs 15ms 1ms 78µs 20ms ✓ Successful GET request Iteration 2/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/my_text_classifier_scripted_v3.mar&model_name=my_text_classifier_scripted&initial_workers=1&synchronous=true 200 OK ★ 36.7s time ★ 406B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 103B │ { │ "status": "Model \"my_text_classifier_scripted\" Ver │ sion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 265µs (cache) (cache) 36.7s 2ms 54µs 36.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/my_text_classifier_scripted 200 OK ★ 71ms time ★ 362B↑ 372B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 137B │ { │ "World": 0.04559723287820816, │ "Sports": 0.0003771767660509795, │ "Business": 0.08623101562261581, │ "Sci/Tec": 0.8677946329116821 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 330µs (cache) (cache) 68ms 1ms 42µs 73ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/my_text_classifier_scripted 200 OK ★ 26ms time ★ 263B↑ 335B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 69B │ { │ "status": "Model \"my_text_classifier_scripted\" unr │ egistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 273µs (cache) (cache) 21ms 4ms 49µs 27ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 1.53kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1.24kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ _scripted",model_version="default",} 66832.388 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ ",model_version="default",} 39558.154 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="my_text_classifier_scr │ ipted",model_version="default",} 85.622 │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="my_text_classifier",mo │ del_version="default",} 131.854 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 256µs (cache) (cache) 2ms 1ms 60µs 5ms ✓ Successful GET request Iteration 3/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 2s time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 242µs (cache) (cache) 2s 1ms 29µs 2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1 200 OK ★ 133ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299158692359924, │ "tiger_cat": 0.24496474862098694, │ "Egyptian_cat": 0.21722468733787537, │ "cougar": 0.002217544475570321 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 314µs (cache) (cache) 130ms 1ms 28µs 134ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 17ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 196µs (cache) (cache) 14ms 1ms 43µs 17ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 1.94kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1.65kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1",mod │ el_version="default",} 127025.194 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ _scripted",model_version="default",} 66832.388 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ ",model_version="default",} 39558.154 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="squeezenet1_1",model_v │ ersion="default",} 86.835 │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="my_text_classifier_scr │ ipted",model_version="default",} 85.622 │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="my_text_classifier",mo │ del_version="default",} 131.854 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 217µs (cache) (cache) 1ms 1ms 36µs 4ms ✓ Successful GET request Iteration 4/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/squeezenet1_1_scripted.mar&model_name=squeezenet1_1_scripted&initial_workers=1&synchronous=true 200 OK ★ 3.9s time ★ 393B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 98B │ { │ "status": "Model \"squeezenet1_1_scripted\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 195µs (cache) (cache) 3.9s 1ms 27µs 3.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/squeezenet1_1_scripted 200 OK ★ 113ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299158692359924, │ "tiger_cat": 0.24496474862098694, │ "Egyptian_cat": 0.21722468733787537, │ "cougar": 0.002217544475570321 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 200µs (cache) (cache) 110ms 1ms 24µs 114ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/squeezenet1_1_scripted 200 OK ★ 17ms time ★ 258B↑ 330B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 64B │ { │ "status": "Model \"squeezenet1_1_scripted\" unregist │ ered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 227µs (cache) (cache) 14ms 1ms 36µs 17ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 2.37kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.08kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1",mod │ el_version="default",} 127025.194 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1_scri │ pted",model_version="default",} 107293.17 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ _scripted",model_version="default",} 66832.388 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ ",model_version="default",} 39558.154 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="squeezenet1_1",model_v │ ersion="default",} 86.835 │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="squeezenet1_1_scripted │ ",model_version="default",} 89.304 │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="my_text_classifier_scr │ ipted",model_version="default",} 85.622 │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="my_text_classifier",mo │ del_ │ (showing 2.05kB/2.08kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 276µs (cache) (cache) 2ms 1ms 57µs 5ms ✓ Successful GET request Iteration 5/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161&initial_workers=1&synchronous=true 200 OK ★ 4.7s time ★ 330B↑ 353B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 87B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 232µs (cache) (cache) 4.7s 1ms 31µs 4.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161 200 OK ★ 449ms time ★ 111.24kB↑ 418B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 183B │ { │ "tabby": 0.46661895513534546, │ "tiger_cat": 0.46449047327041626, │ "Egyptian_cat": 0.06614057719707489, │ "lynx": 0.0012924439506605268, │ "plastic_bag": 0.00022909804829396307 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 211µs (cache) (cache) 447ms 1ms 33µs 450ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161 200 OK ★ 19ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 237µs (cache) (cache) 16ms 1ms 35µs 19ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 2.77kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1",mod │ el_version="default",} 127025.194 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="densenet161",model │ _version="default",} 443633.096 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1_scri │ pted",model_version="default",} 107293.17 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ _scripted",model_version="default",} 66832.388 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ ",model_version="default",} 39558.154 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="squeezenet1_1",model_v │ ersion="default",} 86.835 │ ts_queue_latency_microseconds{uuid="8e24d624-acb8-4057 │ -bad5-f3d13de70db4",model_name="densenet161",model_ver │ sion="default",} 86.44 │ ts_qu │ (showing 2.05kB/2.47kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 213µs (cache) (cache) 2ms 1ms 38µs 5ms ✓ Successful GET request Iteration 6/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=alexnet.mar&model_name=alexnet&initial_workers=1&synchronous=true 200 OK ★ 7.1s time ★ 322B↑ 349B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 83B │ { │ "status": "Model \"alexnet\" Version: 1.0 registered │ with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 197µs (cache) (cache) 7.1s 1ms 35µs 7.1s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet 200 OK ★ 104ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.31847333908081055, │ "tiger_cat": 0.25793954730033875, │ "Egyptian_cat": 0.24254852533340454, │ "lynx": 0.16879378259181976, │ "tiger": 0.006487946026027203 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 216µs (cache) (cache) 102ms 1ms 35µs 107ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet 200 OK ★ 19ms time ★ 243B↑ 315B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 49B │ { │ "status": "Model \"alexnet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 226µs (cache) (cache) 16ms 1ms 34µs 19ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 3.15kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.86kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1",mod │ el_version="default",} 127025.194 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="alexnet",model_ver │ sion="default",} 98667.946 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="densenet161",model │ _version="default",} 443633.096 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1_scri │ pted",model_version="default",} 107293.17 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ _scripted",model_version="default",} 66832.388 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ ",model_version="default",} 39558.154 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_lat │ (showing 2.05kB/2.86kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 214µs (cache) (cache) 1ms 1ms 32µs 6ms ✓ Successful GET request Iteration 7/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/alexnet_scripted.mar&model_name=alexnet_scripted&initial_workers=1&synchronous=true 200 OK ★ 49.2s time ★ 381B↑ 358B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 92B │ { │ "status": "Model \"alexnet_scripted\" Version: 1.0 r │ egistered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 213µs (cache) (cache) 49.2s 1ms 29µs 49.2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/alexnet_scripted 200 OK ★ 143ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tabby": 0.31847333908081055, │ "tiger_cat": 0.25793954730033875, │ "Egyptian_cat": 0.24254852533340454, │ "lynx": 0.16879378259181976, │ "tiger": 0.006487946026027203 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 241µs (cache) (cache) 141ms 1ms 30µs 145ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/alexnet_scripted 200 OK ★ 19ms time ★ 252B↑ 324B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58B │ { │ "status": "Model \"alexnet_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 209µs (cache) (cache) 17ms 1ms 34µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 3.57kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.27kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1",mod │ el_version="default",} 127025.194 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="alexnet",model_ver │ sion="default",} 98667.946 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="densenet161",model │ _version="default",} 443633.096 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1_scri │ pted",model_version="default",} 107293.17 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ _scripted",model_version="default",} 66832.388 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="my_text_classifier │ ",model_version="default",} 39558.154 │ ts_i │ (showing 2.05kB/3.27kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 310µs (cache) (cache) 1ms 2ms 41µs 5ms ✓ Successful GET request Iteration 8/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&initial_workers=1&synchronous=true 200 OK ★ 2.8s time ★ 326B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 199µs (cache) (cache) 2.8s 1ms 32µs 2.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18 200 OK ★ 106ms time ★ 111.24kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tabby": 0.40966302156448364, │ "tiger_cat": 0.3467046916484833, │ "Egyptian_cat": 0.13002900779247284, │ "lynx": 0.023919543251395226, │ "bucket": 0.011532166972756386 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 211µs (cache) (cache) 103ms 1ms 26µs 107ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18 200 OK ★ 17ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 173µs (cache) (cache) 15ms 1ms 30µs 17ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 3.96kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 3.67kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1",mod │ el_version="default",} 127025.194 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="alexnet",model_ver │ sion="default",} 98667.946 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="densenet161",model │ _version="default",} 443633.096 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="resnet-18",model_v │ ersion="default",} 100536.542 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1_scri │ pted",model_version="default",} 107293.17 │ ts_inference_latency_microseconds{uuid="8e │ (showing 2.05kB/3.67kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 188µs (cache) (cache) 1ms 1ms 29µs 3ms ✓ Successful GET request Iteration 9/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-18_scripted.mar&model_name=resnet-18_scripted&initial_workers=1&synchronous=true 200 OK ★ 16.3s time ★ 385B↑ 360B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "status": "Model \"resnet-18_scripted\" Version: 1.0 │ registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 194µs (cache) (cache) 16.3s 1ms 25µs 16.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-18_scripted 200 OK ★ 189ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tabby": 0.40966302156448364, │ "tiger_cat": 0.3467046916484833, │ "Egyptian_cat": 0.13002900779247284, │ "lynx": 0.023919543251395226, │ "bucket": 0.011532166972756386 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 201µs (cache) (cache) 187ms 1ms 29µs 190ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-18_scripted 200 OK ★ 54ms time ★ 254B↑ 326B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 60B │ { │ "status": "Model \"resnet-18_scripted\" unregistered │ " │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 314µs (cache) (cache) 51ms 1ms 38µs 54ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 7ms time ★ 233B↑ 4.38kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.09kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1",mod │ el_version="default",} 127025.194 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="alexnet",model_ver │ sion="default",} 98667.946 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="densenet161",model │ _version="default",} 443633.096 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="resnet-18",model_v │ ersion="default",} 100536.542 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057 │ (showing 2.05kB/4.09kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 268µs (cache) (cache) 4ms 1ms 45µs 9ms ✓ Successful GET request Iteration 10/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=vgg16.mar&model_name=vgg16&initial_workers=1&synchronous=true 200 OK ★ 15.6s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"vgg16\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 223µs (cache) (cache) 15.6s 1ms 32µs 15.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16 200 OK ★ 417ms time ★ 111.24kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tiger_cat": 0.44697263836860657, │ "tabby": 0.4408797323703766, │ "Egyptian_cat": 0.05904562026262283, │ "tiger": 0.020596399903297424, │ "lynx": 0.009934580884873867 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 309µs (cache) (cache) 414ms 1ms 34µs 419ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16 200 OK ★ 19ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"vgg16\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 237µs (cache) (cache) 16ms 1ms 42µs 19ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 8ms time ★ 233B↑ 4.76kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.47kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1",mod │ el_version="default",} 127025.194 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="alexnet",model_ver │ sion="default",} 98667.946 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="densenet161",model │ _version="default",} 443633.096 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db │ (showing 2.05kB/4.47kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 233µs (cache) (cache) 1ms 5ms 42µs 9ms ✓ Successful GET request Iteration 11/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/vgg16_scripted.mar&model_name=vgg16_scripted&initial_workers=1&synchronous=true 200 OK ★ 1m 47.8s time ★ 377B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"vgg16_scripted\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 202µs (cache) (cache) 1m 47.8s 1ms 31µs 1m 47.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/vgg16_scripted 200 OK ★ 518ms time ★ 111.25kB↑ 408B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 173B │ { │ "tiger_cat": 0.44697263836860657, │ "tabby": 0.4408797323703766, │ "Egyptian_cat": 0.05904562026262283, │ "tiger": 0.020596399903297424, │ "lynx": 0.009934580884873867 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 253µs (cache) (cache) 515ms 1ms 30µs 519ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/vgg16_scripted 200 OK ★ 19ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"vgg16_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 195µs (cache) (cache) 16ms 1ms 34µs 19ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 5.17kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 4.87kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1",mod │ el_version="default",} 127025.194 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="vgg16_scripted",mo │ del_version="default",} 512346.751 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4" │ (showing 2.05kB/4.87kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 205µs (cache) (cache) 1ms 3ms 62µs 6ms ✓ Successful GET request Iteration 12/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_v2.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 3.9s time ★ 362B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 2.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 248µs (cache) (cache) 3.9s 1ms 39µs 3.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist 200 OK ★ 22ms time ★ 537B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 191µs (cache) (cache) 19ms 1ms 29µs 23ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 18ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 984µs 160µs (cache) (cache) 15ms 1ms 33µs 18ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 5.54kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.25kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1",mod │ el_version="default",} 127025.194 │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="alexnet", │ (showing 2.05kB/5.25kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 179µs (cache) (cache) 1ms 1ms 32µs 4ms ✓ Successful GET request Iteration 13/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/mnist_scripted_v2.mar&model_name=mnist_scripted&initial_workers=1&synchronous=true 200 OK ★ 2.9s time ★ 380B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"mnist_scripted\" Version: 2.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 180µs (cache) (cache) 2.9s 945µs 25µs 2.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/mnist_scripted 200 OK ★ 66ms time ★ 546B↑ 234B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 1B │ 0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 178µs (cache) (cache) 64ms 1ms 24µs 67ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist_scripted 200 OK ★ 22ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"mnist_scripted\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 217µs (cache) (cache) 18ms 2ms 39µs 22ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 5.95kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.66kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="8e24d624-acb8- │ 4057-bad5-f3d13de70db4",model_name="squeezenet1_1",mod │ el_ │ (showing 2.05kB/5.66kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 228µs (cache) (cache) 1ms 1ms 38µs 4ms ✓ Successful GET request Iteration 14/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fastrcnn.mar&model_name=fastrcnn&initial_workers=1&synchronous=true 200 OK ★ 5.8s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"fastrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 183µs (cache) (cache) 5.8s 1ms 32µs 5.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fastrcnn 200 OK ★ 4.8s time ★ 289.21kB↑ 3kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.76kB │ [ │ { │ "person": [ │ 167.39585876464844, │ 57.20302200317383, │ 301.3599853515625, │ 436.7974548339844 │ ], │ "score": 0.9995203018188477 │ }, │ { │ "person": [ │ 89.54701232910156, │ 64.8357162475586, │ 191.42431640625, │ 446.76068115234375 │ ], │ "score": 0.9994968175888062 │ }, │ { │ "person": [ │ 362.37823486328125, │ 161.84130859375, │ 515.5023193359375, │ 385.2898254394531 │ ], │ "score": 0.9977061748504639 │ }, │ { │ "handbag": [ │ 67.3762435913086, │ 277.5755920410156, │ 111.67390441894531, │ 400.2067565917969 │ ], │ "score": 0.9924989342689514 │ }, │ { │ "handbag": [ │ 228.6824951171875, │ 146.00692749023438, │ 303.55120849609375, │ 231.08848571777344 │ ], │ "score": 0.9922404289245605 │ }, │ { │ "handbag": [ │ 379.411376953125, │ 259.9578552246094, │ 419.0797424316406, │ 317.9610290527344 │ ], │ "score": 0.9898616075515747 │ }, │ { │ "person": [ │ 518.49609375, │ 149.73033142089844, │ 636.6343994140625, │ 365.4130554199219 │ ], │ "score": 0.9821102023124695 │ }, │ { │ "bench": [ │ 269.0818176269531, │ 217.34210205078125, │ 423.77008056640625, │ 390.37847900390625 │ ], │ "score": 0.9573417901992798 │ }, │ { │ "person": [ │ 539.6298217773438, │ 157.7587127685547, │ 616.1531372070312, │ 253.1112823486328 │ ], │ "score": 0.8995429873466492 │ }, │ { │ "person": [ │ 477.0623779296875, │ 147.8089141845703, │ 610.77734375, │ 296.8367004394531 │ ], │ "score": 0.8751581311225891 │ }, │ { │ "bench": [ │ 286.0877990722656, │ 216.53292846679688, │ 550.7698364257812, │ 383.1821594238281 │ ], │ "score": 0.8436439037322998 │ }, │ { │ "person": [ │ 627.4741821289062, │ 177.05836486816406, │ 640.0, │ 247.72146606445312 │ ], │ "score": 0.8257301449775696 │ }, │ { │ "bench": [ │ 88.76304626464844, │ 226.1649932861328, │ 563.6138916015625, │ 421. │ (showing 2.05kB/2.76kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 3ms 296µs (cache) (cache) 4.8s 1ms 32µs 4.8s ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fastrcnn 200 OK ★ 17ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"fastrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 189µs (cache) (cache) 14ms 1ms 37µs 17ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 6.34kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.05kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microsec │ (showing 2.05kB/6.05kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 214µs (cache) (cache) 1ms 1ms 34µs 4ms ✓ Successful GET request Iteration 15/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=maskrcnn.mar&model_name=maskrcnn&initial_workers=1&synchronous=true 200 OK ★ 5.6s time ★ 324B↑ 350B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 84B │ { │ "status": "Model \"maskrcnn\" Version: 1.0 registere │ d with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 199µs (cache) (cache) 5.6s 1ms 30µs 5.6s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/maskrcnn 200 OK ★ 5.5s time ★ 289.21kB↑ 3kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.76kB │ [ │ { │ "person": [ │ 169.50631713867188, │ 49.98546600341797, │ 300.8945617675781, │ 442.4545593261719 │ ], │ "score": 0.9995434880256653 │ }, │ { │ "person": [ │ 90.4118423461914, │ 66.7905044555664, │ 194.1930389404297, │ 437.27777099609375 │ ], │ "score": 0.9993956089019775 │ }, │ { │ "person": [ │ 362.3172912597656, │ 158.1734619140625, │ 521.2589721679688, │ 385.7226867675781 │ ], │ "score": 0.9952279329299927 │ }, │ { │ "handbag": [ │ 68.57540893554688, │ 279.3182067871094, │ 111.15328979492188, │ 400.9166564941406 │ ], │ "score": 0.9938817024230957 │ }, │ { │ "person": [ │ 474.01568603515625, │ 147.34800720214844, │ 638.1206665039062, │ 364.6508483886719 │ ], │ "score": 0.9897462129592896 │ }, │ { │ "handbag": [ │ 225.59584045410156, │ 142.90005493164062, │ 302.48638916015625, │ 230.3284149169922 │ ], │ "score": 0.9891214966773987 │ }, │ { │ "handbag": [ │ 380.2604675292969, │ 259.20135498046875, │ 419.53656005859375, │ 318.27728271484375 │ ], │ "score": 0.9688040614128113 │ }, │ { │ "bench": [ │ 273.48565673828125, │ 217.4885711669922, │ 441.06439208984375, │ 396.241455078125 │ ], │ "score": 0.961753249168396 │ }, │ { │ "person": [ │ 541.289794921875, │ 156.6612548828125, │ 619.938232421875, │ 249.45277404785156 │ ], │ "score": 0.8177030086517334 │ }, │ { │ "person": [ │ 362.9620666503906, │ 163.8992462158203, │ 500.7699279785156, │ 293.9123840332031 │ ], │ "score": 0.8016932010650635 │ }, │ { │ "chair": [ │ 455.20855712890625, │ 207.54006958007812, │ 491.0853271484375, │ 274.64752197265625 │ ], │ "score": 0.77582848072052 │ }, │ { │ "person": [ │ 549.1541137695312, │ 177.42166137695312, │ 640.0, │ 364.5395202636719 │ ], │ "score": 0.7176401615142822 │ }, │ { │ "person": [ │ 626.230712890625, │ 178.65342712402344, │ 640.0, │ 24 │ (showing 2.05kB/2.76kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 325µs (cache) (cache) 5.5s 1ms 29µs 5.5s ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/maskrcnn 200 OK ★ 17ms time ★ 244B↑ 316B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 50B │ { │ "status": "Model \"maskrcnn\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 211µs (cache) (cache) 14ms 1ms 35µs 18ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 6.73kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.44kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ (showing 2.05kB/6.44kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 197µs (cache) (cache) 1ms 2ms 42µs 5ms ✓ Successful GET request Iteration 16/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=fcn_resnet_101.mar&model_name=fcn_resnet_101&initial_workers=1&synchronous=true 200 OK ★ 6.9s time ★ 336B↑ 356B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "status": "Model \"fcn_resnet_101\" Version: 1.0 reg │ istered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 187µs (cache) (cache) 6.9s 1ms 31µs 6.9s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101 200 OK ★ 1237ms time ★ 289.22kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993864297866821 │ ], │ [ │ 0.0, │ 0.9993854761123657 │ ], │ [ │ 0.0, │ 0.9993811845779419 │ ], │ [ │ 0.0, │ 0.9993742108345032 │ ], │ [ │ 0.0, │ 0.9993641972541809 │ ], │ [ │ 0.0, │ 0.9993515610694885 │ ], │ [ │ 0.0, │ 0.9993364214897156 │ ], │ [ │ 0.0, │ 0.9993187189102173 │ ], │ [ │ 0.0, │ 0.9992934465408325 │ ], │ [ │ 0.0, │ 0.9992607235908508 │ ], │ [ │ 0.0, │ 0.9992249011993408 │ ], │ [ │ 0.0, │ 0.9991866946220398 │ ], │ [ │ 0.0, │ 0.9991452693939209 │ ], │ [ │ 0.0, │ 0.9991005659103394 │ ], │ [ │ 0.0, │ 0.9990523457527161 │ ], │ [ │ 0.0, │ 0.9990004897117615 │ ], │ [ │ 0.0, │ 0.998968243598938 │ ], │ [ │ 0.0, │ 0.9989572763442993 │ ], │ [ │ 0.0, │ 0.9989456534385681 │ ], │ [ │ 0.0, │ 0.9989331364631653 │ ], │ [ │ 0.0, │ 0.9989200830459595 │ ], │ [ │ 0.0, │ 0.9989060163497925 │ ], │ [ │ 0.0, │ 0.998891294002533 │ ], │ [ │ 0.0, │ 0.9988754391670227 │ ], │ [ │ 0.0, │ 0.9988815188407898 │ ], │ [ │ 0.0, │ 0.9989089965820312 │ ], │ [ │ 0.0, │ 0.9989352822303772 │ ], │ [ │ 0.0, │ 0.9989607334136963 │ ], │ [ │ 0.0, │ 0.9989848732948303 │ ], │ [ │ 0.0, │ 0.9990076422691345 │ ], │ [ │ 0.0, │ 0.9990293979644775 │ ], │ [ │ 0.0, │ 0.9990498423576355 │ ], │ [ │ 0.0, │ 0.999051034450531 │ ], │ [ │ 0.0, │ 0.9990313053131104 │ ], │ [ │ 0.0, │ 0.9990096092224121 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.9989587068557739 │ ], │ [ │ 0.0, │ 0.99892979860 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 268µs (cache) (cache) 1232ms 4ms 1ms 1240ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101 200 OK ★ 19ms time ★ 250B↑ 322B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 56B │ { │ "status": "Model \"fcn_resnet_101\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 281µs (cache) (cache) 17ms 1ms 56µs 21ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 3ms time ★ 233B↑ 7.14kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 6.85kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_ │ (showing 2.05kB/6.85kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 197µs (cache) (cache) 1ms 1ms 35µs 4ms ✓ Successful GET request Iteration 17/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/fcn_resnet_101_scripted.mar&model_name=fcn_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 41.8s time ★ 395B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 99B │ { │ "status": "Model \"fcn_resnet_101_scripted\" Version │ : 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 198µs (cache) (cache) 41.8s 3ms 48µs 41.8s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/fcn_resnet_101_scripted 200 OK ★ 1542ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993855953216553 │ ], │ [ │ 0.0, │ 0.9993864297866821 │ ], │ [ │ 0.0, │ 0.9993854761123657 │ ], │ [ │ 0.0, │ 0.9993811845779419 │ ], │ [ │ 0.0, │ 0.9993742108345032 │ ], │ [ │ 0.0, │ 0.9993641972541809 │ ], │ [ │ 0.0, │ 0.9993515610694885 │ ], │ [ │ 0.0, │ 0.9993364214897156 │ ], │ [ │ 0.0, │ 0.9993187189102173 │ ], │ [ │ 0.0, │ 0.9992934465408325 │ ], │ [ │ 0.0, │ 0.9992607235908508 │ ], │ [ │ 0.0, │ 0.9992249011993408 │ ], │ [ │ 0.0, │ 0.9991866946220398 │ ], │ [ │ 0.0, │ 0.9991452693939209 │ ], │ [ │ 0.0, │ 0.9991005659103394 │ ], │ [ │ 0.0, │ 0.9990523457527161 │ ], │ [ │ 0.0, │ 0.9990004897117615 │ ], │ [ │ 0.0, │ 0.998968243598938 │ ], │ [ │ 0.0, │ 0.9989572763442993 │ ], │ [ │ 0.0, │ 0.9989456534385681 │ ], │ [ │ 0.0, │ 0.9989331364631653 │ ], │ [ │ 0.0, │ 0.9989200830459595 │ ], │ [ │ 0.0, │ 0.9989060163497925 │ ], │ [ │ 0.0, │ 0.998891294002533 │ ], │ [ │ 0.0, │ 0.9988754391670227 │ ], │ [ │ 0.0, │ 0.9988815188407898 │ ], │ [ │ 0.0, │ 0.9989089965820312 │ ], │ [ │ 0.0, │ 0.9989352822303772 │ ], │ [ │ 0.0, │ 0.9989607334136963 │ ], │ [ │ 0.0, │ 0.9989848732948303 │ ], │ [ │ 0.0, │ 0.9990076422691345 │ ], │ [ │ 0.0, │ 0.9990293979644775 │ ], │ [ │ 0.0, │ 0.9990498423576355 │ ], │ [ │ 0.0, │ 0.999051034450531 │ ], │ [ │ 0.0, │ 0.9990313053131104 │ ], │ [ │ 0.0, │ 0.9990096092224121 │ ], │ [ │ 0.0, │ 0.9989851117134094 │ ], │ [ │ 0.0, │ 0.9989587068557739 │ ], │ [ │ 0.0, │ 0.99892979860 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 477µs (cache) (cache) 1537ms 3ms 1ms 1548ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/fcn_resnet_101_scripted 200 OK ★ 18ms time ★ 259B↑ 331B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 65B │ { │ "status": "Model \"fcn_resnet_101_scripted\" unregis │ tered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 320µs (cache) (cache) 15ms 1ms 71µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 7.58kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 7.29kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16",model_version="de │ fault",} 1. │ (showing 2.05kB/7.29kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 220µs (cache) (cache) 1ms 1ms 42µs 4ms ✓ Successful GET request Iteration 18/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=deeplabv3_resnet_101_eager.mar&model_name=deeplabv3_resnet_101_eager&initial_workers=1&synchronous=true 200 OK ★ 7.7s time ★ 360B↑ 369B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" Vers │ ion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 194µs (cache) (cache) 7.7s 1ms 35µs 7.7s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_eager 200 OK ★ 1411ms time ★ 289.23kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988669157028198 │ ], │ [ │ 0.0, │ 0.998843789100647 │ ], │ [ │ 0.0, │ 0.9988172650337219 │ ], │ [ │ 0.0, │ 0.9987861514091492 │ ], │ [ │ 0.0, │ 0.9987504482269287 │ ], │ [ │ 0.0, │ 0.9987117052078247 │ ], │ [ │ 0.0, │ 0.9986679553985596 │ ], │ [ │ 0.0, │ 0.9986202716827393 │ ], │ [ │ 0.0, │ 0.9985180497169495 │ ], │ [ │ 0.0, │ 0.9983478784561157 │ ], │ [ │ 0.0, │ 0.9981531500816345 │ ], │ [ │ 0.0, │ 0.9979324340820312 │ ], │ [ │ 0.0, │ 0.9976813793182373 │ ], │ [ │ 0.0, │ 0.9973964691162109 │ ], │ [ │ 0.0, │ 0.9970728158950806 │ ], │ [ │ 0.0, │ 0.9967058300971985 │ ], │ [ │ 0.0, │ 0.9964045286178589 │ ], │ [ │ 0.0, │ 0.9961955547332764 │ ], │ [ │ 0.0, │ 0.9959736466407776 │ ], │ [ │ 0.0, │ 0.9957382678985596 │ ], │ [ │ 0.0, │ 0.9954884648323059 │ ], │ [ │ 0.0, │ 0.9952237010002136 │ ], │ [ │ 0.0, │ 0.9949429631233215 │ ], │ [ │ 0.0, │ 0.9946452379226685 │ ], │ [ │ 0.0, │ 0.9945149421691895 │ ], │ [ │ 0.0, │ 0.9945646524429321 │ ], │ [ │ 0.0, │ 0.9946145415306091 │ ], │ [ │ 0.0, │ 0.9946632385253906 │ ], │ [ │ 0.0, │ 0.9947116374969482 │ ], │ [ │ 0.0, │ 0.9947587847709656 │ ], │ [ │ 0.0, │ 0.9948055148124695 │ ], │ [ │ 0.0, │ 0.9948518872261047 │ ], │ [ │ 0.0, │ 0.9948964715003967 │ ], │ [ │ 0.0, │ 0.9949400424957275 │ ], │ [ │ 0.0, │ 0.9949837923049927 │ ], │ [ │ 0.0, │ 0.9950267672538757 │ ], │ [ │ 0.0, │ 0.9950693845748901 │ ], │ [ │ 0.0, │ 0.995111525 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 211µs (cache) (cache) 1405ms 4ms 1ms 1414ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_eager 200 OK ★ 22ms time ★ 262B↑ 334B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 68B │ { │ "status": "Model \"deeplabv3_resnet_101_eager\" unre │ gistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 6ms 1ms (cache) (cache) 17ms 1ms 57µs 27ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 8.02kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 7.73kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="deeplabv3_resnet_101_eag │ er",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1_scripted", │ model_v │ (showing 2.05kB/7.73kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 286µs (cache) (cache) 1ms 1ms 40µs 4ms ✓ Successful GET request Iteration 19/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.s3.amazonaws.com/mar_files/deeplabv3_resnet_101_scripted.mar&model_name=deeplabv3_resnet_101_scripted&initial_workers=1&synchronous=true 200 OK ★ 53s time ★ 412B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 291µs (cache) (cache) 53s 2ms 46µs 53s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/deeplabv3_resnet_101_scripted 200 OK ★ 1494ms time ★ 289.24kB↑ 2.47MB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 288.94kB │ (showing 2.05kB/288.94kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 2.47MB │ [ │ [ │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988765120506287 │ ], │ [ │ 0.0, │ 0.9988669157028198 │ ], │ [ │ 0.0, │ 0.998843789100647 │ ], │ [ │ 0.0, │ 0.9988172650337219 │ ], │ [ │ 0.0, │ 0.9987861514091492 │ ], │ [ │ 0.0, │ 0.9987504482269287 │ ], │ [ │ 0.0, │ 0.9987117052078247 │ ], │ [ │ 0.0, │ 0.9986679553985596 │ ], │ [ │ 0.0, │ 0.9986202716827393 │ ], │ [ │ 0.0, │ 0.9985180497169495 │ ], │ [ │ 0.0, │ 0.9983478784561157 │ ], │ [ │ 0.0, │ 0.9981531500816345 │ ], │ [ │ 0.0, │ 0.9979324340820312 │ ], │ [ │ 0.0, │ 0.9976813793182373 │ ], │ [ │ 0.0, │ 0.9973964691162109 │ ], │ [ │ 0.0, │ 0.9970728158950806 │ ], │ [ │ 0.0, │ 0.9967058300971985 │ ], │ [ │ 0.0, │ 0.9964045286178589 │ ], │ [ │ 0.0, │ 0.9961955547332764 │ ], │ [ │ 0.0, │ 0.9959736466407776 │ ], │ [ │ 0.0, │ 0.9957382678985596 │ ], │ [ │ 0.0, │ 0.9954884648323059 │ ], │ [ │ 0.0, │ 0.9952237010002136 │ ], │ [ │ 0.0, │ 0.9949429631233215 │ ], │ [ │ 0.0, │ 0.9946452379226685 │ ], │ [ │ 0.0, │ 0.9945149421691895 │ ], │ [ │ 0.0, │ 0.9945646524429321 │ ], │ [ │ 0.0, │ 0.9946145415306091 │ ], │ [ │ 0.0, │ 0.9946632385253906 │ ], │ [ │ 0.0, │ 0.9947116374969482 │ ], │ [ │ 0.0, │ 0.9947587847709656 │ ], │ [ │ 0.0, │ 0.9948055148124695 │ ], │ [ │ 0.0, │ 0.9948518872261047 │ ], │ [ │ 0.0, │ 0.9948964715003967 │ ], │ [ │ 0.0, │ 0.9949400424957275 │ ], │ [ │ 0.0, │ 0.9949837923049927 │ ], │ [ │ 0.0, │ 0.9950267672538757 │ ], │ [ │ 0.0, │ 0.9950693845748901 │ ], │ [ │ 0.0, │ 0.995111525 │ (showing 2.05kB/2.47MB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 6ms 569µs (cache) (cache) 1489ms 3ms 388µs 1500ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/deeplabv3_resnet_101_scripted 200 OK ★ 19ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"deeplabv3_resnet_101_scripted\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 273µs (cache) (cache) 16ms 1ms 56µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 8.48kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.19kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="deeplabv3_resnet_101_scr │ ipted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="deeplabv3_resnet_101_eag │ er",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18", │ (showing 2.05kB/8.19kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 230µs (cache) (cache) 1ms 1ms 38µs 4ms ✓ Successful GET request Iteration 20/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&initial_workers=1&synchronous=true 200 OK ★ 7.4s time ★ 333B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 254µs (cache) (cache) 7.4s 1ms 30µs 7.4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet152 200 OK ★ 378ms time ★ 111.24kB↑ 411B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 176B │ { │ "tiger_cat": 0.5798617601394653, │ "tabby": 0.38344141840934753, │ "Egyptian_cat": 0.03421143442392349, │ "lynx": 0.0005819805082865059, │ "quilt": 0.00027331963065080345 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 333µs (cache) (cache) 375ms 1ms 26µs 380ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet152 200 OK ★ 16ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 220µs (cache) (cache) 14ms 1ms 30µs 16ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 8.87kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 8.58kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="deeplabv3_resnet_101_scr │ ipted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet152",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="deeplabv3_resnet_101_eag │ er", │ (showing 2.05kB/8.58kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 187µs (cache) (cache) 1ms 1ms 67µs 4ms ✓ Successful GET request Iteration 21/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/resnet-152-batch_scripted.mar&model_name=resnet-152-batch_scripted&initial_workers=1&synchronous=true 200 OK ★ 46.5s time ★ 399B↑ 368B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "status": "Model \"resnet-152-batch_scripted\" Versi │ on: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 191µs (cache) (cache) 46.5s 2ms 78µs 46.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/resnet-152-batch_scripted 200 OK ★ 680ms time ★ 111.26kB↑ 411B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 176B │ { │ "tiger_cat": 0.5798617601394653, │ "tabby": 0.38344141840934753, │ "Egyptian_cat": 0.03421143442392349, │ "lynx": 0.0005819805082865059, │ "quilt": 0.00027331963065080345 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 265µs (cache) (cache) 677ms 1ms 29µs 681ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/resnet-152-batch_scripted 200 OK ★ 17ms time ★ 261B↑ 333B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 67B │ { │ "status": "Model \"resnet-152-batch_scripted\" unreg │ istered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 243µs (cache) (cache) 13ms 2ms 44µs 18ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 9.31kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.02kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="deeplabv3_resnet_101_scr │ ipted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet152",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="deeplabv3_resnet_101_eag │ er", │ (showing 2.05kB/9.02kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 455µs (cache) (cache) 2ms 1ms 41µs 6ms ✓ Successful GET request Iteration 22/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/distill_bert_qa_eager.mar&model_name=distill_bert_qa_eager&initial_workers=1&synchronous=true 200 OK ★ 51.3s time ★ 391B↑ 363B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "status": "Model \"distill_bert_qa_eager\" Version: │ 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 213µs (cache) (cache) 51.3s 1ms 35µs 51.3s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/distill_bert_qa_eager 200 OK ★ 244ms time ★ 359B↑ 247B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 79B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13B │ a nice puppet └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 254µs (cache) (cache) 241ms 1ms 26µs 246ms ✓ Successful POST request → Model Zoo - Unregister model DELETE http://localhost:8081/models/distill_bert_qa_eager 200 OK ★ 18ms time ★ 257B↑ 329B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Model \"distill_bert_qa_eager\" unregiste │ red" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 205µs (cache) (cache) 15ms 1ms 35µs 18ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 9.74kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.45kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="deeplabv3_resnet_101_scr │ ipted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="distill_bert_qa_eager",m │ odel_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet152",model │ (showing 2.05kB/9.45kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 368µs (cache) (cache) 2ms 1ms 39µs 6ms ✓ Successful GET request Iteration 23/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_token_classification_no_torchscript.mar&model_name=bert_token_classification_no_torchscript&initial_workers=1&synchronous=true 200 OK ★ 1m 24s time ★ 429B↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 116B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" Version: 1.0 registered with 1 initial worke │ rs" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 214µs (cache) (cache) 1m 24s 1ms 33µs 1m 24s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_token_classification_no_torchscript 200 OK ★ 409ms time ★ 375B↑ 5.45kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 5.22kB │ [ │ [ │ "[CLS]", │ "B-LOC" │ ], │ [ │ "bloomberg", │ "O" │ ], │ [ │ "has", │ "I-ORG" │ ], │ [ │ "decided", │ "B-PER" │ ], │ [ │ "to", │ "O" │ ], │ [ │ "publish", │ "I-PER" │ ], │ [ │ "a", │ "I-MISC" │ ], │ [ │ "new", │ "B-PER" │ ], │ [ │ "report", │ "O" │ ], │ [ │ "on", │ "O" │ ], │ [ │ "global", │ "B-PER" │ ], │ [ │ "economic", │ "O" │ ], │ [ │ "situation", │ "B-LOC" │ ], │ [ │ ".", │ "B-PER" │ ], │ [ │ "[SEP]", │ "B-PER" │ ], │ [ │ "[PAD]", │ "I-ORG" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ "B-MISC" │ ], │ [ │ "[PAD]", │ (showing 2.05kB/5.22kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 236µs (cache) (cache) 406ms 1ms 35µs 410ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_token_classification_no_torchscript 200 OK ★ 18ms time ★ 276B↑ 348B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 82B │ { │ "status": "Model \"bert_token_classification_no_torc │ hscript\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 226µs (cache) (cache) 15ms 1ms 36µs 19ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 5ms time ★ 233B↑ 10.23kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 9.93kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet_scripted",model_ │ version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="deeplabv3_resnet_101_scr │ ipted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="maskrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist",model_version="de │ fault",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="distill_bert_qa_eager",m │ odel_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="mnist_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet152",model │ (showing 2.05kB/9.93kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 203µs (cache) (cache) 2ms 2ms 45µs 5ms ✓ Successful GET request Iteration 24/24 → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/bert_seqc_without_torchscript.mar&model_name=bert_seqc_without_torchscript&initial_workers=1&synchronous=true 200 OK ★ 1m 32.5s time ★ 407B↑ 372B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 105B │ { │ "status": "Model \"bert_seqc_without_torchscript\" V │ ersion: 1.0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 219µs (cache) (cache) 1m 32.5s 1ms 30µs 1m 32.5s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/bert_seqc_without_torchscript 200 OK ★ 354ms time ★ 364B↑ 246B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 76B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 12B │ Not Accepted └ prepare wait dns-lookup tcp-handshake transfer-start download process total 4ms 229µs (cache) (cache) 351ms 1ms 31µs 357ms ✓ Successful POST request ✓ Test expected TEXT response → Model Zoo - Unregister model DELETE http://localhost:8081/models/bert_seqc_without_torchscript 200 OK ★ 19ms time ★ 265B↑ 337B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 71B │ { │ "status": "Model \"bert_seqc_without_torchscript\" u │ nregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 205µs (cache) (cache) 17ms 1ms 33µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 4ms time ★ 233B↑ 10.68kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 10.39kB │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="squeezenet1_1",model_ver │ sion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fastrcnn",model_version= │ "default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="bert_seqc_without_torchs │ cript",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="densenet161",model_versi │ on="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier_scrip │ ted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="deeplabv3_resnet_101_scr │ ipted",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="vgg16_scripted",model_ve │ rsion="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet152",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="deeplabv3_resnet_101_eag │ er",model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18",model_version │ ="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="my_text_classifier",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="resnet-18_scripted",mode │ l_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="fcn_resnet_101_scripted" │ ,model_version="default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db4",model_name="alexnet",model_version=" │ default",} 1.0 │ ts_inference_requests_total{uuid="8e24d624-acb8-4057-b │ ad5-f3d13de70db │ (showing 2.05kB/10.39kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 244µs (cache) (cache) 1ms 1ms 45µs 6ms ✓ Successful GET request ┌─────────────────────────┬──────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ iterations │ 24 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ requests │ 96 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ test-scripts │ 96 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ assertions │ 113 │ 0 │ ├─────────────────────────┴──────────────────────┴─────────────────────┤ │ total run duration: 11m 58.6s │ ├──────────────────────────────────────────────────────────────────────┤ │ total data received: 10.04MB (approx) │ ├──────────────────────────────────────────────────────────────────────┤ │ average response time: 7.4s [min: 3ms, max: 1m 47.8s, s.d.: 19.7s] │ ├──────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 822µs [min: 179µs, max: 1ms, s.d.: 508µs] │ ├──────────────────────────────────────────────────────────────────────┤ │ average first byte time: 7.4s [min: 1ms, max: 1m 47.8s, s.d.: 19.7s] │ └──────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman inference → Model Zoo - Register Model POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161_scripted.mar&model_name=densenet161_scripted&initial_workers=1&synchronous=true 200 OK ★ 26.4s time ★ 389B↑ 362B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 96B │ { │ "status": "Model \"densenet161_scripted\" Version: 1 │ .0 registered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 29ms 4ms 1ms 344µs 26.3s 5ms 320µs 26.4s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/predictions/densenet161_scripted 200 OK ★ 990ms time ★ 111.25kB↑ 418B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 183B │ { │ "tabby": 0.46661895513534546, │ "tiger_cat": 0.46449047327041626, │ "Egyptian_cat": 0.06614057719707489, │ "lynx": 0.0012924439506605268, │ "plastic_bag": 0.00022909804829396307 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 6ms 1ms 313µs 214µs 985ms 2ms 44µs 996ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/densenet161_scripted 200 OK ★ 19ms time ★ 256B↑ 328B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 62B │ { │ "status": "Model \"densenet161_scripted\" unregister │ ed" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 427µs (cache) (cache) 15ms 2ms 103µs 20ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 18ms time ★ 233B↑ 1.09kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 800B │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="81408b26-116d-4a30 │ -b690-73ae89a291c2",model_name="densenet161_scripted", │ model_version="default",} 140.97 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="81408b26-116d- │ 4a30-b690-73ae89a291c2",model_name="densenet161_script │ ed",model_version="default",} 962067.112 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="81408b26-116d-4a30-b │ 690-73ae89a291c2",model_name="densenet161_scripted",mo │ del_version="default",} 1.0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 651µs 185µs 220µs 15ms 1ms 70µs 19ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 27.6s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 1.14kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 6.8s [min: 18ms, max: 26.4s, s.d.: 11.2s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 628µs [min: 185µs, max: 1ms, s.d.: 475µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 6.8s [min: 15ms, max: 26.3s, s.d.: 11.2s] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 172ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 32ms 12ms 1ms 3ms 116ms 31ms 5ms 403µs 204ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 46ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 1ms 172µs 350µs 29ms 12ms 1ms 61µs 47ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 35ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.0" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 924µs 151µs 351µs 27ms 3ms 1ms 60µs 35ms ✓ Status code is 200 → HTTPS Register Model - SqueezeNet POST https://localhost:8444/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true 200 OK ★ 1936ms time ★ 334B↑ 355B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 89B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 513µs (cache) (cache) (cache) 1933ms 1ms 74µs 1938ms ✓ Successful POST request → HTTPS Get SqueezeNet Model Description GET https://localhost:8444/models/squeezenet1_1 200 OK ★ 6ms time ★ 246B↑ 753B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 486B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-03-22T12:16:56.551Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 50275, │ "gpu": false, │ "gpuUsage": "N/A" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 250µs (cache) (cache) (cache) 3ms 1ms 47µs 6ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 4ms time ★ 308B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 290µs (cache) (cache) (cache) 2ms 1ms 43µs 5ms ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous PUT https://localhost:8444/models/squeezenet1_1?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 5ms time ★ 309B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 313µs (cache) (cache) (cache) 2ms 1ms 38µs 5ms ✓ Successful PUT request → HTTPS - Inference - SqueezeNet POST https://localhost:8443/predictions/squeezenet1_1 200 OK ★ 66ms time ★ 111.25kB↑ 409B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 174B │ { │ "tabby": 0.27850738167762756, │ "lynx": 0.25299158692359924, │ "tiger_cat": 0.24496474862098694, │ "Egyptian_cat": 0.21722468733787537, │ "cougar": 0.002217544475570321 │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 5ms 757µs (cache) (cache) (cache) 63ms 1ms 38µs 71ms ✓ Status code is 200 → HTTPS UnRegister Model SqueezeNet DELETE https://localhost:8444/models/squeezenet1_1 200 OK ★ 18ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 254µs (cache) (cache) (cache) 16ms 1ms 31µs 18ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 9 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 2.5s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 85.45kB (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 254ms [min: 4ms, max: 1936ms, s.d.: 596ms] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 526µs [min: 151µs, max: 1ms, s.d.: 515µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 229ms [min: 2ms, max: 1933ms, s.d.: 602ms] │ └─────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman management_api_collection Iteration 1/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 220ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 30ms 5ms 1ms 282µs 205ms 6ms 375µs 250ms ✓ Successful request Iteration 2/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist 200 OK ★ 113ms time ★ 283B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 518µs (cache) (cache) 110ms 1ms 58µs 113ms ✓ Successful request Iteration 3/82 → management request POST http://localhost:8081/models?url=densenet161.mar&model_name=densenet161 200 OK ★ 2.1s time ★ 295B↑ 407B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 140B │ { │ "status": "Model \"densenet161\" Version: 1.0 regist │ ered with 0 initial workers. Use scale workers API to │ add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 277µs (cache) (cache) 2.1s 1ms 49µs 2.1s ✓ Successful request Iteration 4/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/densenet161.mar&model_name=densenet161 500 Internal Server Error ★ 5ms time ★ 336B↑ 394B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 113B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Model file already exists densenet161.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 227µs (cache) (cache) 2ms 1ms 49µs 5ms ✓ Successful request Iteration 5/82 → management request DELETE http://localhost:8081/models/densenet161 200 OK ★ 17ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Model \"densenet161\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 609µs 173µs 244µs 13ms 1ms 61µs 17ms ✓ Successful request Iteration 6/82 → management request POST http://localhost:8081/models 400 Bad Request ★ 5ms time ★ 252B↑ 364B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Parameter url is required." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 281µs (cache) (cache) 2ms 1ms 52µs 5ms ✓ Successful request Iteration 7/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 17ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 587µs 171µs 249µs 13ms 1ms 41µs 17ms ✓ Successful request Iteration 8/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 103ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 265µs (cache) (cache) 101ms 1ms 31µs 104ms ✓ Successful request Iteration 9/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 7ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 931µs 171µs (cache) (cache) 4ms 1ms 29µs 7ms ✓ Successful request Iteration 10/82 → management request POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&handler=invalidHandler 200 OK ★ 91ms time ★ 306B↑ 401B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 0 initial workers. Use scale workers API to add wo │ rkers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 166µs (cache) (cache) 89ms 1ms 26µs 91ms ✓ Successful request Iteration 11/82 → management request PUT http://localhost:8081/models/mnist?min_worker=1&synchronous=true 500 Internal Server Error ★ 925ms time ★ 287B↑ 406B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 125B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model mnist │ version: null" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 215µs (cache) (cache) 923ms 1ms 31µs 925ms ✓ Successful request Iteration 12/82 → management request DELETE http://localhost:8081/models/mnist 200 OK ★ 18ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 600µs 179µs 231µs 15ms 1ms 34µs 18ms ✓ Successful request Iteration 13/82 → management request GET http://localhost:8081/models/squeezenet1_1/all 200 OK ★ 6ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 235µs (cache) (cache) 4ms 1ms 35µs 6ms ✓ Successful request Iteration 14/82 → management request GET http://localhost:8081/models/squeezenet1_1/1.0 200 OK ★ 3ms time ★ 250B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 217µs (cache) (cache) 1ms 1ms 40µs 4ms ✓ Successful request Iteration 15/82 → management request GET http://localhost:8081/models/squeezenet1_1 200 OK ★ 4ms time ★ 246B↑ 538B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 271B │ [ │ { │ "modelName": "squeezenet1_1", │ "modelVersion": "1.0", │ "modelUrl": "squeezenet1_1.mar", │ "runtime": "python", │ "minWorkers": 0, │ "maxWorkers": 0, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [] │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 206µs (cache) (cache) 1ms 1ms 33µs 4ms ✓ Successful request Iteration 16/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 226µs (cache) (cache) 4ms 1ms 36µs 6ms ✓ Successful request Iteration 17/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&runtime=python4 400 Bad Request ★ 4ms time ★ 315B↑ 373B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "Invalid RuntimeType value: python4" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 249µs (cache) (cache) 1ms 1ms 31µs 4ms ✓ Successful request Iteration 18/82 → management request GET http://localhost:8081/models?limit=&next_page_token= 200 OK ★ 14ms time ★ 256B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 540µs 164µs 227µs 11ms 1ms 30µs 14ms ✓ Successful request Iteration 19/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 101ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 244µs (cache) (cache) 99ms 942µs 23µs 101ms ✓ Successful request Iteration 20/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1 202 Accepted ★ 4ms time ★ 278B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 170µs (cache) (cache) 2ms 1ms 32µs 4ms ✓ Successful request Iteration 21/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&synchronous=true 200 OK ★ 3ms time ★ 295B↑ 329B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 63B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 999µs 171µs (cache) (cache) 1ms 1ms 29µs 3ms ✓ Successful request Iteration 22/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0?min_worker=1&synchronous=true 200 OK ★ 4ms time ★ 299B↑ 343B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 77B │ { │ "status": "Workers scaled to 1 for model: squeezenet │ 1_1, version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 198µs (cache) (cache) 1ms 1ms 27µs 4ms ✓ Successful request Iteration 23/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0?min_worker=1&synchronous=true 404 Not Found ★ 3ms time ★ 299B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 136B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version: 0.0 does not exist for mo │ del: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 199µs (cache) (cache) 1ms 1ms 30µs 3ms ✓ Successful request Iteration 24/82 → management request PUT http://localhost:8081/models/squeezenet1_1?min_worker=1&number_gpu=1 202 Accepted ★ 13ms time ★ 291B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 504µs 137µs 211µs 10ms 1ms 29µs 13ms ✓ Successful request Iteration 25/82 → management request PUT http://localhost:8081/models/squeezenet1_1/1.0/set-default 200 OK ★ 3ms time ★ 281B↑ 359B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 93B │ { │ "status": "Default vesion succsesfully updated for m │ odel \"squeezenet1_1\" to \"1.0\"" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 944µs 192µs (cache) (cache) 1ms 965µs 26µs 3ms ✓ Successful request Iteration 26/82 → management request PUT http://localhost:8081/models/squeezenet1_1/0.0/set-default 404 Not Found ★ 3ms time ★ 281B↑ 403B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 134B │ { │ "code": 404, │ "type": "ModelVersionNotFoundException", │ "message": "Model version 0.0 does not exist for mod │ el squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 272µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 27/82 → management request PUT http://localhost:8081/models/squeezenet0_1/1.0/set-default 404 Not Found ★ 13ms time ★ 281B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet0_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 907µs 499µs 138µs 209µs 10ms 1ms 29µs 13ms ✓ Successful request Iteration 28/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 24ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 901µs 473µs 128µs 192µs 21ms 1ms 28µs 24ms ✓ Successful request Iteration 29/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&handler=serve/ts/torch_handler/image_classifier.py:handle 200 OK ★ 100ms time ★ 357B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 202µs (cache) (cache) 98ms 917µs 24µs 100ms ✓ Successful request Iteration 30/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 6ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 993µs 176µs (cache) (cache) 3ms 1ms 30µs 6ms ✓ Successful request Iteration 31/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&batch_size=3&initial_workers=3&response_timeout=0 500 Internal Server Error ★ 1108ms time ★ 349B↑ 413B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 132B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Failed to start workers for model squeez │ enet1_1 version: 1.0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 961µs 183µs (cache) (cache) 1106ms 1ms 28µs 1108ms ✓ Successful request Iteration 32/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&response_timeout=0 200 OK ★ 108ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 935µs 546µs 130µs 202µs 105ms 905µs 25µs 108ms ✓ Successful request Iteration 33/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 954µs 200µs (cache) (cache) 3ms 1ms 25µs 5ms ✓ Successful request Iteration 34/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2 200 OK ★ 4.2s time ★ 311B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 200µs (cache) (cache) 4.2s 2ms 50µs 4.2s ✓ Successful request Iteration 35/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 6ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 382µs (cache) (cache) 3ms 1ms 28µs 6ms ✓ Successful request Iteration 36/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=dd&initial_workers=1 200 OK ★ 7.1s time ★ 330B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 229µs (cache) (cache) 7.1s 1ms 30µs 7.1s ✓ Successful request Iteration 37/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 17ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 189µs (cache) (cache) 14ms 1ms 31µs 17ms ✓ Successful request Iteration 38/82 → management request POST http://localhost:8081/models?url=resnet-152-batch.mar&model_name=resnet152&batch_size=2&initial_workers=1&max_batch_delay=junk 200 OK ★ 7.1s time ★ 350B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Model \"resnet152\" Version: 1.0 register │ ed with 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 216µs (cache) (cache) 7.1s 1ms 32µs 7.1s ✓ Successful request Iteration 39/82 → management request DELETE http://localhost:8081/models/resnet152 200 OK ★ 18ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet152\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 193µs (cache) (cache) 16ms 1ms 30µs 19ms ✓ Successful request Iteration 40/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1&initial_workers=-1 200 OK ★ 107ms time ★ 318B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 201µs (cache) (cache) 105ms 1ms 22µs 108ms ✓ Successful request Iteration 41/82 → management request DELETE http://localhost:8081/models/squeezenet1_1 200 OK ★ 5ms time ★ 249B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 186µs (cache) (cache) 3ms 1ms 30µs 6ms ✓ Successful request Iteration 42/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=true 200 OK ★ 825ms time ★ 308B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 172µs (cache) (cache) 823ms 811µs 21µs 826ms ✓ Successful request Iteration 43/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 5ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 160µs (cache) (cache) 3ms 989µs 26µs 5ms ✓ Successful request Iteration 44/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=-1 200 OK ★ 815ms time ★ 306B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 927µs 151µs (cache) (cache) 813ms 1ms 27µs 815ms ✓ Successful request Iteration 45/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 5ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 166µs (cache) (cache) 3ms 994µs 26µs 5ms ✓ Successful request Iteration 46/82 → management request POST http://localhost:8081/models?url=resnet-18.mar&model_name=resnet-18&synchronous=false 200 OK ★ 828ms time ★ 309B↑ 405B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 138B │ { │ "status": "Model \"resnet-18\" Version: 1.0 register │ ed with 0 initial workers. Use scale workers API to ad │ d workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 171µs (cache) (cache) 826ms 875µs 27µs 829ms ✓ Successful request Iteration 47/82 → management request GET http://localhost:8081/models?limit=1 200 OK ★ 3ms time ★ 240B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 172µs (cache) (cache) 1ms 936µs 26µs 3ms ✓ Successful request Iteration 48/82 → management request GET http://localhost:8081/models?limit=-1 200 OK ★ 4ms time ★ 241B↑ 367B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 100B │ { │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 186µs (cache) (cache) 1ms 1ms 29µs 4ms ✓ Successful request Iteration 49/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=1 200 OK ★ 3ms time ★ 258B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 185µs (cache) (cache) 1ms 1ms 26µs 3ms ✓ Successful request Iteration 50/82 → management request GET http://localhost:8081/models?limit=1&next_page_token=-1 200 OK ★ 3ms time ★ 259B↑ 391B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 124B │ { │ "nextPageToken": "1", │ "models": [ │ { │ "modelName": "resnet-18", │ "modelUrl": "resnet-18.mar" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 179µs (cache) (cache) 1ms 908µs 27µs 4ms ✓ Successful request Iteration 51/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=10 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 171µs (cache) (cache) 1ms 1ms 37µs 4ms ✓ Successful request Iteration 52/82 → management request PUT http://localhost:8081/models/resnet-18?number_gpu=-1 202 Accepted ★ 3ms time ★ 275B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 168µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 53/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=true 200 OK ★ 3ms time ★ 304B↑ 325B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 59B │ { │ "status": "Workers scaled to 1 for model: resnet-18" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 940µs 156µs (cache) (cache) 1ms 954µs 26µs 3ms ✓ Successful request Iteration 54/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&max_worker=1&synchronous=false 202 Accepted ★ 3ms time ★ 305B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 166µs (cache) (cache) 1ms 928µs 26µs 3ms ✓ Successful request Iteration 55/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=-1 202 Accepted ★ 3ms time ★ 272B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 935µs 158µs (cache) (cache) 1ms 918µs 26µs 3ms ✓ Successful request Iteration 56/82 → management request PUT http://localhost:8081/models/resnet-18?timeout=0 202 Accepted ★ 3ms time ★ 271B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 887µs 152µs (cache) (cache) 1ms 966µs 27µs 3ms ✓ Successful request Iteration 57/82 → management request POST http://localhost:8081/models?url=&model_name=resnet-18 404 Not Found ★ 3ms time ★ 278B↑ 348B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 80B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "empty url" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 872µs 153µs (cache) (cache) 1ms 1ms 27µs 3ms ✓ Successful request Iteration 58/82 → management request POST http://localhost:8081/models?url=https://torchserve.pytorch.org/mar_files/invalid-resnet-18.mar&model_name=invalid-resnet18 400 Bad Request ★ 863ms time ★ 347B↑ 439B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 168B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.pytorch.org/mar_files/invalid-resnet-18.ma │ r" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 656µs 198µs 199µs 859ms 1ms 27µs 863ms ✓ Successful request Iteration 59/82 → management request GET http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 11ms time ★ 254B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 387µs 118µs 211µs 9ms 986µs 28µs 12ms ✓ Successful request Iteration 60/82 → management request GET http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 14ms time ★ 250B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 525µs 156µs 222µs 11ms 1ms 31µs 14ms ✓ Successful request Iteration 61/82 → management request GET http://localhost:8081/models?next_page_token=12 200 OK ★ 13ms time ★ 251B↑ 285B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 19B │ { │ "models": [] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 507µs 156µs 258µs 10ms 1ms 31µs 13ms ✓ Successful request Iteration 62/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=Nan 202 Accepted ★ 3ms time ★ 290B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 907µs 159µs (cache) (cache) 1ms 1ms 30µs 3ms ✓ Successful request Iteration 63/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=nan&synchronous=nan 202 Accepted ★ 3ms time ★ 292B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 879µs 150µs (cache) (cache) 1ms 896µs 25µs 3ms ✓ Successful request Iteration 64/82 → management request PUT http://localhost:8081/models/resnet-18 202 Accepted ★ 4ms time ★ 261B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 957µs 191µs (cache) (cache) 1ms 1ms 28µs 4ms ✓ Successful request Iteration 65/82 → management request PUT http://localhost:8081/models/resnet181?min_worker=1 404 Not Found ★ 3ms time ★ 274B↑ 365B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 97B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: resnet181" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 880µs 153µs (cache) (cache) 1ms 908µs 27µs 2ms ✓ Successful request Iteration 66/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=2&max_worker=1 400 Bad Request ★ 12ms time ★ 287B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 439µs 130µs 190µs 9ms 1ms 28µs 14ms ✓ Successful request Iteration 67/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=1 202 Accepted ★ 12ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 498µs 148µs 234µs 9ms 1ms 29µs 13ms ✓ Successful request Iteration 68/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=0 202 Accepted ★ 13ms time ★ 274B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 928µs 167µs (cache) (cache) 11ms 1ms 28µs 13ms ✓ Successful request Iteration 69/82 → management request PUT http://localhost:8081/models/resnet-18?min_worker=-1 500 Internal Server Error ★ 5ms time ★ 275B↑ 390B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 500, │ "type": "IndexOutOfBoundsException", │ "message": "Index -1 out of bounds for length 0" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 248µs (cache) (cache) 2ms 1ms 31µs 5ms ✓ Successful request Iteration 70/82 → management request PUT http://localhost:8081/models/resnet-18?max_worker=-1 400 Bad Request ★ 14ms time ★ 275B↑ 381B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 110B │ { │ "code": 400, │ "type": "BadRequestException", │ "message": "max_worker cannot be less than min_worke │ r." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 575µs 171µs 431µs 11ms 1ms 30µs 15ms ✓ Successful request Iteration 71/82 → management request PUT http://localhost:8081/models/invalid_squeezenet1_1/1.0/set-default 404 Not Found ★ 14ms time ★ 289B↑ 378B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 897µs 424µs 130µs 204µs 9ms 3ms 27µs 14ms ✓ Successful request Iteration 72/82 → management request DELETE http://localhost:8081/models/resnet-18 200 OK ★ 6ms time ★ 245B↑ 317B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 51B │ { │ "status": "Model \"resnet-18\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 989µs 466µs 134µs 201µs 4ms 941µs 26µs 6ms ✓ Successful request Iteration 73/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/0.0 404 Not Found ★ 3ms time ★ 253B↑ 370B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 101B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 872µs 157µs (cache) (cache) 1ms 999µs 28µs 3ms ✓ Successful request Iteration 74/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 102ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 465µs 137µs 221µs 99ms 795µs 20µs 102ms ✓ Successful request Iteration 75/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=true 200 OK ★ 5ms time ★ 267B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 922µs 171µs (cache) (cache) 3ms 962µs 26µs 5ms ✓ Successful request Iteration 76/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 98ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 176µs (cache) (cache) 96ms 782µs 21µs 98ms ✓ Successful request Iteration 77/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?synchronous=nan 200 OK ★ 5ms time ★ 266B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 159µs (cache) (cache) 3ms 996µs 28µs 5ms ✓ Successful request Iteration 78/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 100ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 206µs (cache) (cache) 98ms 773µs 21µs 101ms ✓ Successful request Iteration 79/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true 200 OK ★ 5ms time ★ 263B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 866µs 140µs (cache) (cache) 3ms 950µs 27µs 5ms ✓ Successful request Iteration 80/82 → management request POST http://localhost:8081/models?url=squeezenet1_1.mar&model_name=squeezenet1_1 200 OK ★ 98ms time ★ 299B↑ 409B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 142B │ { │ "status": "Model \"squeezenet1_1\" Version: 1.0 regi │ stered with 0 initial workers. Use scale workers API t │ o add workers for the model." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 958µs 164µs (cache) (cache) 96ms 814µs 25µs 98ms ✓ Successful request Iteration 81/82 → management request DELETE http://localhost:8081/models/squeezenet1_1/?timeout=true&synchronous=-1 200 OK ★ 6ms time ★ 278B↑ 321B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Model \"squeezenet1_1\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 204µs (cache) (cache) 4ms 1ms 29µs 7ms ✓ Successful request Iteration 82/82 → management request DELETE http://localhost:8081/models/invalid_squeezenet1_1 404 Not Found ★ 3ms time ★ 257B↑ 378B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 109B │ { │ "code": 404, │ "type": "ModelNotFoundException", │ "message": "Model not found: invalid_squeezenet1_1" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 917µs 161µs (cache) (cache) 1ms 949µs 27µs 3ms ✓ Successful request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 82 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 82 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 29.2s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 7.75kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 338ms [min: 3ms, max: 7.1s, s.d.: 1206ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 710µs [min: 118µs, max: 1ms, s.d.: 625µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 336ms [min: 1ms, max: 7.1s, s.d.: 1206ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman kf_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 31ms 5ms 1ms 309µs 1996ms 5ms 319µs 2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v1/models/mnist:predict 200 OK ★ 32ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 6ms 1ms 254µs 427µs 28ms 1ms 51µs 38ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v1/models/mnist:explain 200 OK ★ 73ms time ★ 680B↑ 25.39kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.15kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.004570948738580575, │ 0.006216969354192105, │ 0.008197565590552321, │ 0.009563574614432992, │ 0.008999274562253209, │ 0.009673474929186786, │ 0.0075999051701512495, │ 0.006361380888646387, │ 0.005768828652797567, │ 0.004394465856527532, │ 0.004948218006895951, │ 0.005273460550333246, │ 0.005523799719808455, │ 0.007789356414668216, │ 0.008759363342655755, │ 0.00430480516324295, │ 0.010970579130997414, │ 0.0032484391606445308, │ 0.0059980330139177435, │ 0.0037543659205210495, │ 0.002765290850626197, │ 0.004314086250589837, │ 0.0014008573838089065, │ 0.004841846519426633, │ 0.00063748405852181, │ 0.0018558539683395966, │ -0.0008280458164171088, │ -0.0 │ ], │ [ │ 0.0016625159121928454, │ 0.00444379180663651, │ 0.01238784029399793, │ 0.009450843952942047, │ 0.016143820706701874, │ 0.007797502042012509, │ 0.013942240780593617, │ 0.007557430476889445, │ 0.005479090571418102, │ 0.009751320337132724, │ 0.004764407385073551, │ 0.007292147658128949, │ 0.01179799937538059, │ 0.006462684558412935, │ 0.003383214208849701, │ 0.009225058866604059, │ 0.001675070306666094, │ 0.0073623047131398675, │ 0.005270058076607665, │ 0.005453598185890411, │ 0.004342725772494382, │ 0.005856133228586909, │ 0.012168384676170195, │ 0.009902719402082216, │ 0.009813113253425626, │ 0.003442758203666522, │ 0.0022924284776334007, │ -0.0 │ ], │ [ │ 0.011528255254417058, │ 0.00914698931587443, │ 0.02226541986175408, │ 0.017558217234640273, │ 0.02477055450087839, │ 0.021412693644484452, │ │ (showing 2.05kB/25.15kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 462µs (cache) (cache) 70ms 1ms 92µs 74ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 21ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 295µs (cache) (cache) 18ms 1ms 50µs 21ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 17ms time ★ 233B↑ 1.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 755B │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="aeb37b31-c5f2-4f6b │ -b346-59333156b1b6",model_name="mnist",model_version=" │ default",} 219.874 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="aeb37b31-c5f2- │ 4f6b-b346-59333156b1b6",model_name="mnist",model_versi │ on="default",} 78238.188 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="aeb37b31-c5f2-4f6b-b │ 346-59333156b1b6",model_name="mnist",model_version="de │ fault",} 2.0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 683µs 186µs 255µs 13ms 1ms 59µs 17ms ✓ Successful GET request ┌─────────────────────────┬──────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴──────────────────────┴─────────────────────┤ │ total run duration: 2.3s │ ├──────────────────────────────────────────────────────────────────────┤ │ total data received: 26.06kB (approx) │ ├──────────────────────────────────────────────────────────────────────┤ │ average response time: 430ms [min: 17ms, max: 2s, s.d.: 789ms] │ ├──────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 720µs [min: 186µs, max: 1ms, s.d.: 598µs] │ ├──────────────────────────────────────────────────────────────────────┤ │ average first byte time: 425ms [min: 13ms, max: 1996ms, s.d.: 785ms] │ └──────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman kf_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 144ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 32ms 7ms 1ms 556µs 102ms 26ms 5ms 341µs 176ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 45ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 1ms 172µs 367µs 28ms 12ms 1ms 56µs 45ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 33ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.0" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 935µs 156µs 402µs 26ms 3ms 1ms 59µs 33ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 1948ms time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 362µs (cache) (cache) (cache) 1945ms 1ms 64µs 1948ms ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 7ms time ★ 238B↑ 737B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 470B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-03-22T12:18:33.441Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 50552, │ "gpu": false, │ "gpuUsage": "N/A" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 263µs (cache) (cache) (cache) 4ms 1ms 53µs 7ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 2s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 300µs (cache) (cache) (cache) 2s 1ms 42µs 2s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 5ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 298µs (cache) (cache) (cache) 2ms 1ms 53µs 6ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v1/models/mnist:predict 200 OK ★ 21ms time ★ 680B↑ 266B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 32B │ { │ "predictions": [ │ 2 │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 5ms 623µs (cache) (cache) (cache) 18ms 1ms 38µs 25ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v1/models/mnist:explain 200 OK ★ 52ms time ★ 680B↑ 25.39kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 409B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 25.15kB │ { │ "explanations": [ │ [ │ [ │ [ │ 0.004570948738580575, │ 0.006216969354192105, │ 0.008197565590552321, │ 0.009563574614432992, │ 0.008999274562253209, │ 0.009673474929186786, │ 0.0075999051701512495, │ 0.006361380888646387, │ 0.005768828652797567, │ 0.004394465856527532, │ 0.004948218006895951, │ 0.005273460550333246, │ 0.005523799719808455, │ 0.007789356414668216, │ 0.008759363342655755, │ 0.00430480516324295, │ 0.010970579130997414, │ 0.0032484391606445308, │ 0.0059980330139177435, │ 0.0037543659205210495, │ 0.002765290850626197, │ 0.004314086250589837, │ 0.0014008573838089065, │ 0.004841846519426633, │ 0.00063748405852181, │ 0.0018558539683395966, │ -0.0008280458164171088, │ -0.0 │ ], │ [ │ 0.0016625159121928454, │ 0.00444379180663651, │ 0.01238784029399793, │ 0.009450843952942047, │ 0.016143820706701874, │ 0.007797502042012509, │ 0.013942240780593617, │ 0.007557430476889445, │ 0.005479090571418102, │ 0.009751320337132724, │ 0.004764407385073551, │ 0.007292147658128949, │ 0.01179799937538059, │ 0.006462684558412935, │ 0.003383214208849701, │ 0.009225058866604059, │ 0.001675070306666094, │ 0.0073623047131398675, │ 0.005270058076607665, │ 0.005453598185890411, │ 0.004342725772494382, │ 0.005856133228586909, │ 0.012168384676170195, │ 0.009902719402082216, │ 0.009813113253425626, │ 0.003442758203666522, │ 0.0022924284776334007, │ -0.0 │ ], │ [ │ 0.011528255254417058, │ 0.00914698931587443, │ 0.02226541986175408, │ 0.017558217234640273, │ 0.02477055450087839, │ 0.021412693644484452, │ │ (showing 2.05kB/25.15kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 277µs (cache) (cache) (cache) 49ms 1ms 40µs 53ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 85ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 262µs (cache) (cache) (cache) 82ms 1ms 33µs 85ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 4.7s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 110.41kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 438ms [min: 5ms, max: 2s, s.d.: 780ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 510µs [min: 156µs, max: 1ms, s.d.: 490µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 418ms [min: 2ms, max: 2s, s.d.: 788ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman kfv2_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 32ms 5ms 1ms 336µs 2s 6ms 394µs 2s ✓ Successful POST request → Model Zoo - Inference Model POST http://localhost:8080/v2/models/mnist/infer 200 OK ★ 29ms time ★ 7kB↑ 473B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 238B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "predict", │ "shape": [], │ "datatype": "INT64", │ "data": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 1ms 460µs 205µs 25ms 1ms 56µs 34ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Explanations Model POST http://localhost:8080/v2/models/mnist/explain 200 OK ★ 42ms time ★ 7kB↑ 14.07kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.83kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "explain", │ "shape": [ │ 1, │ 28, │ 28 │ ], │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004054752831520071, │ -0.0002261287725670648, │ -0.0001273413024972421, │ 0.005648369417700337, │ │ (showing 2.05kB/13.83kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 420µs (cache) (cache) 40ms 1ms 74µs 43ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 20ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 302µs (cache) (cache) 18ms 1ms 51µs 21ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 17ms time ★ 233B↑ 1.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 754B │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="69439331-0eb8-4300 │ -ac85-cb6bc59d73dd",model_name="mnist",model_version=" │ default",} 215.36 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="69439331-0eb8- │ 4300-ac85-cb6bc59d73dd",model_name="mnist",model_versi │ on="default",} 44362.998 │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="69439331-0eb8-4300-a │ c85-cb6bc59d73dd",model_name="mnist",model_version="de │ fault",} 2.0 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 679µs 184µs 246µs 14ms 1ms 63µs 17ms ✓ Successful GET request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 5 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 7 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 2.3s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 14.95kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 424ms [min: 17ms, max: 2s, s.d.: 795ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 801µs [min: 184µs, max: 1ms, s.d.: 540µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 419ms [min: 14ms, max: 2s, s.d.: 790ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. Current date and time : 2023-03-22 11:55:24 ------------------------------------------------------------------------------------------ Environment headers ------------------------------------------------------------------------------------------ Torchserve branch: integration torchserve==0.7.0 torch-model-archiver==0.7.0 Python version: 3.8 (64-bit runtime) Python executable: /Users/lninga/opt/anaconda3/envs/py38/bin/python3 Versions of relevant python libraries: captum==0.6.0 numpy==1.24.2 nvgpu==0.9.0 psutil==5.9.4 pygit2==1.11.1 pylint==2.6.0 pytest==7.2.2 pytest-cov==4.0.0 pytest-mock==3.10.0 requests==2.28.2 requests-toolbelt==0.9.1 torch==1.13.1 torch-model-archiver==0.7.0b20230322 torch-workflow-archiver==0.2.6b20230322 torchaudio==0.13.1 torchserve==0.7.0b20230322 torchserve-nightly==2022.11.14 torchtext==0.14.1 torchvision==0.14.1 transformers==4.25.1 wheel==0.40.0 torch==1.13.1 torchtext==0.14.1 torchvision==0.14.1 torchaudio==0.13.1 Java Version: OS: Mac OSX 13.2.1 (x86_64) GCC version: N/A Clang version: 12.0.0 (clang-1200.0.32.2) CMake version: version 3.19.2 Versions of npm installed packages: markdown-link-check@3.10.3 UNMET PEER DEPENDENCY newman@5.3.2 newman-reporter-html@1.0.5 ## In directory /Volumes/workplace/python_env/serve | Executing command pip install --force-reinstall . ## In directory /Volumes/workplace/python_env/serve | Executing command pip install --force-reinstall model-archiver/. ## In directory /Volumes/workplace/python_env/serve | Executing command pip install --force-reinstall workflow-archiver/. ## Starting generate_mars, mar_config:/Volumes/workplace/python_env/serve/ts_scripts/../ts_scripts/mar_config.json, model_store_dir:/Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name fastrcnn --version 1.0 --model-file examples/object_detector/fast-rcnn/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## fastrcnn.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name alexnet --version 1.0 --model-file examples/image_classifier/alexnet/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet-owt-7be5be79.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## alexnet.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name densenet161 --version 1.0 --model-file examples/image_classifier/densenet_161/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161-8d451a50.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## densenet161.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name mnist --version 1.0 --model-file examples/image_classifier/mnist/mnist.py --serialized-file examples/image_classifier/mnist/mnist_cnn.pt --handler examples/image_classifier/mnist/mnist_handler.py --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## mnist.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name resnet-152-batch --version 1.0 --model-file examples/image_classifier/resnet_152_batch/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet152-394f9c45.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## resnet-152-batch.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name resnet-18 --version 1.0 --model-file examples/image_classifier/resnet_18/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet18-f37072fd.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## resnet-18.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name squeezenet1_1 --version 1.0 --model-file examples/image_classifier/squeezenet/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1-b8a52dc0.pth --handler image_classifier --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## squeezenet1_1.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name vgg16 --version 1.0 --model-file examples/image_classifier/vgg_16/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16-397923af.pth --handler examples/image_classifier/vgg_16/vgg_handler.py --extra-files examples/image_classifier/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## vgg16.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name deeplabv3_resnet_101_eager --version 1.0 --model-file examples/image_segmenter/deeplabv3/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet101_coco-586e9e4e.pth --handler image_segmenter --extra-files examples/image_segmenter/deeplabv3/deeplabv3.py,examples/image_segmenter/deeplabv3/intermediate_layer_getter.py,examples/image_segmenter/deeplabv3/fcn.py --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## deeplabv3_resnet_101_eager.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name fcn_resnet_101 --version 1.0 --model-file examples/image_segmenter/fcn/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet101_coco-7ecb50ca.pth --handler image_segmenter --extra-files examples/image_segmenter/fcn/fcn.py,examples/image_segmenter/fcn/intermediate_layer_getter.py --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## fcn_resnet_101.mar is generated. ## In directory: /Volumes/workplace/python_env/serve | Executing command: torch-model-archiver --model-name maskrcnn --version 1.0 --model-file examples/object_detector/maskrcnn/model.py --serialized-file /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth --handler object_detector --extra-files examples/object_detector/index_to_name.json --export-path /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen --force ## maskrcnn.mar is generated. ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman kfv2_https_test_collection → HTTPS Inference API Description OPTIONS https://localhost:8443 200 OK ★ 183ms time ★ 230B↑ 23.67kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 23.41kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/ping": { │ │ (showing 2.05kB/23.41kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 33ms 7ms 1ms 709µs 133ms 32ms 6ms 415µs 217ms ✓ Status code is 200 → HTTPS Management API Description OPTIONS https://localhost:8444 200 OK ★ 50ms time ★ 230B↑ 58.85kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 58.58kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.0" │ }, │ "paths": { │ "/": { │ "options": { │ "description": "Get openapi description.", │ "operationId": "apiDescription", │ "parameters": [], │ "responses": { │ "200": { │ "description": "A openapi 3.0.1 descriptor │ ", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "openapi", │ "info", │ "paths" │ ], │ "properties": { │ "openapi": { │ "type": "string" │ }, │ "info": { │ "type": "object" │ }, │ "paths": { │ "type": "object" │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ "type": "integer", │ "description": "Error code." │ }, │ "type": { │ "type": "string", │ "description": "Error type." │ }, │ "message": { │ "type": "string", │ "description": "Error message." │ } │ } │ } │ } │ } │ } │ } │ } │ }, │ "/models": { │ │ (showing 2.05kB/58.58kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 1ms 200µs 410µs 31ms 13ms 2ms 74µs 50ms ✓ Status code is 200 → HTTPS Metrics API Description OPTIONS https://localhost:8445 200 OK ★ 35ms time ★ 230B↑ 2.82kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 2.55kB │ { │ "openapi": "3.0.1", │ "info": { │ "title": "TorchServe APIs", │ "description": "TorchServe is a flexible and easy │ to use tool for serving deep learning models", │ "version": "0.7.0" │ }, │ "paths": { │ "/metrics": { │ "get": { │ "description": "Get TorchServe application met │ rics in prometheus format.", │ "operationId": "metrics", │ "parameters": [ │ { │ "in": "query", │ "name": "name[]", │ "description": "Names of metrics to filter │ ", │ "required": false, │ "schema": { │ "type": "string" │ } │ } │ ], │ "responses": { │ "200": { │ "description": "TorchServe application met │ rics", │ "content": { │ "text/plain; version=0.0.4; charset=utf- │ 8": { │ "schema": { │ "type": "object", │ "required": [ │ "# HELP", │ "# TYPE", │ "metric" │ ], │ "properties": { │ "# HELP": { │ "type": "string", │ "description": "Help text for To │ rchServe metric." │ }, │ "# TYPE": { │ "type": "string", │ "description": "Type of TorchSer │ ve metric." │ }, │ "metric": { │ "type": "string", │ "description": "TorchServe appli │ cation metric." │ } │ } │ } │ } │ } │ }, │ "500": { │ "description": "Internal Server Error", │ "content": { │ "application/json": { │ "schema": { │ "type": "object", │ "required": [ │ "code", │ "type", │ "message" │ ], │ "properties": { │ "code": { │ │ (showing 2.05kB/2.55kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 917µs 158µs 360µs 28ms 3ms 1ms 59µs 36ms ✓ Status code is 200 → HTTPS Register Model - Mnist POST https://localhost:8444/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 395µs (cache) (cache) (cache) 2s 1ms 69µs 2s ✓ Successful POST request → HTTPS Get Mnist Model Description GET https://localhost:8444/models/mnist 200 OK ★ 6ms time ★ 238B↑ 737B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 470B │ [ │ { │ "modelName": "mnist", │ "modelVersion": "1.0", │ "modelUrl": "mnist.mar", │ "runtime": "python", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 100, │ "loadedAtStartup": false, │ "workers": [ │ { │ "id": "9000", │ "startTime": "2023-03-22T12:19:22.545Z", │ "status": "READY", │ "memoryUsage": 0, │ "pid": 50857, │ "gpu": false, │ "gpuUsage": "N/A" │ } │ ] │ } │ ] └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 312µs (cache) (cache) (cache) 4ms 1ms 49µs 7ms ✓ Successful GET request → HTTPS Scale up Workers - Synchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=5&max_worker=5&synchronous=true 200 OK ★ 2.1s time ★ 300B↑ 321B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 55B │ { │ "status": "Workers scaled to 5 for model: mnist" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 285µs (cache) (cache) (cache) 2s 1ms 49µs 2.1s ✓ Successful PUT request → HTTPS Scale up Workers - Asynchronous for Mnist PUT https://localhost:8444/models/mnist?min_worker=6&max_worker=6&synchronous=false 202 Accepted ★ 5ms time ★ 301B↑ 319B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Processing worker updates..." │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 311µs (cache) (cache) (cache) 2ms 1ms 52µs 5ms ✓ Successful PUT request → HTTPS - Inference - Mnist_KF POST https://localhost:8443/v2/models/mnist/infer 200 OK ★ 18ms time ★ 7kB↑ 473B↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 238B │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "predict", │ "shape": [], │ "datatype": "INT64", │ "data": [ │ 1 │ ] │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 5ms 611µs (cache) (cache) (cache) 15ms 1ms 40µs 22ms ✓ Status code is 200 → HTTPS - Explanations - Mnist_KF POST https://localhost:8443/v2/models/mnist/explain 200 OK ★ 61ms time ★ 7kB↑ 14.07kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 6.73kB │ (showing 2.05kB/6.73kB) └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 13.83kB │ { │ "id": "d3b15cad-50a2-4eaf-80ce-8b0a428bd298", │ "model_name": "mnist", │ "model_version": "1.0", │ "outputs": [ │ { │ "name": "explain", │ "shape": [ │ 1, │ 28, │ 28 │ ], │ "datatype": "FP64", │ "data": [ │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ -0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.0, │ -0.0, │ -0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ 0.0, │ -0.0, │ -0.004054752831520071, │ -0.0002261287725670648, │ -0.0001273413024972421, │ 0.005648369417700337, │ │ (showing 2.05kB/13.83kB) └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 2ms 290µs (cache) (cache) (cache) 59ms 1ms 38µs 62ms ✓ Status code is 200 → HTTPS UnRegister Model Mnist DELETE https://localhost:8444/models/mnist 200 OK ★ 86ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake ssl-handshake transfer-start download process total 1ms 290µs (cache) (cache) (cache) 84ms 1ms 35µs 87ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 10 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 10 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 4.9s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 99.3kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 462ms [min: 5ms, max: 2.1s, s.d.: 816ms] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 562µs [min: 158µs, max: 1ms, s.d.: 541µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 439ms [min: 2ms, max: 2s, s.d.: 825ms] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman explanation_api_test_collection → Model Zoo - Register Model POST http://localhost:8081/models?url=mnist.mar&model_name=mnist&initial_workers=1&synchronous=true 200 OK ★ 2s time ★ 318B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Model \"mnist\" Version: 1.0 registered w │ ith 1 initial workers" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 31ms 5ms 1ms 302µs 1997ms 6ms 370µs 2s ✓ Successful POST request → Model Zoo - Explanations Model POST http://localhost:8080/explanations/mnist 200 OK ★ 65ms time ★ 538B↑ 23.21kB↓ size ★ 8↑ 6↓ headers ★ 0 cookies ┌ ↑ file ★ 272B │ └ ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 22.97kB │ [ │ [ │ [ │ -0.00039919446961835236, │ -0.00019002183512235264, │ -0.0008597190403451449, │ -0.0003293672081608754, │ -0.0009114924467526127, │ -0.00017816114351215542, │ -0.0005801028758730413, │ -5.7529225598252194e-05, │ -0.00013036391396408506, │ -3.622336897593853e-05, │ 1.2628194609278315e-05, │ -3.1516328135878544e-05, │ -7.058358081496025e-05, │ -6.872769973604391e-05, │ -7.877429140005835e-05, │ -0.0001330244624280486, │ -3.9255966485147735e-05, │ -0.00022929553875323679, │ 1.713108952478634e-05, │ -0.0003846352858902638, │ -0.0005095514386928224, │ -0.00039880105508864643, │ -0.0008459617489628804, │ -0.00050347912400287, │ -0.0006029825256197108, │ -0.0001768611591953732, │ -0.00015337618628128994, │ -0.0 │ ], │ [ │ -0.000459367041736051, │ -0.0006972018770355594, │ -0.0006564253391229728, │ -0.0006832953398625042, │ -0.0004380070958294335, │ -0.0006188232145845846, │ 8.717674968821532e-06, │ -0.0003930582389242723, │ -0.00010256507980626367, │ -0.00018236528139309367, │ -0.0005491941234231536, │ 4.5198500755516835e-05, │ -0.0004397756033804491, │ -0.00032800121287585753, │ -0.0003846252248742354, │ -0.0007280317477269334, │ -0.0003807700215901187, │ -0.0006797484384283201, │ -0.0009410585867905978, │ -0.0005321376784251184, │ -0.0010959784640141236, │ -0.0009239369515597444, │ -0.0007882725965909251, │ -0.000763552145658654, │ -0.0003335173411764801, │ 5.101228322605105e-05, │ 7.699817024785627e-05, │ -0.0 │ ], │ [ │ -0.0013165055828473268, │ -0.0009519237154533972, │ -0.002100062134240652, │ -0.0007986434565747188, │ -0.0015434154623847722, │ -0.0008677095825202944, │ -0.0006676295763002351, │ -0.0010158373275630642, │ -0.0006892678395626808, │ -0.0010906412181402913, │ -0.0009184818011024204, │ -0.0010186683768814075, │ │ (showing 2.05kB/22.97kB) └ prepare wait dns-lookup tcp-handshake transfer-start download process total 5ms 1ms 256µs 470µs 60ms 1ms 55µs 69ms ✓ Successful POST request ✓ Test expected JSON response → Model Zoo - Unregister model DELETE http://localhost:8081/models/mnist 200 OK ★ 21ms time ★ 241B↑ 313B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 47B │ { │ "status": "Model \"mnist\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 438µs (cache) (cache) 18ms 1ms 85µs 21ms ✓ Successful DELETE request → Model Zoo - Model Metrics GET http://localhost:8082/metrics 200 OK ★ 17ms time ★ 233B↑ 1.05kB↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ text/plain ★ text ★ plain ★ utf8 ★ 755B │ # HELP ts_inference_requests_total Total number of inf │ erence requests. │ # TYPE ts_inference_requests_total counter │ ts_inference_requests_total{uuid="16d2bf9f-a70a-4c02-9 │ 7d0-229a90ee0989",model_name="mnist",model_version="de │ fault",} 1.0 │ # HELP ts_queue_latency_microseconds Cumulative queue │ duration in microseconds │ # TYPE ts_queue_latency_microseconds counter │ ts_queue_latency_microseconds{uuid="16d2bf9f-a70a-4c02 │ -97d0-229a90ee0989",model_name="mnist",model_version=" │ default",} 135.169 │ # HELP ts_inference_latency_microseconds Cumulative in │ ference duration in microseconds │ # TYPE ts_inference_latency_microseconds counter │ ts_inference_latency_microseconds{uuid="16d2bf9f-a70a- │ 4c02-97d0-229a90ee0989",model_name="mnist",model_versi │ on="default",} 43473.082 └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 631µs 174µs 254µs 14ms 1ms 60µs 18ms ✓ Successful GET request ┌─────────────────────────┬──────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ iterations │ 1 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ requests │ 4 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ test-scripts │ 4 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼──────────────────────┼─────────────────────┤ │ assertions │ 5 │ 0 │ ├─────────────────────────┴──────────────────────┴─────────────────────┤ │ total run duration: 2.3s │ ├──────────────────────────────────────────────────────────────────────┤ │ total data received: 23.85kB (approx) │ ├──────────────────────────────────────────────────────────────────────┤ │ average response time: 528ms [min: 17ms, max: 2s, s.d.: 856ms] │ ├──────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 678µs [min: 174µs, max: 1ms, s.d.: 587µs] │ ├──────────────────────────────────────────────────────────────────────┤ │ average first byte time: 522ms [min: 14ms, max: 1997ms, s.d.: 851ms] │ └──────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman management_api_collection Iteration 1/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 200 OK ★ 23.1s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 32ms 5ms 1ms 293µs 23.1s 7ms 449µs 23.2s ✓ Successful request Iteration 2/11 → workflow management request GET http://localhost:8081/workflows 200 OK ★ 7ms time ★ 235B↑ 423B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 156B │ { │ "workflows": [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws. │ com/war_files/densenet_wf.war" │ } │ ] │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 616µs (cache) (cache) 4ms 1ms 68µs 8ms ✓ Successful request Iteration 3/11 → workflow management request GET http://localhost:8081/workflows/densenet 200 OK ★ 5ms time ★ 244B↑ 559B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 292B │ [ │ { │ "workflowName": "densenet", │ "workflowUrl": "https://torchserve.s3.amazonaws.co │ m/war_files/densenet_wf.war", │ "minWorkers": 1, │ "maxWorkers": 1, │ "batchSize": 1, │ "maxBatchDelay": 50, │ "workflowDag": "{pre_processing=[densenet], densen │ et=[post_processing]}" │ } │ ] └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 785µs (cache) (cache) 2ms 1ms 60µs 6ms ✓ Successful request Iteration 4/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 500 Internal Server Error ★ 6ms time ★ 321B↑ 370B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 90B │ { │ "code": 500, │ "type": "FileAlreadyExistsException", │ "message": "densenet_wf.war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 304µs (cache) (cache) 3ms 1ms 56µs 7ms ✓ Successful request Iteration 5/11 → workflow management request DELETE http://localhost:8081/workflows/densenet 200 OK ★ 42ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 675µs 203µs 240µs 37ms 2ms 80µs 42ms ✓ Successful request Iteration 6/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/does_not_exist.war 400 Bad Request ★ 375ms time ★ 324B↑ 441B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 170B │ { │ "code": 400, │ "type": "DownloadArchiveException", │ "message": "Failed to download archive from: https:/ │ /torchserve.s3.amazonaws.com/war_files/does_not_exist. │ war" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 236µs (cache) (cache) 372ms 1ms 78µs 375ms ✓ Successful request Iteration 7/11 → workflow management request GET http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 15ms time ★ 250B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 683µs 206µs 277µs 11ms 1ms 49µs 15ms ✓ Successful request Iteration 8/11 → workflow management request DELETE http://localhost:8081/workflows/does_not_exist 404 Not Found ★ 14ms time ★ 253B↑ 377B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 108B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found: does_not_exist" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 687µs 176µs 247µs 11ms 1ms 41µs 15ms ✓ Successful request Iteration 9/11 → workflow management request POST http://localhost:8081/workflows?url=malformed_url,? 404 Not Found ★ 14ms time ★ 275B↑ 396B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 127B │ { │ "code": 404, │ "type": "WorkflowNotFoundException", │ "message": "Workflow not found in workflow store: ma │ lformed_url,?" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 581µs 156µs 253µs 10ms 1ms 38µs 14ms ✓ Successful request Iteration 10/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/custom_python_dep.war 500 Internal Server Error ★ 1381ms time ★ 327B↑ 531B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 250B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow custom_python_dep has failed to │ register. Failures: [Workflow Node custom_python_dep_ │ _custom_python_dep failed to register. Details: Model │ not found at: custom_python_dep.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 541µs 151µs 236µs 1378ms 1ms 34µs 1383ms ✓ Successful request Iteration 11/11 → workflow management request POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/loading-memory-error.war 500 Internal Server Error ★ 1127ms time ★ 330B↑ 543B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 262B │ { │ "code": 500, │ "type": "WorkflowException", │ "message": "Workflow loading-memory-error has failed │ to register. Failures: [Workflow Node loading-memory- │ error__loading-memory-error failed to register. Detail │ s: Model not found at: loading-memory-error.mar]" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 496µs 167µs 216µs 1124ms 1ms 40µs 1127ms ✓ Successful request ┌─────────────────────────┬─────────────────────┬────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ iterations │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ requests │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ test-scripts │ 11 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼────────────────────┤ │ assertions │ 11 │ 0 │ ├─────────────────────────┴─────────────────────┴────────────────────┤ │ total run duration: 26.4s │ ├────────────────────────────────────────────────────────────────────┤ │ total data received: 1.7kB (approx) │ ├────────────────────────────────────────────────────────────────────┤ │ average response time: 2.3s [min: 5ms, max: 23.1s, s.d.: 6.5s] │ ├────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 763µs [min: 151µs, max: 1ms, s.d.: 666µs] │ ├────────────────────────────────────────────────────────────────────┤ │ average first byte time: 2.3s [min: 2ms, max: 23.1s, s.d.: 6.5s] │ └────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. newman: could not find "htmlextra" reporter ensure that the reporter is installed in the same directory as newman please install reporter using npm newman workflow_inference Iteration 1/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/densenet_wf.war 200 OK ★ 24.6s time ★ 321B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 31ms 5ms 1ms 317µs 24.6s 7ms 454µs 24.6s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/densenet 200 OK ★ 468ms time ★ 111.24kB↑ 450B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 183B │ { │ "tabby": 0.46661895513534546, │ "tiger_cat": 0.46449047327041626, │ "Egyptian_cat": 0.06614057719707489, │ "lynx": 0.0012924439506605268, │ "plastic_bag": 0.00022909804829396307 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 6ms 1ms 285µs 342µs 463ms 2ms 54µs 474ms ✓ Successful request ✓ Test expected JSON response → Workflow - Unregister DELETE http://localhost:8081/workflows/densenet 200 OK ★ 35ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 428µs (cache) (cache) 32ms 1ms 90µs 35ms ✓ Successful DELETE request Iteration 2/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/timeout-retry-test.war 200 OK ★ 23.6s time ★ 328B↑ 347B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 81B │ { │ "status": "Workflow densenet has been registered and │ scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 230µs (cache) (cache) 23.6s 1ms 53µs 23.6s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/densenet 500 Internal Server Error ★ 12s time ★ 111.24kB↑ 383B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 110.97kB │ (showing 2.05kB/110.97kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 102B │ { │ "code": 500, │ "type": "InternalServerException", │ "message": "Error executing pre_processing" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 298µs (cache) (cache) 12s 2ms 67µs 12s ✓ Successful request → Workflow - Unregister DELETE http://localhost:8081/workflows/densenet 200 OK ★ 32ms time ★ 247B↑ 319B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 53B │ { │ "status": "Workflow \"densenet\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 270µs (cache) (cache) 29ms 1ms 51µs 33ms ✓ Successful DELETE request Iteration 3/3 → Workflow - Register Workflow POST http://localhost:8081/workflows?url=https://torchserve.s3.amazonaws.com/war_files/dog_breed_wf.war 200 OK ★ 36.3s time ★ 322B↑ 351B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 85B │ { │ "status": "Workflow dog_breed_wf has been registered │ and scaled successfully." │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 257µs (cache) (cache) 36.3s 2ms 45µs 36.3s ✓ Successful POST request → Workflow - Inference POST http://localhost:8080/wfpredict/dog_breed_wf 200 OK ★ 487ms time ★ 77.75kB↑ 473B↓ size ★ 8↑ 7↓ headers ★ 0 cookies ┌ ↑ file ★ 77.47kB │ (showing 2.05kB/77.47kB) └ ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 206B │ { │ "Cardigan_welsh_corgi": 0.6281960606575012, │ "Briard": 0.27462849020957947, │ "Icelandic_sheepdog": 0.07453601807355881, │ "Tibetan_mastiff": 0.015545750968158245, │ "Chow_chow": 0.002062038518488407 │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 2ms 719µs 213µs 240µs 483ms 1ms 54µs 489ms ✓ Successful request ✓ Test expected JSON response → Workflow - Unregister DELETE http://localhost:8081/workflows/dog_breed_wf 200 OK ★ 34ms time ★ 251B↑ 323B↓ size ★ 7↑ 7↓ headers ★ 0 cookies ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 57B │ { │ "status": "Workflow \"dog_breed_wf\" unregistered" │ } └ prepare wait dns-lookup tcp-handshake transfer-start download process total 1ms 225µs (cache) (cache) 30ms 2ms 43µs 34ms ✓ Successful DELETE request ┌─────────────────────────┬─────────────────────┬─────────────────────┐ │ │ executed │ failed │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ iterations │ 3 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ requests │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ test-scripts │ 9 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ prerequest-scripts │ 0 │ 0 │ ├─────────────────────────┼─────────────────────┼─────────────────────┤ │ assertions │ 11 │ 0 │ ├─────────────────────────┴─────────────────────┴─────────────────────┤ │ total run duration: 1m 38s │ ├─────────────────────────────────────────────────────────────────────┤ │ total data received: 901B (approx) │ ├─────────────────────────────────────────────────────────────────────┤ │ average response time: 10.8s [min: 32ms, max: 36.3s, s.d.: 13.2s] │ ├─────────────────────────────────────────────────────────────────────┤ │ average DNS lookup time: 767µs [min: 213µs, max: 1ms, s.d.: 565µs] │ ├─────────────────────────────────────────────────────────────────────┤ │ average first byte time: 10.8s [min: 29ms, max: 36.3s, s.d.: 13.2s] │ └─────────────────────────────────────────────────────────────────────┘ TorchServe has stopped. ============================= test session starts ============================== platform darwin -- Python 3.8.13, pytest-7.2.2, pluggy-1.0.0 -- /Users/lninga/opt/anaconda3/envs/py38/bin/python cachedir: .pytest_cache rootdir: /Volumes/workplace/python_env/serve plugins: mock-3.10.0, cov-4.0.0 collecting ... collected 82 items test_example_dcgan.py::test_model_archive_creation PASSED [ 1%] test_example_dcgan.py::test_model_register_unregister PASSED [ 2%] test_example_dcgan.py::test_image_generation_without_any_input_constraints PASSED [ 3%] test_example_dcgan.py::test_image_generation_with_input_constraints PASSED [ 4%] test_example_intel_extension_for_pytorch.py::test_single_worker_affinity SKIPPED [ 6%] test_example_intel_extension_for_pytorch.py::test_multi_worker_affinity SKIPPED [ 7%] test_example_intel_extension_for_pytorch.py::test_worker_scale_up_affinity SKIPPED [ 8%] test_example_intel_extension_for_pytorch.py::test_worker_scale_down_affinity SKIPPED [ 9%] test_example_scriptable_tokenzier.py::test_handler PASSED [ 10%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_sample_text PASSED [ 12%] test_example_scriptable_tokenzier.py::test_inference_with_untrained_model_and_empty_string PASSED [ 13%] test_example_scriptable_tokenzier.py::test_inference_with_pretrained_model PASSED [ 14%] test_gRPC_inference_api.py::test_inference_apis PASSED [ 15%] test_gRPC_management_apis.py::test_management_apis PASSED [ 17%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model PASSED [ 18%] test_handler.py::test_mnist_model_register_using_non_existent_handler_with_nonzero_workers PASSED [ 19%] test_handler.py::test_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 20%] test_handler.py::test_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 21%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model PASSED [ 23%] test_handler.py::test_kserve_mnist_model_register_scale_inference_with_non_existent_handler PASSED [ 24%] test_handler.py::test_kserve_mnist_model_register_and_inference_on_valid_model_explain PASSED [ 25%] test_handler.py::test_huggingface_bert_batch_inference PASSED [ 26%] test_handler.py::test_MMF_activity_recognition_model_register_and_inference_on_valid_model SKIPPED [ 28%] test_handler.py::test_huggingface_bert_model_parallel_inference PASSED [ 29%] test_metrics.py::test_logs_created PASSED [ 30%] test_metrics.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 31%] test_metrics.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 32%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 34%] test_metrics.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 35%] test_metrics.py::test_log_location_var_snapshot_disabled PASSED [ 36%] test_metrics.py::test_log_location_var_snapshot_enabled PASSED [ 37%] test_metrics.py::test_async_logging PASSED [ 39%] test_metrics.py::test_async_logging_non_boolean PASSED [ 40%] test_metrics.py::test_metrics_location_var_snapshot_disabled PASSED [ 41%] test_metrics.py::test_metrics_location_var_snapshot_enabled PASSED [ 42%] test_metrics.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 43%] test_metrics.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 45%] test_metrics.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 46%] test_metrics_kf.py::test_logs_created PASSED [ 47%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_enabled PASSED [ 48%] test_metrics_kf.py::test_logs_startup_cfg_created_snapshot_disabled PASSED [ 50%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_enabled PASSED [ 51%] test_metrics_kf.py::test_metrics_startup_cfg_created_snapshot_disabled PASSED [ 52%] test_metrics_kf.py::test_log_location_var_snapshot_disabled PASSED [ 53%] test_metrics_kf.py::test_log_location_var_snapshot_enabled PASSED [ 54%] test_metrics_kf.py::test_async_logging PASSED [ 56%] test_metrics_kf.py::test_async_logging_non_boolean PASSED [ 57%] test_metrics_kf.py::test_metrics_location_var_snapshot_disabled PASSED [ 58%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled PASSED [ 59%] test_metrics_kf.py::test_log_location_and_metric_location_vars_snapshot_enabled PASSED [ 60%] test_metrics_kf.py::test_log_location_var_snapshot_disabled_custom_path_read_only PASSED [ 62%] test_metrics_kf.py::test_metrics_location_var_snapshot_enabled_rdonly_dir PASSED [ 63%] test_model_archiver.py::test_multiple_model_versions_registration PASSED [ 64%] test_model_archiver.py::test_duplicate_model_registration_using_local_url_followed_by_http_url PASSED [ 65%] test_model_archiver.py::test_duplicate_model_registration_using_http_url_followed_by_local_url PASSED [ 67%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_without_force PASSED [ 68%] test_model_archiver.py::test_model_archiver_to_regenerate_model_mar_with_force PASSED [ 69%] test_model_archiver.py::test_model_archiver_without_handler_flag PASSED [ 70%] test_model_archiver.py::test_model_archiver_without_model_name_flag PASSED [ 71%] test_model_archiver.py::test_model_archiver_without_model_file_flag PASSED [ 73%] test_model_archiver.py::test_model_archiver_without_serialized_flag PASSED [ 74%] test_onnx.py::test_convert_to_onnx SKIPPED (ONNX is not installed) [ 75%] test_onnx.py::test_model_packaging_and_start SKIPPED (ONNX is not in...) [ 76%] test_onnx.py::test_model_start SKIPPED (ONNX is not installed) [ 78%] test_onnx.py::test_inference SKIPPED (ONNX is not installed) [ 79%] test_onnx.py::test_stop SKIPPED (ONNX is not installed) [ 80%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[/Volumes/workplace/python_env/serve/test/pytest/profiler_utils/resnet_custom.py] PASSED [ 81%] test_pytorch_profiler.py::test_profiler_default_and_custom_handler[image_classifier] PASSED [ 82%] test_pytorch_profiler.py::test_profiler_arguments_override[/Volumes/workplace/python_env/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 84%] test_pytorch_profiler.py::test_batch_input[/Volumes/workplace/python_env/serve/test/pytest/profiler_utils/resnet_profiler_override.py] PASSED [ 85%] test_snapshot.py::test_snapshot_created_on_start_and_stop PASSED [ 86%] test_snapshot.py::test_snapshot_created_on_management_api_invoke PASSED [ 87%] test_snapshot.py::test_start_from_snapshot PASSED [ 89%] test_snapshot.py::test_start_from_latest PASSED [ 90%] test_snapshot.py::test_start_from_read_only_snapshot PASSED [ 91%] test_snapshot.py::test_no_config_snapshots_cli_option PASSED [ 92%] test_snapshot.py::test_start_from_default PASSED [ 93%] test_snapshot.py::test_start_from_non_existing_snapshot PASSED [ 95%] test_snapshot.py::test_torchserve_init_with_non_existent_model_store PASSED [ 96%] test_snapshot.py::test_restart_torchserve_with_last_snapshot_with_model_mar_removed PASSED [ 97%] test_snapshot.py::test_replace_mar_file_with_dummy PASSED [ 98%] test_snapshot.py::test_restart_torchserve_with_one_of_model_mar_removed PASSED [100%] =============================== warnings summary =============================== test/pytest/test_example_scriptable_tokenzier.py::test_handler /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/torch/jit/_recursive.py:262: UserWarning: 'batch_first' was found in ScriptModule constants, but was not actually set in __init__. Consider removing it. warnings.warn("'{}' was found in ScriptModule constants, " test/pytest/test_example_scriptable_tokenzier.py::test_handler /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning) test/pytest/test_example_scriptable_tokenzier.py::test_handler /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) test/pytest/test_example_scriptable_tokenzier.py::test_handler test/pytest/test_example_scriptable_tokenzier.py::test_handler /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) test/pytest/test_example_scriptable_tokenzier.py::test_handler /Users/lninga/opt/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py:1194: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/NestedTensorImpl.cpp:177.) return forward_call(*input, **kwargs) test/pytest/test_example_scriptable_tokenzier.py::test_handler /Volumes/workplace/python_env/serve/test/pytest/../../examples/text_classification_with_scriptable_tokenizer/handler.py:97: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument. data = F.softmax(data) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========== 72 passed, 10 skipped, 7 warnings in 3824.96s (1:03:44) ============ ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kf.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=config.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs --ts-config=resources/config_kfv2.properties >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Starting gen_mar: model_store ## Create symlink for mar files ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/vgg16.mar, model_store/vgg16.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-18.mar, model_store/resnet-18.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/deeplabv3_resnet_101_eager.mar, model_store/deeplabv3_resnet_101_eager.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/maskrcnn.mar, model_store/maskrcnn.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fcn_resnet_101.mar, model_store/fcn_resnet_101.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/mnist.mar, model_store/mnist.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/resnet-152-batch.mar, model_store/resnet-152-batch.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/alexnet.mar, model_store/alexnet.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/squeezenet1_1.mar, model_store/squeezenet1_1.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/densenet161.mar, model_store/densenet161.mar successfully. ## Symlink /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen/fastrcnn.mar, model_store/fastrcnn.mar successfully. ## Starting TorchServe ## Console logs redirected to file: ts_console.log ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --start --model-store=model_store --workflow-store=model_store --ncs >> ts_console.log ## Successfully started TorchServe ## Stopping TorchServe ## In directory: /Volumes/workplace/python_env/serve/test | Executing command: torchserve --stop ## Successfully stopped TorchServe ## Started regression tests ## Started densenet mar creation ## In directory: /private/var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/workspace/model_store | Executing command: torch-model-archiver --model-name densenet161_v1 --version 1.1 --model-file /Volumes/workplace/python_env/serve/ts_scripts/../examples/image_classifier/densenet_161/model.py --serialized-file /var/folders/w6/s5gp9htn2pb9z87lwp6fzjg9hv4nys/T/workspace/model_store/densenet161-8d451a50.pth --extra-files /Volumes/workplace/python_env/serve/ts_scripts/../examples/image_classifier/index_to_name.json --handler image_classifier --force ## Started regression pytests ## In directory: /Volumes/workplace/python_env/serve/test/pytest | Executing command: python -m pytest -v ./ Removing file : inference_pb2_grpc.py Removing file : inference_pb2.py Removing file : management_pb2.py Removing file : management_pb2_grpc.py ## Deleting model_store_gen_dir: /Volumes/workplace/python_env/serve/ts_scripts/../model_store_gen