Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

andy-neuma triggered nightly on refs/heads/upload-to-gcp #169

andy-neuma triggered nightly on refs/heads/upload-to-gcp

andy-neuma triggered nightly on refs/heads/upload-to-gcp #169

Manually triggered June 6, 2024 21:09
Status Failure
Total duration 6h 28m 14s
Artifacts 18

nightly.yml

on: workflow_dispatch
PYTHON-3-10  /  ...  /  BENCHMARK
1h 43m
PYTHON-3-10 / BENCHMARK / BENCHMARK
PYTHON-3-10  /  ...  /  TEST
32m 24s
PYTHON-3-10 / TEST-SOLO / TEST
PYTHON-3-10  /  ...  /  TEST-ACCURACY-SMOKE
8m 20s
PYTHON-3-10 / TEST-ACCURACY-SMOKE / TEST-ACCURACY-SMOKE
PYTHON-3-10  /  ...  /  TEST-ACCURACY-FULL
PYTHON-3-10 / TEST-ACCURACY-FULL / TEST-ACCURACY-FULL
PYTHON-3-11  /  ...  /  BENCHMARK
5h 12m
PYTHON-3-11 / BENCHMARK / BENCHMARK
PYTHON-3-11  /  ...  /  TEST
35m 46s
PYTHON-3-11 / TEST-SOLO / TEST
PYTHON-3-11  /  ...  /  TEST-ACCURACY-SMOKE
10m 20s
PYTHON-3-11 / TEST-ACCURACY-SMOKE / TEST-ACCURACY-SMOKE
PYTHON-3-11  /  ...  /  TEST-ACCURACY-FULL
PYTHON-3-11 / TEST-ACCURACY-FULL / TEST-ACCURACY-FULL
PYTHON-3-8  /  ...  /  BENCHMARK
5h 17m
PYTHON-3-8 / BENCHMARK / BENCHMARK
PYTHON-3-8  /  ...  /  TEST
33m 52s
PYTHON-3-8 / TEST-SOLO / TEST
PYTHON-3-8  /  ...  /  TEST-ACCURACY-SMOKE
9m 29s
PYTHON-3-8 / TEST-ACCURACY-SMOKE / TEST-ACCURACY-SMOKE
PYTHON-3-8  /  ...  /  TEST-ACCURACY-FULL
PYTHON-3-8 / TEST-ACCURACY-FULL / TEST-ACCURACY-FULL
PYTHON-3-9  /  ...  /  BENCHMARK
5h 19m
PYTHON-3-9 / BENCHMARK / BENCHMARK
PYTHON-3-9  /  ...  /  TEST
38m 51s
PYTHON-3-9 / TEST-SOLO / TEST
PYTHON-3-9  /  ...  /  TEST-ACCURACY-SMOKE
11m 9s
PYTHON-3-9 / TEST-ACCURACY-SMOKE / TEST-ACCURACY-SMOKE
PYTHON-3-9  /  ...  /  TEST-ACCURACY-FULL
PYTHON-3-9 / TEST-ACCURACY-FULL / TEST-ACCURACY-FULL
PYTHON-3-10  /  ...  /  BENCHMARK_REPORT
0s
PYTHON-3-10 / BENCHMARK / BENCHMARK_REPORT
PYTHON-3-10  /  ...  /  PUBLISH
1m 21s
PYTHON-3-10 / UPLOAD / PUBLISH
PYTHON-3-11  /  ...  /  BENCHMARK_REPORT
39s
PYTHON-3-11 / BENCHMARK / BENCHMARK_REPORT
PYTHON-3-11  /  ...  /  PUBLISH
1m 19s
PYTHON-3-11 / UPLOAD / PUBLISH
PYTHON-3-8  /  ...  /  BENCHMARK_REPORT
33s
PYTHON-3-8 / BENCHMARK / BENCHMARK_REPORT
PYTHON-3-8  /  ...  /  PUBLISH
1m 20s
PYTHON-3-8 / UPLOAD / PUBLISH
PYTHON-3-9  /  ...  /  BENCHMARK_REPORT
39s
PYTHON-3-9 / BENCHMARK / BENCHMARK_REPORT
PYTHON-3-9  /  ...  /  PUBLISH
1m 18s
PYTHON-3-9 / UPLOAD / PUBLISH
Fit to window
Zoom out
Zoom in

Annotations

8 errors and 4 warnings
PYTHON-3-11 / TEST-ACCURACY-SMOKE / TEST-ACCURACY-SMOKE
Process completed with exit code 1.
PYTHON-3-8 / TEST-ACCURACY-SMOKE / TEST-ACCURACY-SMOKE
Process completed with exit code 1.
PYTHON-3-9 / TEST-ACCURACY-SMOKE / TEST-ACCURACY-SMOKE
Process completed with exit code 1.
PYTHON-3-10 / BENCHMARK / BENCHMARK
System.IO.IOException: No space left on device : '/opt/actions-runner/_diag/Worker_20240606-221047-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset) at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite() at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.Diagnostics.TextWriterTraceListener.Flush() at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id) at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message) at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message) at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut) at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args) System.IO.IOException: No space left on device : '/opt/actions-runner/_diag/Worker_20240606-221047-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset) at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite() at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.Diagnostics.TextWriterTraceListener.Flush() at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id) at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message) at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message) at GitHub.Runner.Common.Tracing.Error(Exception exception) at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args) Unhandled exception. System.IO.IOException: No space left on device : '/opt/actions-runner/_diag/Worker_20240606-221047-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset) at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite() at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.Diagnostics.TextWriterTraceListener.Flush() at System.Diagnostics.TraceSource.Flush() at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing) at GitHub.Runner.Common.TraceManager.Dispose() at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing) at GitHub.Runner.Common.HostContext.Dispose() at GitHub.Runner.Worker.Program.Main(String[] args)
PYTHON-3-10 / BENCHMARK / BENCHMARK
Unable to process file command 'step_summary' successfully.
PYTHON-3-10 / BENCHMARK / BENCHMARK
No space left on device : '/opt/actions-runner/_diag/Worker_20240606-221047-utc.log'
PYTHON-3-9 / BENCHMARK / BENCHMARK_REPORT
# :warning: **Performance Alert** :warning: Possible performance regression was detected for benchmark **'bigger_is_better'**. Benchmark result of this commit is worse than the previous benchmark result exceeding threshold `1.10`. | Benchmark suite | Current: 7723eb7efc1ee09e000b3a86204e1b1a2a4f19e0 | Previous: 367c5ee80cc75f5d5b6af72de5e1e5e463e386f7 | Ratio | |-|-|-|-| | `{"name": "request_throughput", "description": "VLLM Engine prefill throughput - 2:4 Sparse (synthetic)\nmodel - neuralmagic/OpenHermes-2.5-Mistral-7B-pruned2.4\nmax_model_len - 4096\nbenchmark_throughput {\n \"use-all-available-gpus_\": \"\",\n \"input-len\": 128,\n \"output-len\": 1,\n \"num-prompts\": 1,\n \"sparsity\": \"semi_structured_sparse_w16a16\"\n}", "gpu_description": "NVIDIA A10G x 1", "vllm_version": "0.5.0", "python_version": "3.9.17 (main, Jun 7 2023, 12:29:40) \n[GCC 9.4.0]", "torch_version": "2.3.0+cu121"}` | `21.433299156481482` prompts/s | `24.347844802232547` prompts/s | `1.14` | | `{"name": "token_throughput", "description": "VLLM Engine prefill throughput - 2:4 Sparse (synthetic)\nmodel - neuralmagic/OpenHermes-2.5-Mistral-7B-pruned2.4\nmax_model_len - 4096\nbenchmark_throughput {\n \"use-all-available-gpus_\": \"\",\n \"input-len\": 128,\n \"output-len\": 1,\n \"num-prompts\": 1,\n \"sparsity\": \"semi_structured_sparse_w16a16\"\n}", "gpu_description": "NVIDIA A10G x 1", "vllm_version": "0.5.0", "python_version": "3.9.17 (main, Jun 7 2023, 12:29:40) \n[GCC 9.4.0]", "torch_version": "2.3.0+cu121"}` | `2764.895591186111` tokens/s | `3140.8719794879985` tokens/s | `1.14` | This comment was automatically generated by [workflow](https://github.com/neuralmagic/nm-vllm/actions?query=workflow%3ANightly) using [github-action-benchmark](https://github.com/marketplace/actions/continuous-benchmark). Comment was generated at https://github.com/neuralmagic/nm-vllm/commit/7723eb7efc1ee09e000b3a86204e1b1a2a4f19e0#commitcomment-142833027
PYTHON-3-9 / BENCHMARK / BENCHMARK_REPORT
# :warning: **Performance Alert** :warning: Possible performance regression was detected for benchmark **'smaller_is_better'**. Benchmark result of this commit is worse than the previous benchmark result exceeding threshold `1.10`. | Benchmark suite | Current: 7723eb7efc1ee09e000b3a86204e1b1a2a4f19e0 | Previous: 367c5ee80cc75f5d5b6af72de5e1e5e463e386f7 | Ratio | |-|-|-|-| | `{"name": "mean_ttft_ms", "description": "VLLM Serving - 2:4 Sparse\nmodel - neuralmagic/OpenHermes-2.5-Mistral-7B-pruned2.4\nmax-model-len - 4096\nsparsity - semi_structured_sparse_w16a16\nbenchmark_serving {\n \"nr-qps-pair_\": \"1500,5\",\n \"dataset\": \"sharegpt\"\n}", "gpu_description": "NVIDIA A10G x 1", "vllm_version": "0.5.0", "python_version": "3.9.17 (main, Jun 7 2023, 12:29:40) \n[GCC 9.4.0]", "torch_version": "2.3.0+cu121"}` | `1509.6453839919814` ms | `1292.66038330267` ms | `1.17` | This comment was automatically generated by [workflow](https://github.com/neuralmagic/nm-vllm/actions?query=workflow%3ANightly) using [github-action-benchmark](https://github.com/marketplace/actions/continuous-benchmark). Comment was generated at https://github.com/neuralmagic/nm-vllm/commit/7723eb7efc1ee09e000b3a86204e1b1a2a4f19e0#commitcomment-142833030
PYTHON-3-10 / BENCHMARK / BENCHMARK
You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 3 MB
PYTHON-3-9 / BENCHMARK / BENCHMARK_REPORT
Performance alert! Previous value was 24.347844802232547 and current value is 21.433299156481482. It is 1.1359821287647964x worse than previous exceeding a ratio threshold 1.1
PYTHON-3-9 / BENCHMARK / BENCHMARK_REPORT
Performance alert! Previous value was 3140.8719794879985 and current value is 2764.895591186111. It is 1.1359821287647964x worse than previous exceeding a ratio threshold 1.1
PYTHON-3-9 / BENCHMARK / BENCHMARK_REPORT
Performance alert! Previous value was 1292.66038330267 and current value is 1509.6453839919814. It is 1.1678592486411068x worse than previous exceeding a ratio threshold 1.1

Artifacts

Produced during runtime
Name Size
3.10.12-nm-vllm-nightly-0.5.0.20240606.tar.gz Expired
578 KB
3.11.4-nm-vllm-nightly-0.5.0.20240606.tar.gz Expired
578 KB
3.8.17-nm-vllm-nightly-0.5.0.20240606.tar.gz Expired
578 KB
3.9.17-nm-vllm-nightly-0.5.0.20240606.tar.gz Expired
578 KB
9407868780-aws-test-a10g-24G-3.11.4 Expired
124 KB
9407868780-aws-test-a10g-24G-3.8.17 Expired
124 KB
9407868780-aws-test-a10g-24G-3.9.17 Expired
124 KB
cc-vllm-html-aws-avx2-32G-a10g-24G-3.10.12 Expired
2.56 MB
cc-vllm-html-aws-test-a10g-24G-3.11.4 Expired
2.58 MB
cc-vllm-html-aws-test-a10g-24G-3.8.17 Expired
2.58 MB
cc-vllm-html-aws-test-a10g-24G-3.9.17 Expired
2.58 MB
gh_action_benchmark_jsons-9407868780-aws-test-a10g-24G-3.11.4 Expired
28.9 KB
gh_action_benchmark_jsons-9407868780-aws-test-a10g-24G-3.8.17 Expired
28.5 KB
gh_action_benchmark_jsons-9407868780-aws-test-a10g-24G-3.9.17 Expired
28.7 KB
nm_vllm_nightly-0.5.0.20240606-cp310-cp310-manylinux_2_17_x86_64.whl Expired
130 MB
nm_vllm_nightly-0.5.0.20240606-cp311-cp311-manylinux_2_17_x86_64.whl Expired
130 MB
nm_vllm_nightly-0.5.0.20240606-cp38-cp38-manylinux_2_17_x86_64.whl Expired
130 MB
nm_vllm_nightly-0.5.0.20240606-cp39-cp39-manylinux_2_17_x86_64.whl Expired
130 MB