Skip to content

Commit

Permalink
stats: improve handling of missing steps
Browse files Browse the repository at this point in the history
When the sample time is too high, step metrics might be missing which
makes the stats generation confused. Insert steps for these with 0
samples to make sure that the number of steps matches with the expected
number of steps to avoid that safety mechamism kick in.
  • Loading branch information
DylanVanAssche committed May 17, 2023
1 parent bc892d0 commit 5363280
Show file tree
Hide file tree
Showing 14 changed files with 390 additions and 3,678 deletions.
15 changes: 5 additions & 10 deletions bench_executor/data/collector/results/run_1/case-info.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
===> CASE <===
Timestamp: 2023-03-02T07:56:21.542222
Timestamp: 2023-05-17T09:41:02.431418
Directory: /home/dylan/Projects/challenge-tool/tests/./../bench_executor/log
Run: 1
Number of steps: 4
Expand All @@ -8,16 +8,16 @@ Number of steps: 4
System
Hostname: dell-laptop
OS name: Fedora Linux
OS version: 37 (Workstation Edition)
Kernel: Linux-6.1.9-200.fc37.x86_64-x86_64-with-glibc2.36
OS version: 38 (Workstation Edition)
Kernel: Linux-6.2.15-300.fc38.x86_64-x86_64-with-glibc2.37
Architecture: x86_64
CPU
Name: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
Cores: 8
Minimum frequency: 400 Hz
Maximum frequency: 4200 Hz
Memory
RAM memory: 16622 MB
RAM memory: 16605 MB
SWAP memory: 8589 MB
Storage
Disk "/": 510.48 GB
Expand All @@ -28,13 +28,8 @@ Network
Interface "lo"
Interface "enp0s31f6": 1000 mbps
Interface "wlp2s0"
Interface "tun0": 10 mbps
Interface "br-199916b8f7a4"
Interface "br-8121eb5f4e6f"
Interface "br-82f8d8cfa22d"
Interface "docker0"
Interface "br-e35e8c84b838"
Interface "br-0b46b8270d29"
Interface "br-fc27d82a1e7f"

===> DOCKER <===
Version: 20.10.23
Expand Down
668 changes: 335 additions & 333 deletions bench_executor/data/collector/results/run_1/metrics.csv

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion bench_executor/data/stats/expected/aggregated.csv
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ index,step,timestamp,version,cpu_user,cpu_system,cpu_user_system,cpu_idle,cpu_io
3,2,1.9,2,1.0,0.5,1.5,0.2,0.0,1500000,1000000,2500000,110,1100,14080,140800,30,170,120,20,40,5120,20480,0,0,0,0
4,2,2.9,2,1.5,2.0,3.5,0.3,0.0,1000000,1000000,2000000,120,1200,15360,153600,40,190,140,30,60,7680,30720,0,0,0,0
5,2,3.9,2,2.0,2.5,4.5,0.4,0.0,2500000,1000000,3500000,130,1300,16640,166400,50,210,160,40,80,10240,40960,0,0,0,0
6,3,5.0,2,2.5,3.0,5.5,0.5,0.0,500000,1000000,1500000,140,1400,17920,179200,60,230,180,50,100,12800,51200,0,0,0,0
6,3,4.5,2,2.5,3.0,5.5,0.5,0.0,500000,1000000,1500000,140,1400,17920,179200,60,230,180,50,100,12800,51200,0,0,0,0
7,4,6.0,2,5.0,5.0,10.0,0.6,0.0,5000000,1000000,6000000,150,1500,19200,192000,70,250,200,60,120,15360,61440,0,0,0,0
8,4,7.0,2,7.0,8.0,15.0,0.7,0.0,1000000,1000000,2000000,160,1600,20480,204800,80,270,220,70,140,17920,71680,0,0,0,0
9,4,8.9,2,7.0,8.0,15.0,0.8,0.0,1000000,1000000,2000000,170,1700,21760,217600,90,290,240,80,160,20480,81920,0,0,0,0
Expand Down
2 changes: 1 addition & 1 deletion bench_executor/data/stats/results/aggregated.csv
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ index,step,timestamp,version,cpu_user,cpu_system,cpu_user_system,cpu_idle,cpu_io
3,2,1.9,2,1.0,0.5,1.5,0.2,0.0,1500000,1000000,2500000,110,1100,14080,140800,30,170,120,20,40,5120,20480,0,0,0,0
4,2,2.9,2,1.5,2.0,3.5,0.3,0.0,1000000,1000000,2000000,120,1200,15360,153600,40,190,140,30,60,7680,30720,0,0,0,0
5,2,3.9,2,2.0,2.5,4.5,0.4,0.0,2500000,1000000,3500000,130,1300,16640,166400,50,210,160,40,80,10240,40960,0,0,0,0
6,3,5.0,2,2.5,3.0,5.5,0.5,0.0,500000,1000000,1500000,140,1400,17920,179200,60,230,180,50,100,12800,51200,0,0,0,0
6,3,4.5,2,2.5,3.0,5.5,0.5,0.0,500000,1000000,1500000,140,1400,17920,179200,60,230,180,50,100,12800,51200,0,0,0,0
7,4,6.0,2,5.0,5.0,10.0,0.6,0.0,5000000,1000000,6000000,150,1500,19200,192000,70,250,200,60,120,15360,61440,0,0,0,0
8,4,7.0,2,7.0,8.0,15.0,0.7,0.0,1000000,1000000,2000000,160,1600,20480,204800,80,270,220,70,140,17920,71680,0,0,0,0
9,4,8.9,2,7.0,8.0,15.0,0.8,0.0,1000000,1000000,2000000,170,1700,21760,217600,90,290,240,80,160,20480,81920,0,0,0,0
Expand Down
12 changes: 12 additions & 0 deletions bench_executor/data/test-cases/log.txt
Original file line number Diff line number Diff line change
Expand Up @@ -148,3 +148,15 @@
[2023-03-02 09:09:40,620] DEBUG Copying generated files for run
[2023-03-02 09:09:40,620] DEBUG Writing run checkpoint...
[2023-03-02 09:09:40,620] DEBUG Cooling down for 15s
[2023-05-17 11:34:41,088] INFO Logger (INFO) initialized for bench_executor.executor
[2023-05-17 11:34:41,112] ERROR /home/dylan/Projects/challenge-tool/bench_executor/data/test-cases/test-error-command/metadata.json: Unknown command "DOES_NOT_EXIST" for resource "RMLMapper"
[2023-05-17 11:34:41,113] ERROR /home/dylan/Projects/challenge-tool/bench_executor/data/test-cases/test-incomplete-data/metadata.json: JSON schema violation
[2023-05-17 11:34:41,114] ERROR /home/dylan/Projects/challenge-tool/bench_executor/data/test-cases/test-missing-parameter/metadata.json: Missing required parameter "output_file" for command "execute_mapping" of resource "RMLMapper"
[2023-05-17 11:34:41,115] ERROR /home/dylan/Projects/challenge-tool/bench_executor/data/test-cases/test-missing-resource/metadata.json: Unknown resource "DOES_NOT_EXIST"
[2023-05-17 11:34:41,116] ERROR /home/dylan/Projects/challenge-tool/bench_executor/data/test-cases/test-no-steps/metadata.json: JSON schema violation
[2023-05-17 11:34:41,116] INFO Logger (DEBUG) initialized for bench_executor.executor
[2023-05-17 11:34:41,146] ERROR /home/dylan/Projects/challenge-tool/bench_executor/data/test-cases/test-error-command/metadata.json: Unknown command "DOES_NOT_EXIST" for resource "RMLMapper"
[2023-05-17 11:34:41,148] ERROR /home/dylan/Projects/challenge-tool/bench_executor/data/test-cases/test-incomplete-data/metadata.json: JSON schema violation
[2023-05-17 11:34:41,149] ERROR /home/dylan/Projects/challenge-tool/bench_executor/data/test-cases/test-missing-parameter/metadata.json: Missing required parameter "output_file" for command "execute_mapping" of resource "RMLMapper"
[2023-05-17 11:34:41,151] ERROR /home/dylan/Projects/challenge-tool/bench_executor/data/test-cases/test-missing-resource/metadata.json: Unknown resource "DOES_NOT_EXIST"
[2023-05-17 11:34:41,153] ERROR /home/dylan/Projects/challenge-tool/bench_executor/data/test-cases/test-no-steps/metadata.json: JSON schema violation

This file was deleted.

This file was deleted.

This file was deleted.

Loading

0 comments on commit 5363280

Please sign in to comment.