Skip to content

Commit

Permalink
Single source self test status output for maintainability
Browse files Browse the repository at this point in the history
  • Loading branch information
@kbatuigas
kbatuigas committed Jul 22, 2024
1 parent 8490645 commit f98a441
Show file tree
Hide file tree
Showing 3 changed files with 231 additions and 448 deletions.
228 changes: 1 addition & 227 deletions modules/manage/pages/cluster-maintenance/cluster-diagnostics.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,233 +82,7 @@ rpk cluster self-test status --format=json
If benchmarks have completed, `self-test status` returns their results.

.Example status output: test results
[%collapsible]
====
Test results are grouped by node ID. Each test returns the following:
- **NAME**: Description of the test.
- **INFO**: Detail about the test run attached by Redpanda itself.
- **TYPE**: Either `disk` or `network` test.
- **TEST ID**: Unique identifier given to jobs of a run. All IDs in a test should match. If they don't match, then newer and/or older test results have been included erroneously.
- **TIMEOUTS**: Number of timeouts incurred during the test.
- **DURATION**: Duration of the test.
- **IOPS**: Number of operations per second. For disk, it's `seastar::dma_read` and `seastar::dma_write`. For network, it's `rpc.send()`
- **THROUGHPUT**: For disk, it's throughput rate in bytes per second. For network, it's throughput rate in bits per second in. (Note: GiB vs. Gib is the correct notation displayed by the UI.)
- **LATENCY**: 50th, 90th, etc. percentiles of operation latency, reported in microseconds.
```
$ rpk cluster self-test status
NODE ID: 0 | STATUS: IDLE
=========================
NAME 512KB sequential r/w
INFO write run (iodepth: 4, dsync: true)
TYPE disk
TEST ID 21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS 0
START TIME Fri Jul 19 15:02:45 UTC 2024
END TIME Fri Jul 19 15:03:15 UTC 2024
AVG DURATION 30002ms
IOPS 1182 req/sec
THROUGHPUT 591.4MiB/sec
LATENCY P50 P90 P99 P999 MAX
3199us 3839us 9727us 12799us 21503us

NAME 512KB sequential r/w
INFO read run
TYPE disk
TEST ID 21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS 0
START TIME Fri Jul 19 15:03:15 UTC 2024
END TIME Fri Jul 19 15:03:45 UTC 2024
AVG DURATION 30000ms
IOPS 6652 req/sec
THROUGHPUT 3.248GiB/sec
LATENCY P50 P90 P99 P999 MAX
607us 671us 831us 991us 2431us

NAME 4KB sequential r/w, low io depth
INFO write run (iodepth: 1, dsync: true)
TYPE disk
TEST ID 21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS 0
START TIME Fri Jul 19 15:03:45 UTC 2024
END TIME Fri Jul 19 15:04:15 UTC 2024
AVG DURATION 30000ms
IOPS 406 req/sec
THROUGHPUT 1.59MiB/sec
LATENCY P50 P90 P99 P999 MAX
2431us 2559us 2815us 5887us 9215us

NAME 4KB sequential r/w, low io depth
INFO read run
TYPE disk
TEST ID 21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS 0
START TIME Fri Jul 19 15:04:15 UTC 2024
END TIME Fri Jul 19 15:04:45 UTC 2024
AVG DURATION 30000ms
IOPS 430131 req/sec
THROUGHPUT 1.641GiB/sec
LATENCY P50 P90 P99 P999 MAX
1us 2us 12us 28us 511us

NAME 4KB sequential write, medium io depth
INFO write run (iodepth: 8, dsync: true)
TYPE disk
TEST ID 21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS 0
START TIME Fri Jul 19 15:04:45 UTC 2024
END TIME Fri Jul 19 15:05:15 UTC 2024
AVG DURATION 30013ms
IOPS 513 req/sec
THROUGHPUT 2.004MiB/sec
LATENCY P50 P90 P99 P999 MAX
15871us 16383us 21503us 32767us 40959us

NAME 4KB sequential write, high io depth
INFO write run (iodepth: 64, dsync: true)
TYPE disk
TEST ID 21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS 0
START TIME Fri Jul 19 15:05:15 UTC 2024
END TIME Fri Jul 19 15:05:45 UTC 2024
AVG DURATION 30114ms
IOPS 550 req/sec
THROUGHPUT 2.151MiB/sec
LATENCY P50 P90 P99 P999 MAX
118783us 118783us 147455us 180223us 180223us

NAME 4KB sequential write, very high io depth
INFO write run (iodepth: 256, dsync: true)
TYPE disk
TEST ID 21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS 0
START TIME Fri Jul 19 15:05:45 UTC 2024
END TIME Fri Jul 19 15:06:16 UTC 2024
AVG DURATION 30460ms
IOPS 558 req/sec
THROUGHPUT 2.183MiB/sec
LATENCY P50 P90 P99 P999 MAX
475135us 491519us 507903us 524287us 524287us

NAME 4KB sequential write, no dsync
INFO write run (iodepth: 64, dsync: false)
TYPE disk
TEST ID 21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS 0
START TIME Fri Jul 19 15:06:16 UTC 2024
END TIME Fri Jul 19 15:06:46 UTC 2024
AVG DURATION 30000ms
IOPS 424997 req/sec
THROUGHPUT 1.621GiB/sec
LATENCY P50 P90 P99 P999 MAX
135us 183us 303us 543us 9727us

NAME 16KB sequential r/w, high io depth
INFO write run (iodepth: 64, dsync: false)
TYPE disk
TEST ID 21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS 0
START TIME Fri Jul 19 15:06:46 UTC 2024
END TIME Fri Jul 19 15:07:16 UTC 2024
AVG DURATION 30000ms
IOPS 103047 req/sec
THROUGHPUT 1.572GiB/sec
LATENCY P50 P90 P99 P999 MAX
511us 1087us 1343us 1471us 15871us

NAME 16KB sequential r/w, high io depth
INFO read run
TYPE disk
TEST ID 21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS 0
START TIME Fri Jul 19 15:07:16 UTC 2024
END TIME Fri Jul 19 15:07:46 UTC 2024
AVG DURATION 30000ms
IOPS 193966 req/sec
THROUGHPUT 2.96GiB/sec
LATENCY P50 P90 P99 P999 MAX
319us 383us 735us 1023us 6399us

NAME 8K Network Throughput Test
INFO Test performed against node: 1
TYPE network
TEST ID 5e4052f3-b828-4c0d-8fd0-b34ff0b6c35d
TIMEOUTS 0
DURATION 5000ms
IOPS 61612 req/sec
THROUGHPUT 3.76Gib/sec
LATENCY P50 P90 P99 P999 MAX
159us 207us 303us 431us 1151us

NAME 8K Network Throughput Test
INFO Test performed against node: 2
TYPE network
TEST ID 5e4052f3-b828-4c0d-8fd0-b34ff0b6c35d
TIMEOUTS 0
DURATION 5000ms
IOPS 60306 req/sec
THROUGHPUT 3.68Gib/sec
LATENCY P50 P90 P99 P999 MAX
159us 215us 351us 495us 11263us

NAME Cloud Storage Test
INFO Put
TYPE cloud
TEST ID a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS 0
START TIME Tue Jul 16 18:06:30 UTC 2024
END TIME Tue Jul 16 18:06:30 UTC 2024
AVG DURATION 8ms

NAME Cloud Storage Test
INFO List
TYPE cloud
TEST ID a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS 0
START TIME Tue Jul 16 18:06:30 UTC 2024
END TIME Tue Jul 16 18:06:30 UTC 2024
AVG DURATION 1ms

NAME Cloud Storage Test
INFO Get
TYPE cloud
TEST ID a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS 0
START TIME Tue Jul 16 18:06:30 UTC 2024
END TIME Tue Jul 16 18:06:30 UTC 2024
AVG DURATION 1ms

NAME Cloud Storage Test
INFO Head
TYPE cloud
TEST ID a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS 0
START TIME Tue Jul 16 18:06:30 UTC 2024
END TIME Tue Jul 16 18:06:30 UTC 2024
AVG DURATION 0ms

NAME Cloud Storage Test
INFO Delete
TYPE cloud
TEST ID a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS 0
START TIME Tue Jul 16 18:06:30 UTC 2024
END TIME Tue Jul 16 18:06:30 UTC 2024
AVG DURATION 1ms

NAME Cloud Storage Test
INFO Plural Delete
TYPE cloud
TEST ID a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS 0
START TIME Tue Jul 16 18:06:30 UTC 2024
END TIME Tue Jul 16 18:06:30 UTC 2024
AVG DURATION 47ms
```
====
NOTE: If self-test returns write results that are unexpectedly and significantly lower than read results, it may be because the Redpanda `rpk` client hardcodes the `DSync` option to `true`. When `DSync` is enabled, files are opened with the `O_DSYNC` flag set, and this represents the actual setting that Redpanda uses when it writes to disk.
include::reference:partial$rpk-self-test-status-output.adoc[]

To stop a running self-test, run the `self-test stop` command.

Expand Down
Loading

0 comments on commit f98a441

Please sign in to comment.