test/system: Various improvements to the 'help' tests #1386
Conversation
debarshiray
force-pushed
the
wip/rishi/test-system-help-groff-1.23-font-C
branch
from
1b4bc08 to
dacecd8
Compare
debarshiray
force-pushed
the
wip/rishi/test-system-help-groff-1.23-font-C
branch
from
dacecd8 to
16519d1
Compare
|
Build failed. ✔️ unit-test SUCCESS in 8m 29s |
|
recheck |
|
Build failed. ✔️ unit-test SUCCESS in 8m 56s |
Commit 5e63e9e added a 'help' command to show the toolbox(1) manual or a manual page for a specific command, and made the --help flag identical to it. Therefore it's misleading to say that the --help flag should show the usage screen. The usage screen is a brief listing of the commands and options, which isn't the same thing as the more detailed manuals. Fallout from b27795a containers#1386
|
Build failed. ✔️ unit-test SUCCESS in 8m 45s |
Commit 5e63e9e added a 'help' command to show the toolbox(1) manual or a manual page for a specific command, and made the --help flag identical to it. Therefore it's misleading to say that the --help flag should show the usage screen. The usage screen is a brief listing of the commands and options, which isn't the same thing as the more detailed manuals. Later, after this test was written, commit 40fc168 added a fallback for host operating systems without man(1), like Fedora CoreOS, that would show a very brief usage screen with only the most common commands. To make it more confusing, the test was checking for a string that's common to both the toolbox(1) manual and the fallback brief usage screen that might be shown by 'toolbox --help'. This meant that it was neither able to distinguish between the code paths nor ensure that they were working as intended. This was resolved by adapting the existing 'toolbox --help' test to strictly ensure that it's showing the toolbox(1) manual when man(1) is present, and by adding a new test to strictly ensure that it's showing the fallback brief usage screen when man(1) is absent. Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38. Fallout from b27795a [1] Bats commit 6648e2143bffb933 bats-core/bats-core@6648e2143bffb933 bats-core/bats-core#708 containers#1386
debarshiray
force-pushed
the
wip/rishi/test-system-help-groff-1.23-font-C
branch
from
f34b2ee to
033eb34
Compare
|
Build failed. ✔️ unit-test SUCCESS in 8m 21s |
With the recent expansion of the test suite, it's necessary to increase the timeout for all Fedora nodes to prevent the CI from timing out. containers#1386
|
Build failed. ✔️ unit-test SUCCESS in 8m 43s |
|
recheck |
|
Build failed. ✔️ unit-test SUCCESS in 8m 35s |
Currently, there's no way to get assert_line to use the stderr_lines array [1]. This is worked around by assigning stderr_lines to the 'lines' array. [1] bats-core/bats-assert#42 containers#1386
|
Build failed. ✔️ unit-test SUCCESS in 8m 43s |
|
recheck |
|
Build failed. ✔️ unit-test SUCCESS in 9m 15s |
Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted
the trailing newline on the last line as a separate line [1]. However,
Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras
37 and 38.
[1] Bats commit 6648e2143bffb933
bats-core/bats-core@6648e2143bffb933
bats-core/bats-core#708
containers#1387
Commit 5e63e9e added a 'help' command to show the toolbox(1) manual or a manual page for a specific command, and made the --help flag identical to it. Therefore it's misleading to say that the --help flag should show the usage screen. The usage screen is a brief listing of the commands and options, which isn't the same thing as the more detailed manuals. Later, after this test was written, commit 40fc168 added a fallback for host operating systems without man(1), like Fedora CoreOS, that would show a very brief usage screen with only the most common commands. To make it more confusing, the test was checking for a string that's common to both the toolbox(1) manual and the fallback brief usage screen that might be shown by 'toolbox --help'. This meant that it was neither able to distinguish between the code paths nor ensure that they were working as intended. This was resolved by adapting the existing 'toolbox --help' test to strictly ensure that it's showing the toolbox(1) manual when man(1) is present, and by adding a new test to strictly ensure that it's showing the fallback brief usage screen when man(1) is absent. Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38. Fallout from b27795a [1] Bats commit 6648e2143bffb933 bats-core/bats-core@6648e2143bffb933 bats-core/bats-core#708 containers#1386
With the recent expansion of the test suite, it's necessary to increase the timeout for all Fedora nodes to prevent the CI from timing out. containers#1386
debarshiray
force-pushed
the
wip/rishi/test-system-help-groff-1.23-font-C
branch
from
12c39ed to
3bd780b
Compare
Currently, there's no way to get assert_line to use the stderr_lines array [1]. This is worked around by assigning stderr_lines to the 'lines' array. [1] bats-core/bats-assert#42 containers#1386
|
Build failed. ✔️ unit-test SUCCESS in 9m 10s |
With the recent expansion of the test suite, it's necessary to increase the timeout for all Fedora nodes to prevent the CI from timing out. containers#1387
Commit 5e63e9e added a 'help' command to show the toolbox(1) manual or a manual page for a specific command, and made the --help flag identical to it. Therefore it's misleading to say that the --help flag should show the usage screen. The usage screen is a brief listing of the commands and options, which isn't the same thing as the more detailed manuals. Later, after this test was written, commit 40fc168 added a fallback for host operating systems without man(1), like Fedora CoreOS, that would show a very brief usage screen with only the most common commands. To make it more confusing, the test was checking for a string that's common to both the toolbox(1) manual and the fallback brief usage screen that might be shown by 'toolbox --help'. This meant that it was neither able to distinguish between the code paths nor ensure that they were working as intended. This was resolved by adapting the existing 'toolbox --help' test to strictly ensure that it's showing the toolbox(1) manual when man(1) is present, and by adding a new test to strictly ensure that it's showing the fallback brief usage screen when man(1) is absent. Until Bats 1.10.0, 'run --keep-empty-lines' had a bug where it counted the trailing newline on the last line as a separate line [1]. However, Bats 1.10.0 is only available in Fedora >= 39 and is absent from Fedoras 37 and 38. Fallout from b27795a [1] Bats commit 6648e2143bffb933 bats-core/bats-core@6648e2143bffb933 bats-core/bats-core#708 containers#1386
Currently, there's no way to get assert_line to use the stderr_lines array [1]. This is worked around by assigning stderr_lines to the 'lines' array. [1] bats-core/bats-assert#42 containers#1386
debarshiray
force-pushed
the
wip/rishi/test-system-help-groff-1.23-font-C
branch
from
3bd780b to
e7f729f
Compare
... for both 'toolbox help' and 'toolbox --help'. containers#1386
|
Build failed. ✔️ unit-test SUCCESS in 8m 10s |
Currently, some of the names of the tests were too long, and had inconsistent and verbose wording. This made it difficult to look at them and get a gist of all the scenarios being tested. The names are like headings. They shouldn't be too long, should capture the primary objective of the test and be consistent in their wording. Note that the term 'usage screen' was particularly confusing. Prior to commit 3dc106e, 'usage screen' in the names of the tests also referred to the very brief listing of the commands and options that's shown by 'toolbox help' and 'toolbox --help' in the absence of man(1). In the context of this change, the term referred to the brief two line error message that's shown when an unknown command or flag is used. So, it will be good to not use it anymore. containers#1386
|
Build failed. ✔️ unit-test SUCCESS in 10m 13s |
|
Build failed. ✔️ unit-test SUCCESS in 8m 38s |
|
recheck |
|
Build failed. ✔️ unit-test SUCCESS in 8m 42s |
Ansible's built-in 'package' module doesn't show any details when
installing the RPMs. All that can be seen is:
TASK [Install RPM packages]
fedora-rawhide | changed
Therefore, there's no way to know what version of the packages got
installed.
In this case, not knowing the go-md2man(1) version being used by the CI
makes it difficult to know why the tests are failing on Fedora Rawhide
and Fedora 39 with:
not ok 3 help: Command 'help' in 177ms
# (from function `assert_line' in file
test/system/libs/bats-assert/src/assert.bash, line 479,
# in test file test/system/002-help.bats, line 48)
# `assert_line --index 0 --partial "toolbox(1)"' failed
# /usr/bin/man
#
# -- line does not contain substring --
# index : 0
# substring : toolbox(1)
# line : troff:<standard input>:33: warning: cannot select font
'C'
# --
#
It could be either because the CI is still using an older version of
go-md2man(1) [1,2], or that there's some other problem.
[1] Fedora golang-github-cpuguy83-md2man commit 117806d50e401c19
https://src.fedoraproject.org/rpms/golang-github-cpuguy83-md2man/c/117806d50e401c19
https://src.fedoraproject.org/rpms/golang-github-cpuguy83-md2man/pull-request/3
[2] go-md2man commit d85280db9b54b574
cpuguy83/go-md2man@d85280db9b54b574
cpuguy83/go-md2man#99
containers#1386
|
Build failed. ✔️ unit-test SUCCESS in 8m 48s |
|
This is interesting: |
debarshiray
changed the title
|
Those failures are because of some changes in GNU roff 1.23. |
No description provided.