add iscsi+multipath test; add some improvements to test strategy #3789

dustymabe · 2024-05-02T03:20:16Z

See individual commit messages.

jlebon

Very nice series! Some comments but LGTM overall.

mantle/cmd/kola/resources/iscsi_butane_setup.yaml

jlebon · 2024-05-07T03:29:56Z

mantle/cmd/kola/testiso.go

@@ -90,6 +90,7 @@ var (
 		"iso-offline-install.mpath.bios",
 		"iso-offline-install-fromram.4k.uefi",
 		"iso-offline-install-iscsi.ibft.uefi",
+		"iso-offline-install-iscsi.ibft-with-multipath.bios",


Minor/optional: we use mpath in the one other test that exercises it. Maybe iso-offline-install-iscsi.ibft-mpath.bios for consistency?

ironically enough we actually don't want consistency here. We don't want to match

coreos-assembler/mantle/cmd/kola/testiso.go

Lines 580 to 583 in 4d4e284

if kola.HasString("mpath", components) {

enableMultipath = true

inst.MultiPathDisk = true

}

It's probably worth a comment to explain the nuance here?

ok. I looked a little bit closer.. kola.HasString() will only match if the full string of the component matches. so ibft-with-mpath wouldn't match and we'd be fine.

Are you good with ibft-with-mpath?

Updated to ibft-with-mpath

jlebon · 2024-05-07T03:32:15Z

mantle/cmd/kola/testiso.go

-				butane_config = strings.ReplaceAll(iscsi_butane_config, "COREOS_INSTALLER_KARGS", "--append-karg netroot=iscsi:10.0.2.15::::iqn.2024-05.com.coreos")
+				butane_config = strings.ReplaceAll(iscsi_butane_config, "COREOS_INSTALLER_KARGS", "--append-karg netroot=iscsi:10.0.2.15::::iqn.2024-05.com.coreos:0")
+			case "ibft-with-multipath":
+				butane_config = strings.ReplaceAll(iscsi_butane_config, "COREOS_INSTALLER_KARGS", "--append-karg rd.iscsi.firmware=1 --append-karg rd.multipath=default --append-karg root=/dev/disk/by-label/dm-mpath-root")


Interesting. This normally should also require --append-karg rw as per https://github.com/openshift/os/blob/master/docs/faq.md#q-does-rhcos-support-multipath-on-the-primary-disk. Wonder why it's not required here.

In my original testing I always added rw but then when I got here I was challenging every kernel argument and testing without them and it still passed the test. I went back and removed it in my original setup and it wasn't needed there either from what I could tell.

followup in #3797

mantle/cmd/kola/testiso.go

mantle/cmd/kola/resources/iscsi_butane_setup.yaml

jbtrystram

Thanks Dusty for going in depth on the iscsi test and making all those tweaks !

The unit kept getting killed when entering emergency.target, which isn't very useful since some of the messages that hadn't been flushed yet were the ones that revealed the root cause of the failure. Let's set up the ordering/deps such that it doesn't get taken down when entering emergency.target. Also add a sleep 1 in the Destroy to allow for some remaining messages to get flushed.

In my experience the value of this has been little to none. It mostly just forces us to filter this output through jq -r .MESSAGE Let's make it just output text like the non-testiso journal.txt files are today.

There's no bios on aarch64, ppc64le or s390x.

The ip=ibft seems to only dictate what interface to use when bringing up networking, but isn't important if you are using DHCP IIUC. The rd.iscsi.initiator also isn't important here. It appears to work without it.

This will help get more test coverage.

Makes it a little bit easier for the reader to understand what the options are doing without having to study the iscsiadm man page.

We can use a random value here because we don't have ACL's enabled for this test. Let's just make it a random UUID to make it more clear the value isn't significant.

This has the same effect, just uses one fewer command!

When we go to emergency.target it's standard to use OnFailureJobMode=isolate.

The format doesn't need the extra pieces, which might be confused as important. Let's use the simplest possible name here.

Here we modify the iscsi setup to define two ports `:0` and `:1` that both expose the same underlying block device, which means that multipath can be set up on top of those two paths. We then add a new test that provides `root=/dev/disk/by-label/dm-mpath-root` and `rd.multipath=default` to the kernel command line to perform the test. Note that this modifies the targetd setup and the iPXE config for all of the tests, but they still work fine, just accessing `:0` and leaving `:1` alone.

jbtrystram

lgtm

dustymabe · 2024-05-07T17:56:49Z

/cherry-pick rhcos-4.16

openshift-cherrypick-robot · 2024-05-07T17:57:38Z

@dustymabe: #3789 failed to apply on top of branch "rhcos-4.16":

Applying: mantle/platform/qemu: Adjust dependencies on journal-stream unit
Applying: mantle/platform/qemu: drop json formatting from journal output
Applying: mantle/kola/testiso: fix firmware entries for multi-arch iscsi tests
Using index info to reconstruct a base tree...
M	mantle/cmd/kola/testiso.go
Falling back to patching base and 3-way merge...
Auto-merging mantle/cmd/kola/testiso.go
CONFLICT (content): Merge conflict in mantle/cmd/kola/testiso.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0003 mantle/kola/testiso: fix firmware entries for multi-arch iscsi tests
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick rhcos-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This is required by ostree. This sometimes worked I think because `ignition-remount-sysroot.service` has no ordering against `ostree-prepare-root.service` and so they'd race. But really, we shouldn't rely on that Ignition unit. See also: coreos#3789 (comment)

This is required by ostree. This sometimes worked I think because `ignition-remount-sysroot.service` has no ordering against `ostree-prepare-root.service` and so they'd race. But really, we shouldn't rely on that Ignition unit. See also: #3789 (comment)

jlebon · 2024-05-10T13:23:01Z

Follow-up in #3799.

jlebon previously approved these changes May 7, 2024

View reviewed changes

jbtrystram reviewed May 7, 2024

View reviewed changes

mantle/cmd/kola/testiso.go Show resolved Hide resolved

jbtrystram reviewed May 7, 2024

View reviewed changes

mantle/cmd/kola/resources/iscsi_butane_setup.yaml Outdated Show resolved Hide resolved

jbtrystram previously approved these changes May 7, 2024

View reviewed changes

dustymabe added 12 commits May 7, 2024 09:26

mantle/platform/qemu: drop json formatting from journal output

f06767d

In my experience the value of this has been little to none. It mostly just forces us to filter this output through jq -r .MESSAGE Let's make it just output text like the non-testiso journal.txt files are today.

mantle/kola/testiso: fix firmware entries for multi-arch iscsi tests

cc18190

There's no bios on aarch64, ppc64le or s390x.

mantle/kola: drop unnecessary kargs for iscsi tests

7b9d3da

The ip=ibft seems to only dictate what interface to use when bringing up networking, but isn't important if you are using DHCP IIUC. The rd.iscsi.initiator also isn't important here. It appears to work without it.

mantle/kola/testiso: switch one iscsi test to use uefi

6ca617c

This will help get more test coverage.

mantle/kola: make options to iscsiadm long options

3c5b777

Makes it a little bit easier for the reader to understand what the options are doing without having to study the iscsiadm man page.

mantle/kola: use random uuid for ipxe initiator-iqn

a3d49ea

We can use a random value here because we don't have ACL's enabled for this test. Let's just make it a random UUID to make it more clear the value isn't significant.

mantle/kola: login from discovery mode

9e6f0f8

This has the same effect, just uses one fewer command!

mantle/kola: use OnFailureJobMode=isolate for iscsi test units

1f96d25

When we go to emergency.target it's standard to use OnFailureJobMode=isolate.

mantle/kola: simplify naming of iscsi targets

dcf3f71

The format doesn't need the extra pieces, which might be confused as important. Let's use the simplest possible name here.

mantle/kola: add -o pipefail to one of the scripts in iscsi test

4f7f345

dustymabe dismissed stale reviews from jbtrystram and jlebon via d7d352d May 7, 2024 13:40

dustymabe force-pushed the dusty-iscsi-tests branch from 462d48f to d7d352d Compare May 7, 2024 13:40

dustymabe enabled auto-merge (rebase) May 7, 2024 13:41

jbtrystram approved these changes May 7, 2024

View reviewed changes

dustymabe merged commit c317213 into coreos:main May 7, 2024
5 checks passed

dustymabe deleted the dusty-iscsi-tests branch May 7, 2024 15:43

dustymabe mentioned this pull request May 7, 2024

[rhcos-4.16] add more iscsi test cases #3795

Merged

jlebon mentioned this pull request May 8, 2024

testiso: add rw karg to iSCSI multipath test #3797

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add iscsi+multipath test; add some improvements to test strategy #3789

add iscsi+multipath test; add some improvements to test strategy #3789

dustymabe commented May 2, 2024

jlebon left a comment

jlebon May 7, 2024

dustymabe May 7, 2024

dustymabe May 7, 2024

dustymabe May 7, 2024

jlebon May 7, 2024

jlebon May 7, 2024

dustymabe May 7, 2024

dustymabe May 8, 2024

jbtrystram left a comment

jbtrystram left a comment

dustymabe commented May 7, 2024

openshift-cherrypick-robot commented May 7, 2024

jlebon commented May 10, 2024

	if kola.HasString("mpath", components) {
	enableMultipath = true
	inst.MultiPathDisk = true
	}

add iscsi+multipath test; add some improvements to test strategy #3789

add iscsi+multipath test; add some improvements to test strategy #3789

Conversation

dustymabe commented May 2, 2024

jlebon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbtrystram left a comment

Choose a reason for hiding this comment

jbtrystram left a comment

Choose a reason for hiding this comment

dustymabe commented May 7, 2024

openshift-cherrypick-robot commented May 7, 2024

jlebon commented May 10, 2024