[test]Exclude features, add any feature check #1853

helsaawy · 2023-07-27T17:30:16Z

Add -exclude flag to be able to specify all but a certain set of flags to run.
Eg, functional.test.exe -exclude LCOWIntegrity will run all test except for those that require rego enforcement.

Additionally, added require.AnyFeature functions to check that at least one feature is specified.
This allows skipping a test that with subtests that individually require non-overlapping features, which avoids running generalized test setup

test/cri-containerd/main_test.go

test/cri-containerd/logging_binary_test.go

test/functional/uvm_mem_backingtype_test.go

test/pkg/flag/flag.go

test/pkg/flag/flag_test.go

test/pkg/require/requires.go

test/pkg/flag/flag.go

yyatmsft

🕐

test/pkg/flag/flag.go

yyatmsft

yyatmsft · 2023-08-29T22:41:39Z

It says this is from me but it's not...

In reply to: 1601395068

revoking review

yyatmsft · 2023-08-29T22:42:56Z

Can you create a work item to port this to azcri?
#Closed

yyatmsft

yyatmsft

Approved despite one minor comment.

Add `-exclude` flag to be able to specify all but a certain set of flags to run, since it is common (especially locally) to want to run all but a certain subset of tests. eg, `functional.test.exe -exclude LCOWIntegrity` will run all test except for those that require rego enforcement. Add `require.AnyFeature` function to check that at least one of the listed feature is specified. This allows skipping a test that with subtests that individually require non-overlapping features, which avoids running generalized test setup Moved `"test/cri-containerd".requireBinary` to `"test/pkg/require".Binary". Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

Incorporate several CI changes from multiple PRs: - microsoft#1632 - microsoft#1752 - microsoft#1755 - microsoft#1799 - microsoft#1820 - microsoft#1836 - microsoft#1853 - microsoft#1854 It is easier to checkout the changes directly rather than cherry-pick commits and undo any changes that affect code not backported to this release. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

Incorporate several CI changes from multiple PRs: - microsoft#1632 - microsoft#1752 - microsoft#1755 - microsoft#1799 - microsoft#1820 - microsoft#1836 - microsoft#1853 - microsoft#1854 It is easier to checkout the changes directly rather than cherry-pick commits and undo any changes that affect code not backported to this release. Fix bug in `internal\winapi\winapi.go` where `user.go` is referenced but does not exist. Remove `/usr/local/include` from `Protobuild.toml`, since it does not exist on Windows, and our CI build logic does not rely on it. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

Incorporate several CI changes from multiple PRs: - microsoft#1632 - microsoft#1752 - microsoft#1755 - microsoft#1799 - microsoft#1820 - microsoft#1836 - microsoft#1853 - microsoft#1854 It is easier to checkout the changes directly rather than cherry-pick commits and undo any changes that affect code not backported to this release. Remove `/usr/local/include` from `Protobuild.toml`, since it does not exist on Windows, and our CI build logic does not rely on it. Update `.gitignore` and `.gitattributes` to match main. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

* [release/0.9] Update CI Incorporate several CI changes from multiple PRs: - #1632 - #1752 - #1755 - #1799 - #1820 - #1836 - #1853 - #1854 It is easier to checkout the changes directly rather than cherry-pick commits and undo any changes that affect code not backported to this release. Remove `/usr/local/include` from `Protobuild.toml`, since it does not exist on Windows, and our CI build logic does not rely on it. Update `.gitignore` and `.gitattributes` to match main. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * [release/0.9] Skip failaing symlink tests CRI integration tests `TestContainerSymlinkVolumes/*` are consistently failing. Skip until a fix is backported from containerd 1.7 to 1.6. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> [release/0.9] Skip failing symlink tests Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * [release/0.9] Update proto files to v3 Many files were last updated with `github.com/gogo/protobuf@v1.2`, hence the `proto.GoGoProtoPackageIsVersion2` assertion in the `*.pb.go` files. Regenerate them using new version (v1.6.2) of `protoc-gen-gogoctrd`. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * [release/0.9] Update generated files Run `go generate` on repo to bring all auto-generated files up to date. Fix bug in `internal\winapi\winapi.go` where `user.go` is referenced but does not exist. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

Backport testing support functions in the `test/pkg` directory. This includes changes from: - microsoft#1536 - microsoft#1704 - microsoft#1853 - microsoft#1893 Rather than cherry-pick them, only changes to `test/pkg` are included, since tests themselves will require significant changes to bring up-to-date. The goal is to expose testing functions so that tests can be moved out of the repo. Updated go version in `test` to 1.8, as required by `test/pkg/flag`. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

* test: Exclude features, add any feature check Add `-exclude` flag to be able to specify all but a certain set of flags to run, since it is common (especially locally) to want to run all but a certain subset of tests. eg, `functional.test.exe -exclude LCOWIntegrity` will run all test except for those that require rego enforcement. Add `require.AnyFeature` function to check that at least one of the listed feature is specified. This allows skipping a test that with subtests that individually require non-overlapping features, which avoids running generalized test setup Moved `"test/cri-containerd".requireBinary` to `"test/pkg/require".Binary". Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * PR: docs, comments, naming Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

Backport testing support functions in the `test/pkg` directory. This includes changes from: - #1536 - #1704 - #1853 - #1893 Rather than cherry-pick them, only changes to `test/pkg` are included, since tests themselves will require significant changes to bring up-to-date. The goal is to expose testing functions so that tests can be moved out of the repo. Updated go version in `test` to 1.8, as required by `test/pkg/flag`. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

* test: Exclude features, add any feature check Add `-exclude` flag to be able to specify all but a certain set of flags to run, since it is common (especially locally) to want to run all but a certain subset of tests. eg, `functional.test.exe -exclude LCOWIntegrity` will run all test except for those that require rego enforcement. Add `require.AnyFeature` function to check that at least one of the listed feature is specified. This allows skipping a test that with subtests that individually require non-overlapping features, which avoids running generalized test setup Moved `"test/cri-containerd".requireBinary` to `"test/pkg/require".Binary". Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * PR: docs, comments, naming Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>

helsaawy requested a review from a team as a code owner July 27, 2023 17:30

helsaawy changed the title ~~test: Exclude features, add any feature check~~ [test]Exclude features, add any feature check Jul 27, 2023

msscotb assigned katiewasnothere and ambarve Aug 2, 2023

helsaawy force-pushed the test-flags branch from 57d6947 to 71b4c4b Compare August 21, 2023 16:51

yyatmsft reviewed Aug 23, 2023

View reviewed changes

test/cri-containerd/main_test.go Show resolved Hide resolved