The repository contains a set of test cases that run different test scenarios from cnf-certification-test project and verifies if these scenarios behave correctly under different environment conditions.
The cnfcert-tests-verification project is based on golang+ginkgo framework.
cnfcert-tests-verification
project triggers the same test scenario from
cnf-certification-test
several times using different pre-configured OCP environment.
Once the triggered scenario is completed, the test case processes the report and verifies that the scenario is completed with the excepted result: skip/fail/pass.
The tests are run on the OCP cluster with certain requirements that are listed below.
Conditions | Mandatory | |
---|---|---|
OCP Cluster | Version: >= 4.12, Node Count >= 3 with 2 cnf-worker nodes | Yes |
Installed Operators | Performance Addon, Machine-config-operator | Yes |
Machine config pool, PTP operator, SR-IOV operator | No |
Bare-minimum requirements consists of a OCP cluster with 3 nodes where 2 are cnf-worker nodes and 1 worker node.
The following test features are can run selectively or altogether.
- accesscontrol
- affiliatedcertification
- lifecycle
- manageability
- networking
- observability
- platformalteration
- performance
- operator
- preflight
Choose the variant that suits you best:
make test-features
- will only run tests for the features that were defined in theFEATURES
environment variablemake test-all
- will run the test suite for all features
The following environment variables are used to configure the test setup.
Env Variable Name | Purpose |
---|---|
FEATURES | To select the test scenarios that you are going to test, comma separated |
TNF_REPO_PATH | Points to the absolute path to cnf-certification-test on your machine |
TNF_IMAGE | Links to the TNF image. Default is quay.io/testnetworkfunction/cnf-certification-test |
TNF_IMAGE_TAG | image tag that is going to be tested. Default is latest |
TEST_IMAGE | Test image that is going to be used for all test resources such as deployments, daemonsets and so on. Default is quay.io/testnetworkfunction/cnf-test-partner |
DEBUG_TNF | Generate Debug folder that will contain TNF suites folders with TNF logs for each test. |
TNF_LOG_LEVEL | Log level. Default is 4 |
DISABLE_INTRUSIVE_TESTS | Turns off the intrusive tests for faster execution. Default is false . |
ENABLE_PARALLEL | Enable ginkgo -p parallel flags (experimental). Default is false . |
FORCE_DOWNLOAD_UNSTABLE | Force download the unstable image. Default is false . |
NON_LINUX_ENV | Allow the test suites to run in a non Linux environment. Default is false . |
Make sure docker or podman is running on the local machine.
Set your local container runtime to your environment with:
export TNF_CONTAINER_CLIENT=docker
git clone https://github.com/test-network-function/cnfcert-tests-verification.git
cd cnfcert-tests-verification
make install
- To run all tests
# Mac user
DOCKER_CONFIG_DIR=$HOME/.docker \
KUBECONFIG=$HOME/.kube/config \
NON_LINUX_ENV= \
TNF_REPO_PATH=$HOME/path/to/cnf-certification-test \
make test-all
# Linux user
KUBECONFIG=$HOME/.kube/config \
TNF_REPO_PATH=$HOME/path/to/cnf-certification-test \
make test-all
# Linux user with force download unstable image
\
FORCE_DOWNLOAD_UNSTABLE=true \
KUBECONFIG=$HOME/.kube/config \
TNF_REPO_PATH=$HOME/path/to/cnf-certification-test \
make test-all
- To run a specific test-suite:
# Mac user
DOCKER_CONFIG_DIR=$HOME/.docker \
FEATURES=platformalteration \
KUBECONFIG=$HOME/.kube/config \
NON_LINUX_ENV= \
TNF_REPO_PATH=$HOME/path/to/cnf-certification-test \
make test-features
# Linux user
FEATURES=platformalteration \
KUBECONFIG=$HOME/.kube/config \
DOCKER_CONFIG_DIR=$HOME/.docker \
TNF_REPO_PATH=$HOME/path/to/cnf-certification-test \
make test-features
- To debug
Use DEBUG_TNF=true
and TNF_LOG_LEVEL=debug
while running the above commands.
This would create a Debug
folder containing suites folders with TNF logs for each of the tests.
# Mac user
DEBUG_TNF=true \
DOCKER_CONFIG_DIR=$HOME/.docker \
FEATURES=platformalteration \
KUBECONFIG=$HOME/.kube/config \
NON_LINUX_ENV= \
TNF_LOG_LEVEL=debug \
TNF_REPO_PATH=$HOME/path/to/cnf-certification-test \
make test-features
# Linux user
DEBUG_TNF=true \
FEATURES=platformalteration \
KUBECONFIG=$HOME/.kube/config \
TNF_LOG_LEVEL=debug \
TNF_REPO_PATH=$HOME/path/to/cnf-certification-test \
make test-features
To execute the unit tests in the repository, run the following:
make test
- access-control-security-context
- affiliated-certification-container-is-certified-digest
- affiliated-certification-operator-is-certified
- platform-alteration-tainted-node-kernel
The QE repo is being used in nightly automated runs in the following files:
Fork the repo, create a new branch and create a PR with your changes.
CNFCert Tests Verification is copyright Red Hat, Inc. and available under an Apache 2 license.