Skip to content

Latest commit

 

History

History
367 lines (296 loc) · 17.2 KB

ci.md

File metadata and controls

367 lines (296 loc) · 17.2 KB
Raw

CI Tests

For CI, OVN-Kubernetes runs the Kubernetes E2E tests and some locally defined tests. GitHub Actions are used to run a subset of the Kubernetes E2E tests on each pull request. The local workflow that controls the test run is located in ovn-kubernetes/.github/workflows/test.yml.

The following tasks are performed:

  • Build OVN-Kubernetes
  • Check out the Kubernetes source tree and compiles some dependencies
  • Install KIND
  • Run a matrix of End-To-End Tests using KIND

The following sections should help you understand (and if needed modify) the set of tests that run and how to run these tests locally.

Understanding the CI Test Suite

The tests are broken into 2 categories, shard tests which execute tests from the Kubernetes E2E test suite and the control-plane tests which run locally defined tests.

Shard tests

The shard tests are broken into a set of shards, which is just a grouping of tests, and each shard is run in a separate job in parallel. Shards execute the shard-% target in ovn-kubernetes/test/Makefile. The set of shards may change in the future. Below is an example of the shards at time of this writing:

  • shard-network
    • All E2E tests that match [sig-network]
  • shard-conformance
    • All E2E tests that match [Conformance]|[sig-network]
  • shard-test
    • Single E2E test that matches the name of the test specified with a regex.
    • When selecting the shard-test target, you focus on a specific test by appending WHAT=<test name> to the make command.
    • See bottom of this document for an example.

Shards use the E2E framework. By selecting a specific shard, you modify ginkgo's --focus parameter.

The regex expression for determining which E2E test is run in which shard, as well as the list of skipped tests is defined in ovn-kubernetes/test/scripts/e2e-kind.sh.

Control-plane tests

In addition to the shard-% tests, there is also a control-plane target in ovn-kubernetes/test/Makefile. Below is a description of this target:

  • control-plane
    • All locally defined tests by default.
    • You can focus on a specific test by appending WHAT=<test name> to the make command.
    • See bottom of this document for an example.

All local tests are run by make control-plane. The local tests are controlled in ovn-kubernetes/test/scripts/e2e-cp.sh and the actual tests are defined in the directory ovn-kubernetes/test/e2e/.

Node IP migration tests

The node IP migration tests are part of the control-plane tests but due to their impact they cannot be run concurrently with other tests and they are disabled when running make control-plane. Instead, they must explicitly be requested with make -C test control-plane WHAT="Node IP address migration".

Github CI integration through Github Actions Matrix

Each of these shards and control-plane tests can then be run in a Github Actions matrix of:

  • HA setup (3 masters and 0 workers) and a non-HA setup (1 master and 2 workers)
  • Local Gateway Mode and Shared Gateway Mode. See: Enable Node-Local Services Access in Shared Gateway Mode
  • IPv4 Only, IPv6 Only and Dualstack
  • Disabled SNAT Multiple Gateways or Enabled SNAT Gateways
  • Single bridge or two bridges

To reduce the explosion of tests being run in CI, the test cases run are limited using an exclude: statement in ovn-kubernetes/.github/workflows/test.yml.

Running CI Locally

This section describes how to run CI tests on a local deployment. This may be useful for expanding the CI test coverage or testing a private fix before creating a pull request.

Download and Build Kubernetes Components

Go Version

Older versions of Kubernetes do not build with newer versions of Go, specifically Kubernetes v1.16.4 doesn't build with Go version 1.13.x. If this is a version of Kubernetes that needs to be tested with, as a workaround, Go version 1.12.1 can be downloaded to a local directory and the $PATH variable updated only where kubernetes is being built.

$ go version
go version go1.13.8 linux/amd64

$ mkdir -p /home/$USER/src/golang/go1-12-1/; cd /home/$USER/src/golang/go1-12-1/
$ wget https://dl.google.com/go/go1.12.1.linux-amd64.tar.gz
$ tar -xzf go1.12.1.linux-amd64.tar.gz
$ PATH=/home/$USER/src/golang/go1-12-1/go/bin:$GOPATH/src/k8s.io/kubernetes/_output/local/bin/linux/amd64:$PATH

Download and Build Kubernetes Components (E2E Tests, ginkgo, kubectl):

Determine which version of Kubernetes is currently used in CI (See ovn-kubernetes/.github/workflows/test.yml) and set the environmental variable K8S_VERSION to the same value. Also make sure to export a GOPATH which points to your go directory with export GOPATH=(...).

K8S_VERSION=v1.28.0
git clone --single-branch --branch $K8S_VERSION https://github.com/kubernetes/kubernetes.git $GOPATH/src/k8s.io/kubernetes/
pushd $GOPATH/src/k8s.io/kubernetes/
make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl"
rm -rf .git

sudo cp _output/local/go/bin/e2e.test /usr/local/bin/.
sudo cp _output/local/go/bin/kubectl /usr/local/bin/kubectl-$K8S_VERSION
sudo ln -s /usr/local/bin/kubectl-$K8S_VERSION /usr/local/bin/kubectl
cp _output/local/go/bin/ginkgo /usr/local/bin/.
popd

If you have any failures during the build, verify $PATH has been updated to point to correct GO version. Also may need to change settings on some of the generated binaries. For example:

chmod +x $GOPATH/src/k8s.io/kubernetes/_output/bin/deepcopy-gen

Export environment variables

Before setting up KIND and before running the actual tests, export essential environment variables.

The environment variables and their values depend on the actual test scenario that you want to run.

Look at the e2e action (search for name: e2e) in ovn-kubernetes/.github/workflows/test.yml. Prior to installing kind, set the following environment variables according to your needs:

export KIND_CLUSTER_NAME=ovn
export KIND_INSTALL_INGRESS=[true|false]
export KIND_ALLOW_SYSTEM_WRITES=[true|false]
export PARALLEL=[true|false]
export JOB_NAME=(... job name ...)
export OVN_HYBRID_OVERLAY_ENABLE=[true|false]
export OVN_MULTICAST_ENABLE=[true|false]
export OVN_EMPTY_LB_EVENTS=[true|false]
export OVN_HA=[true|false]
export OVN_DISABLE_SNAT_MULTIPLE_GWS=[true|false]
export OVN_GATEWAY_MODE=["local"|"shared"]
export KIND_IPV4_SUPPORT=[true|false]
export KIND_IPV6_SUPPORT=[true|false]
# not required for the OVN Kind installation script, but export this already for later
OVN_SECOND_BRIDGE=[true|false]

You can refer to a recent CI run from any pull request in https://github.com/ovn-org/ovn-kubernetes/actions to get a valid set of settings.

As an example for the control-plane-noHA-local-ipv4-snatGW-1br job, the settings are at time of this writing:

export KIND_CLUSTER_NAME=ovn
export KIND_INSTALL_INGRESS=true
export KIND_ALLOW_SYSTEM_WRITES=true
export PARALLEL=true
export JOB_NAME=control-plane-noHA-local-ipv4-snatGW-1br
export OVN_HYBRID_OVERLAY_ENABLE=true
export OVN_MULTICAST_ENABLE=true
export OVN_EMPTY_LB_EVENTS=true
export OVN_HA=false
export OVN_DISABLE_SNAT_MULTIPLE_GWS=false
export OVN_GATEWAY_MODE="local"
export KIND_IPV4_SUPPORT=true
export KIND_IPV6_SUPPORT=false
# not required for the OVN Kind installation script, but export this already for later
export OVN_SECOND_BRIDGE=false

KIND

Kubernetes in Docker (KIND) is used to deploy Kubernetes locally where a docker container is created per Kubernetes node. The CI tests run on this Kubernetes deployment. Therefore, KIND will need to be installed locally.

Generic instructions for installing and running OVNKubernetes with KIND can be found at: https://github.com/ovn-org/ovn-kubernetes/blob/master/docs/kind.md

Make sure to set the required environment variables first (see section above). Then, deploy kind:

$ pushd
$ ./kind.sh
$ popd

Run Tests

To run the tests locally, run a KIND deployment as described above. The E2E tests look for the kube config file in a special location, so make a copy:

cp ~/ovn.conf ~/.kube/kind-config-kind

To run the desired shard, first make sure that the necessary environment variables are exported (see section above). Then, go to the location of your local copy of the ovn-kubernetes repository:

$ REPO=$GOPATH/src/github.com/ovn-org/ovn-kubernetes
$ cd $REPO

Running a suite of shards or control-plane tests

Finally, run the the shard that you want to test against (each shard can take 30+ minutes to complete)

$ pushd test
# run either
$ make shard-network
# or
$ make shard-conformance
# or
$ GITHUB_WORKSPACE="$REPO" make control-plane
# or
$ make conformance
$ popd

Running a single E2E test

To run a single E2E test instead, target the shard-test action, as follows:

$ cd $REPO
$ pushd test
$ make shard-test WHAT="should enforce egress policy allowing traffic to a server in a different namespace based on PodSelector and NamespaceSelector"
$ popd

As a reminder, shards use the E2E framework. The value of WHAT= will be used to modify the --focus parameter. Individual tests can be retrieved from https://github.com/kubernetes/kubernetes/tree/master/test/e2e. For network tests, one could run:

grep ginkgo.It $GOPATH/src/k8s.io/kubernetes/test/e2e/network/ -Ri

For example:

# grep ginkgo.It $GOPATH/src/k8s.io/kubernetes/test/e2e/network/ -Ri | head -1
/root/go/src/k8s.io/kubernetes/test/e2e/network/conntrack.go:	ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() {
# make shard-test WHAT="should enforce policy to allow traffic from pods within server namespace based on PodSelector"
(...)
+ case "$SHARD" in
++ echo should be able to preserve UDP traffic when server pod cycles for a NodePort service
++ sed 's/ /\\s/g'
+ FOCUS='should\sbe\sable\sto\spreserve\sUDP\straffic\swhen\sserver\spod\scycles\sfor\sa\sNodePort\sservice'
+ export KUBERNETES_CONFORMANCE_TEST=y
+ KUBERNETES_CONFORMANCE_TEST=y
+ export KUBE_CONTAINER_RUNTIME=remote
+ KUBE_CONTAINER_RUNTIME=remote
+ export KUBE_CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
+ KUBE_CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock
+ export KUBE_CONTAINER_RUNTIME_NAME=containerd
+ KUBE_CONTAINER_RUNTIME_NAME=containerd
+ export FLAKE_ATTEMPTS=5
+ FLAKE_ATTEMPTS=5
+ export NUM_NODES=20
+ NUM_NODES=20
+ export NUM_WORKER_NODES=3
+ NUM_WORKER_NODES=3
+ ginkgo --nodes=20 '--focus=should\sbe\sable\sto\spreserve\sUDP\straffic\swhen\sserver\spod\scycles\sfor\sa\sNodePort\sservice' '--skip=Networking\sIPerf\sIPv[46]|\[Feature:PerformanceDNS\]|Disruptive|DisruptionController|\[sig-apps\]\sCronJob|\[sig-storage\]|\[Feature:Federation\]|should\shave\sipv4\sand\sipv6\sinternal\snode\sip|should\shave\sipv4\sand\sipv6\snode\spodCIDRs|kube-proxy|should\sset\sTCP\sCLOSE_WAIT\stimeout|should\shave\ssession\saffinity\stimeout\swork|named\sport.+\[Feature:NetworkPolicy\]|\[Feature:SCTP\]|service.kubernetes.io/headless|should\sresolve\sconnection\sreset\sissue\s#74839|sig-api-machinery|\[Feature:NoSNAT\]|Services.+(ESIPP|cleanup\sfinalizer)|configMap\snameserver|ClusterDns\s\[Feature:Example\]|should\sset\sdefault\svalue\son\snew\sIngressClass|should\sprevent\sIngress\screation\sif\smore\sthan\s1\sIngressClass\smarked\sas\sdefault|\[Feature:Networking-IPv6\]|\[Feature:.*DualStack.*\]' --flake-attempts=5 /usr/local/bin/e2e.test -- --kubeconfig=/root/ovn.conf --provider=local --dump-logs-on-failure=false --report-dir=/root/ovn-kubernetes/test/_artifacts --disable-log-dump=true --num-nodes=3
Running Suite: Kubernetes e2e suite
(...)
• [SLOW TEST:28.091 seconds]
[sig-network] Conntrack
/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:23
  should be able to preserve UDP traffic when server pod cycles for a NodePort service
  /root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/conntrack.go:128
------------------------------
{"msg":"PASSED [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a NodePort service","total":-1,"completed":1,"skipped":229,"failed":0}
Aug 17 14:46:42.842: INFO: Running AfterSuite actions on all nodes


Aug 17 14:46:15.264: INFO: Running AfterSuite actions on all nodes
Aug 17 14:46:42.885: INFO: Running AfterSuite actions on node 1
Aug 17 14:46:42.885: INFO: Skipping dumping logs from cluster


Ran 1 of 5667 Specs in 30.921 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Flaked | 0 Pending | 5666 Skipped


Ginkgo ran 1 suite in 38.489055861s
Test Suite Passed

Running a control-plane test

All local tests are defined as control-plane tests. To run a single control-plane test, target the control-plane action and append the WHAT=<test name> parameter, as follows:

$ cd $REPO
$ pushd test
$ make control-plane WHAT="should be able to send multicast UDP traffic between nodes"
$ popd

The value of WHAT= will be used to modify the -ginkgo.focus parameter. Individual tests can be retrieved from this repository under test/e2e. To see a list of individual tests, one could run:

grep -R ginkgo.It test/

For example:

# grep -R ginkgo.It . | head -1
./e2e/multicast.go:	ginkgo.It("should be able to send multicast UDP traffic between nodes", func() {
# make control-plane WHAT="should be able to send multicast UDP traffic between nodes"
(...)
+ go test -timeout=0 -v . -ginkgo.v -ginkgo.focus 'should\sbe\sable\sto\ssend\smulticast\sUDP\straffic\sbetween\snodes' -ginkgo.flakeAttempts 2 '-ginkgo.skip=recovering from deleting db files while maintain connectivity|Should validate connectivity before and after deleting all the db-pods at once in HA mode|Should be allowed to node local cluster-networked endpoints by nodeport services with externalTrafficPolicy=local|e2e ingress to host-networked pods traffic validation|host to host-networked pods traffic validation' -provider skeleton -kubeconfig /root/ovn.conf --num-nodes=2 --report-dir=/root/ovn-kubernetes/test/_artifacts --report-prefix=control-plane_
I0817 15:26:21.762483 1197731 test_context.go:457] Tolerating taints "node-role.kubernetes.io/control-plane" when considering if nodes are ready
=== RUN   TestE2e
I0817 15:26:21.762635 1197731 e2e_suite_test.go:67] Saving reports to /root/ovn-kubernetes/test/_artifacts
Running Suite: E2e Suite
(...)
• [SLOW TEST:12.332 seconds]
Multicast
/root/ovn-kubernetes/test/e2e/multicast.go:25
  should be able to send multicast UDP traffic between nodes
  /root/ovn-kubernetes/test/e2e/multicast.go:75
------------------------------
SSSSSSSSSSSS
JUnit report was created: /root/ovn-kubernetes/test/_artifacts/junit_control-plane_01.xml

Ran 1 of 60 Specs in 12.333 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Flaked | 0 Pending | 59 Skipped
--- PASS: TestE2e (12.34s)
PASS
ok  	github.com/ovn-org/ovn-kubernetes/test/e2e	12.371s
+ popd
~/ovn-kubernetes/test

IPv6 tests

To skip the IPv4 only tests (in a IPv6 only deployment), pass the KIND_IPV6_SUPPORT=true environmental variable to make:

$ cd $GOPATH/src/github.com/ovn-org/ovn-kubernetes

$ pushd test
$ KIND_IPV6_SUPPORT=true make shard-conformance
$ popd

Github CI doesn´t offer IPv6 connectivity, so IPv6 only tests are always skipped. To run those tests locally, comment out the following line from ovn-kubernetes/test/scripts/e2e-kind.sh

# Github CI doesn´t offer IPv6 connectivity, so always skip IPv6 only tests.
SKIPPED_TESTS=$SKIPPED_TESTS$IPV6_ONLY_TESTS

Conformance Tests

We have a conformance test suit that can be invoked using the make conformance command. Currently we run the TestNetworkPolicyV2Conformance tests there. The actual tests are defined in https://github.com/kubernetes-sigs/network-policy-api/tree/master/conformance and then invoked from this repo. Any changes to the tests first have to be submitted upstream to network-policy-api repo and then brought downstream into the ovn-kubernetes repo through version bump.