Provision VMs for automated benchmark testing

.gitignore

@@ -15,4 +15,5 @@ __*
 scapy-script/*
 code_check/_checkpatch.pl
 .out
-test/benchmark_test/access_details.json
+test/benchmark_test/provision_tmpls
+test/benchmark_test/config_templates

docs/testing/automated_benchmark_test.md

@@ -1,55 +1,32 @@
 # When to perform automated benchmarking tests
 Automated benchmarking tests are additional functional and performance tests to the existing TAP device-based ones. This test suit relies on a configured environment including hypervisors and started VMs, as well as configured SSH authorized key of the execution machine starting the benchmarking tests. In the end, running these benchmarking tests is useful for verifying if dpservice works correctly together with actual running VMs for both offloading and non-offloading modes. It also verifies if networking performance meets specified values during dpservice development.
 
-# Required hypervisor and VM setup
-To successfully run these automated benchmarking tests, currently, 2 hypervisors and 3 VMs need to be prepared beforehand, especially putting the ssh key of the machine executing the benchmarking tests into the above mentioned hypervisors and VMs.
+# Required hypervisor setup
+To successfully run these automated benchmarking tests, currently, 2 hypervisors and 3 VMs need to be prepared beforehand.
+
+Please prepare the ssh private/public key pairs and put them under the `.ssh` directory of the server executing the provision script.
+
+The provided script, `hack/connectivity_test/prepare_hypervisor.sh`, can perform extra setups at one time. Please run this script on the involved servers, and the following manual steps can be ignored.
 
 ## Prerequisite
 
 1. Ensure the script execution machine can compile dpservice, especiall dpservice-cli within the directory.
 2. Install the following python libraries on your executing machine by executing
 ```
-apt install python3-termcolor
-apt install python3-psutil
-apt install python3-paramiko
+apt install -y python3-termcolor python3-psutil python3-paramiko python3-jinja2
 ```
 
-## Extra configuration on hypervisors running Gardenlinux
-On hypervisors running gardenlinux, it is also necessary to open ports to allow the DHCP service to provide IP addresses to VMs to be able for access. For example, the most convenient way is to  change the default input filter policy to 'accept' by importing the following nft table rules.
-
+## Configuration on hypervisors running Gardenlinux
+If the two Servers, that host VMs in tests, run Gardenlinux, and they require extra configurations so that provisioning and benchmarking tests can work.
 ```
-command: sudo nft -f filter_table.nft
-filter_table.nft:
-	table inet filter {
-			chain input {
-					type filter hook input priority filter; policy accept;
-					counter packets 1458372 bytes 242766426
-					iifname "lo" counter packets 713890 bytes 141369289 accept
-					ip daddr 127.0.0.1 counter packets 0 bytes 0 accept
-					icmp type echo-request limit rate 5/second burst 5 packets accept
-					ip6 saddr ::1 ip6 daddr ::1 counter packets 0 bytes 0 accept
-					icmpv6 type { echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept
-					ct state established,related counter packets 627814 bytes 93897896 accept
-					tcp dport 22 ct state new counter packets 362 bytes 23104 accept
-					rt type 0 counter packets 0 bytes 0 drop
-					meta l4proto ipv6-icmp counter packets 0 bytes 0 accept
-			}
-
-			chain forward {
-					type filter hook forward priority filter; policy accept;
-			}
-
-			chain output {
-					type filter hook output priority filter; policy accept;
-			}
-	}
+sudo nft add chain inet filter input '{ policy accept; }'
 ```
 
 Additionally, if the used hypervisors are running Gardenlinux, it is needed to remount `/tmp` to allow execute binary files being uploaded to it, due to the strict security policy. Simply execute `sudo mount -o remount,exec /tmp`.
 
 
-## Interface configuration in VMs
-To ssh into VMs, QEMU's default networking needs to be activated, and VMs need to be configured to have two interfaces, one using NIC's VF and one connecting to qemu's network bridge. Here is an example of the libvirt default networking configuration file.
+## Enable QEMU's default networking
+To ssh into VMs, QEMU's default networking needs to be activated and configured to support IP address assignment via DHCP. Enter the libvirt's default network editing mode by running `sudo virsh net-edit default`, copy the configureation and restart libvirt service by running `sudo systemctl restart libvirtd`.
 
 ```
 <network connections='1'>
@@ -70,54 +47,64 @@ To ssh into VMs, QEMU's default networking needs to be activated, and VMs need t
 </network>
 ```
 
-In order to add one extra interface dedicated for ssh connection, please modify the VM's libvirt configuration file in the format of XML and add the following section to setup an interface.
+The above steps are needed on hypervsiors to support automated provision of VMs and benchmark testing.
 
-```
-<interface type='network'>
-<mac address='52:54:00:eb:09:93'/>
-<source network='default'/>
-<model type='virtio'/>
-</interface>
-```
+# Provision VMs
+The script, `provision.py`, is able to create needed VMs according to the test_configuration.json file. This configuration file is copied into a newly created directory `/test/benchmark_test/provision_templates` and updated with VM's accessing IP address during the provision process. Right now, it provisions three VMs to meet the setup requirement of running benchmark tests.
+
+## Prepare Gardenlinux image (.raw)
+This step is manual, as the compilation of the kernel is time consuming and once it is done, it can be reused for quite some time. Two steps are needed to prepare the gardenlinux VM image.
 
-# Configuration file for test environment
-The configuration file, `/test/benchmark_test/test_configurations.json` for the test environment provides machine access information and the most of test configurations to the execution script. The following fields need to be double-checked and therefore changed according to the actual environment setup.
+1. Clone [Gardenlinux](https://github.com/gardenlinux/gardenlinux) source code using git.
+2. Inside the cloned repo, run `./build kvm-amd64`. The built image can be found under `./build/kvm-amd64-today-local.raw`, and remember the absolute path of this image file.
 
-1. "host_address", "user_name" and "port" fields in "hypervisors" and "vm" sections. They are needed to remotely access machines which are the foundations for the following operations.
+## Configuration file for test environment
+The configuration file, `/test/benchmark_test/config_templates/test_configurations.json` for the test environment provides machine access information and the most of test configurations to the execution script. The following fields need to be double-checked and therefore changed according to the actual environment setup.
+
+1. "host_address", "user_name" and "port" fields in "hypervisors" sections. They are needed to remotely access machines which are the foundations for the following operations.
 
 2. "expected_throughput" values need to adapted to the actual environment, as depending on the hardware capability, e.g., CPU speed and cabling specification, the maximum achievable throughput can be different. If these values are too high, tests will always fail.
 
 3. "pci_addr" in "vm" sections needs to match the VF and VM configuration on hypervisors.
 
 4. "machine_name" field is NOT expected to be changed.
 
+## Ignition file
+To have a complete ignition file template, `./benchmark_test/config_templates/provision_tmpl.ign`, please contact the maintainers for a proper hashed password to fill in.
+
+
+## Run the provision script
+The most commonly used commands to run the provision script are as follows.
+1. `./provision.py --disk-template <path to the compiled gardenlinux image file>`. For example, , e.g., `./provision.py --disk-template /home/gardenlinux/.build/kvm-amd64-today-local.raw`. It is expected that the defined VMs are provisioned on two hypervsiors, and their access IPs are updated in the `test_configurations.json` file.
+
+2. `./provision.py --clean-up`. It is expected that the provisioned VMs are destroyed and undefined.
+
 
 # Execution of test script
-This test suite is invoked by executing the script `runtest.py` under the repository `/test/benchmark_test`.
+This test suite is invoked by executing the script `runtest.py` under the repository `/test/benchmark_test`. In oder to run dpservice either natively or via container, please make sure that a valid dp_service.conf file is created under `/tmp`.
 
 ## dpservice-cli
 The testing script assumes that dpservice-cli exists under '/tmp' on hypervisors. If you have never run this test suite before, please first compile your local dpservice project by using `meson` and `ninja` commands. Because dpservice-cli is already included in the dpservice repository, the compiled dpservice-cli binary will be transferred to hypervisors automatically.
 
 ## Test script's parameters
 
 This script accepts several parameters, which are explained as follows.
-
 1. `--mode`. This option specifies which operation mode of dpservice needs to be tested. Select from 'offload', 'non-offload' and 'both'. It must be specified.
 
 2. `--stage`. This option specifies which testing stage needs to be used. Choose its value from 'dev' and 'cicd'. The stage of 'dev' is intended for carrying out tests during the development. If this option is set to 'dev', a docker image will be generated from the local repository of dpservice, and this image will be transferred to the hypervisors and executed. For example, a command like `./runtest.py --mode non-offloading --stage deploy -v` will achieve this purpose.
 Alternatively, if this option is set as 'cicd', the above described docker image generating process will not happen. Instead, a docker image specified by the option "--docker-image" will be used on hypervisors. This is a required option.
 
 3. `--docker-image`. This option specifies the container image to be deployed to hypervisors. It is optional but required for the 'cicd' stage.
 
-4. `--reboot`. This option specifies if a reboot process needs to be performed on VMs. It is needed if test configurations have changed, e.g., private IP addresses of VMs, and VMs need to obtain new configurations. If you want to ensure a fresh start of VMs, this option can also be enabled. It is optional.
+4. `--reboot`. This option specifies if a reboot process needs to be performed on VMs. It is needed if test configurations have changed, e.g., private IP addresses of VMs, and VMs need to obtain new configurations. If you want to ensure a fresh start of VMs, this option can also be enabled. It is optional, but it is recommended to set this flag so that each machine is able to receive newest interface configurations.
 
 5. `--env-config-file` and `--env-config-name`. They provide information of the above described `test_configurations.json`. It is possible this file is renamed or located somewhere else. And it is also possible to create several configurations within this file and specify one of them for the tests.
 
 6. `--verbose`. Specify if pytest runs in the verbose mode to see all steps and results during test execution.
 
 # Examplary command to invoke tests
 ```
-./run_benchmarktest.py --mode offload --stage cicd --docker-image ghcr.io/ironcore-dev/dpservice:sha-e9b4272 -v
+./run_benchmarktest.py --mode offload --stage cicd --docker-image ghcr.io/ironcore-dev/dpservice:sha-e9b4272 --reboot -v
 
-./run_benchmarktest.py --mode both --stage dev -v
+./run_benchmarktest.py --mode both --stage dev --reboot -v
 ```

hack/connectivity_test/prepare_hypervisor.sh

@@ -0,0 +1,68 @@
+#!/bin/bash
+
+# Ensure the script is run as root
+if [ "$EUID" -ne 0 ]; then
+  echo "Please run as root"
+  exit
+fi
+
+echo "Installing required Python libraries..."
+apt update
+apt install -y python3-termcolor python3-psutil python3-paramiko python3-jinja2
+
+echo "Checking for Gardenlinux-specific configuration..."
+
+# Check if the system is running Gardenlinux
+if grep -qi "gardenlinux" /etc/os-release; then
+  echo "Gardenlinux detected. Configuring firewall and remounting /tmp..."
+
+  # Apply the nft rules -- temporarily allow input traffics
+  sudo nft add chain inet filter input '{ policy accept; }'
+
+  # Remount /tmp with exec option
+  sudo mount -o remount,exec /tmp
+
+  sudo sysctl -w net.ipv4.ip_forward=1
+
+  echo "Gardenlinux-specific configuration completed."
+else
+  echo "Non-Gardenlinux system detected. Skipping Gardenlinux-specific configuration."
+fi
+
+# Define the XML configuration for the default network
+NETWORK_XML=$(cat <<EOF
+<network connections='1'>
+  <name>default</name>
+  <uuid>$(uuidgen)</uuid>
+  <forward mode='nat'>
+    <nat>
+      <port start='1024' end='65535'/>
+    </nat>
+  </forward>
+  <bridge name='virbr0' stp='on' delay='0'/>
+  <mac address='52:54:00:8c:3c:6f'/>
+  <ip address='192.168.122.1' netmask='255.255.255.0'>
+    <dhcp>
+      <range start='192.168.122.2' end='192.168.122.254'/>
+    </dhcp>
+  </ip>
+</network>
+EOF
+)
+
+# Backup the existing network configuration
+sudo cp /etc/libvirt/qemu/networks/default.xml /etc/libvirt/qemu/networks/default.xml.backup
+
+# Apply the new network configuration
+echo "$NETWORK_XML" | sudo tee /etc/libvirt/qemu/networks/default.xml > /dev/null
+
+# Restart the libvirt service
+sudo systemctl restart libvirtd
+
+# Start net default
+sudo virsh net-start default
+
+# Confirm the default network is active
+sudo virsh net-list --all
+
+echo "Script execution completed."

test/benchmark_test/benchmark_test_config.py

@@ -88,6 +88,7 @@ def init_vms(env_config, reboot_vm):
 					remote_machine_op_reboot(vm_info["machine_name"])
 				remote_machine_op_vm_config_rm_default_route(
 					vm_info["machine_name"])
+				remote_machine_op_vm_config_nft_default_accept(vm_info["machine_name"])
 				remote_machine_op_vm_config_tmp_dir(vm_info["machine_name"])
 				remote_machine_op_terminate_processes(vm_info["machine_name"])
 				remote_machine_op_upload(

test/benchmark_test/config_templates/provision_tmpl.ign

@@ -0,0 +1 @@
+{"ignition":{"version":"3.2.0"},"passwd":{"users":[{"name":"root","passwordHash":"PLEASE ASK MAINTAINER FOR A PROPER HASHEDPASS","sshAuthorizedKeys":["{{ pub_rsa_key }}"],"shell":"/bin/bash"}]},"systemd":{"units":[{"contents":"[Unit]\nDescription=Allow SSH Root Login\nAfter=network.target\n\n[Service]\nType=oneshot\nExecStart=/bin/bash -c \"sed -i 's/^#\\?PermitRootLogin .*/PermitRootLogin yes/' /etc/ssh/sshd_config\"\nExecStartPost=/bin/systemctl restart sshd\n\n[Install]\nWantedBy=multi-user.target\n","enabled":true,"name":"allowrootssh.service"},{"contents":"[Unit]\nDescription=Setup iperf3 on Debian\nAfter=network-online.target\nWants=network-online.target\n\n[Service]\nType=oneshot\nExecStart=/bin/bash -c 'echo \"deb http://deb.debian.org/debian/ bookworm main\" \u003e /etc/apt/sources.list.d/bookworm.list'\nExecStart=/usr/bin/apt-get update\nExecStart=/usr/bin/apt-get install -y iperf3 nftables\nExecStart=/bin/rm /etc/apt/sources.list.d/bookworm.list\nExecStart=/usr/bin/apt-get update\nRemainAfterExit=true\n\n[Install]\nWantedBy=multi-user.target\n","enabled":true,"name":"setup-iperf3.service"}]}}

test/benchmark_test/config_templates/test_configurations.json

@@ -0,0 +1,77 @@
+{
+	"key_file": "~/.ssh/id_rsa",
+	"public_key_file": "~/.ssh/id_rsa.pub",
+	"default_dpservice_image": "ghcr.io/ironcore-dev/dpservice:sha-e9b4272",
+	"concurrent_flow_count": 3,
+	"expected_throughput": {
+		"sw": {
+			"local_vm2vm": 10,
+			"remote_vm2vm": 8,
+			"lb": 5
+		},
+		"hw": {
+			"local_vm2vm": 20,
+			"remote_vm2vm": 20,
+			"lb": 12
+		}
+	},
+	"hypervisors": [
+		{
+			"machine_name": "hypervisor-1",
+			"host_address": "192.168.23.166",
+			"user_name": "",
+			"port": 22,
+			"vms": [
+				{
+					"machine_name": "vm1",
+					"if_config":{
+						"ipv4": "192.168.129.5",
+						"ipv6": "2002::123",
+						"vni": 66,
+						"pci_addr": "0000:8a:00.0_representor_vf2"
+					}
+				},
+				{
+					"machine_name": "vm2",
+					"if_config":{
+						"ipv4": "192.168.129.6",
+						"ipv6": "2002::124",
+						"vni": 66,
+						"pci_addr": "0000:8a:00.0_representor_vf1"
+					}
+				}
+			]
+		},
+		{
+			"machine_name": "hypervisor-2",
+			"role": "local",
+			"host_address": "192.168.23.86",
+			"user_name": "",
+			"port": 22,
+			"vms": [
+				{
+					"machine_name": "vm3",
+					"if_config":{
+						"ipv4": "172.32.4.9",
+						"ipv6": "2003::123",
+						"vni": 66,
+						"pci_addr": "0000:3b:00.0_representor_vf0"
+					},
+					"nat": {
+						"ip": "10.10.20.20",
+						"ports": [10240, 10360]
+					}
+				}
+			]
+		}
+	],
+	"lb": {
+		"name": "test_lb",
+		"ip": "10.20.30.30",
+		"ports": "TCP/5201,TCP/50007",
+		"vni": 66,
+		"lb_nodes": ["hypervisor-2"],
+		"lb_machines": ["vm3"]
+	}
+}
+

test/benchmark_test/config_templates/vm_tmpl.xml

@@ -0,0 +1,78 @@
+<domain type='kvm'>
+  <name>{{ VM_NAME }}</name>
+  <metadata>
+    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
+      <libosinfo:os id="https://gardenlinux.io"/>
+    </libosinfo:libosinfo>
+  </metadata>
+  <memory unit='GB'>8</memory>
+  <currentMemory unit='GB'>8</currentMemory>
+  <vcpu placement='static'>4</vcpu>
+  <os>
+    <type arch='x86_64' machine='pc-q35-5.2'>hvm</type>
+    <boot dev='hd'/>
+  </os>
+  <features>
+    <acpi/>
+    <apic/>
+  </features>
+  <sysinfo type='fwcfg'>
+	  <entry name='opt/com.coreos/config' file='{{ IGNITION_FILE }}'/>
+  </sysinfo>
+  <cpu mode='host-model' check='partial'/>
+  <clock offset='utc'>
+    <timer name='rtc' tickpolicy='catchup'/>
+    <timer name='pit' tickpolicy='delay'/>
+    <timer name='hpet' present='no'/>
+  </clock>
+  <on_poweroff>destroy</on_poweroff>
+  <on_reboot>restart</on_reboot>
+  <on_crash>destroy</on_crash>
+  <pm>
+    <suspend-to-mem enabled='no'/>
+    <suspend-to-disk enabled='no'/>
+  </pm>
+  <devices>
+    <emulator>/usr/bin/qemu-system-x86_64</emulator>
+    <hostdev mode="subsystem" type="pci" managed="yes">
+      <driver name="vfio"/>
+      <source>
+        <address {{ VF_PCI_ADDRESS }} />
+      </source>
+      <alias name="hostdev0"/>
+      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
+    </hostdev>
+    <disk type='file' device='disk'>
+      <driver name='qemu' type='raw'/>
+      <source file='{{ DISK_IMAGE }}' index='1'/>
+      <backingStore/>
+      <target dev='vda' bus='virtio'/>
+      <alias name='virtio-disk0'/>
+    </disk>
+    <controller type='virtio-serial' index='0'>
+      <alias name='virtio-serial0'/>
+    </controller>
+    <controller type='usb' index='0' model='qemu-xhci'>
+      <alias name='usb'/>
+    </controller>
+    <interface type='network'>
+      <mac address='{{ BRIDGE_IFACE_MAC }}' />
+      <source network='default'/>
+      <model type='virtio'/>
+    </interface>
+      <serial type='pty'>
+      <target type='isa-serial' port='0'>
+        <model name='isa-serial'/>
+      </target>
+    </serial>
+    <input type='mouse' bus='ps2'/>
+    <input type='keyboard' bus='ps2'/>
+    <memballoon model='virtio'>
+      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
+    </memballoon>
+    <rng model='virtio'>
+      <backend model='random'>/dev/urandom</backend>
+    </rng>
+  </devices>
+</domain>
+