diff --git a/.gitignore b/.gitignore
index 5e70dcf02..06bc56f53 100644
--- a/.gitignore
+++ b/.gitignore
@@ -15,4 +15,5 @@ __*
scapy-script/*
code_check/_checkpatch.pl
.out
-test/benchmark_test/access_details.json
+test/benchmark_test/provision_tmpls
+test/benchmark_test/config_templates
diff --git a/docs/testing/automated_benchmark_test.md b/docs/testing/automated_benchmark_test.md
index 8e4cea781..5c0a621b2 100644
--- a/docs/testing/automated_benchmark_test.md
+++ b/docs/testing/automated_benchmark_test.md
@@ -1,55 +1,32 @@
# When to perform automated benchmarking tests
Automated benchmarking tests are additional functional and performance tests to the existing TAP device-based ones. This test suit relies on a configured environment including hypervisors and started VMs, as well as configured SSH authorized key of the execution machine starting the benchmarking tests. In the end, running these benchmarking tests is useful for verifying if dpservice works correctly together with actual running VMs for both offloading and non-offloading modes. It also verifies if networking performance meets specified values during dpservice development.
-# Required hypervisor and VM setup
-To successfully run these automated benchmarking tests, currently, 2 hypervisors and 3 VMs need to be prepared beforehand, especially putting the ssh key of the machine executing the benchmarking tests into the above mentioned hypervisors and VMs.
+# Required hypervisor setup
+To successfully run these automated benchmarking tests, currently, 2 hypervisors and 3 VMs need to be prepared beforehand.
+
+Please prepare the ssh private/public key pairs and put them under the `.ssh` directory of the server executing the provision script.
+
+The provided script, `hack/connectivity_test/prepare_hypervisor.sh`, can perform extra setups at one time. Please run this script on the involved servers, and the following manual steps can be ignored.
## Prerequisite
1. Ensure the script execution machine can compile dpservice, especiall dpservice-cli within the directory.
2. Install the following python libraries on your executing machine by executing
```
-apt install python3-termcolor
-apt install python3-psutil
-apt install python3-paramiko
+apt install -y python3-termcolor python3-psutil python3-paramiko python3-jinja2
```
-## Extra configuration on hypervisors running Gardenlinux
-On hypervisors running gardenlinux, it is also necessary to open ports to allow the DHCP service to provide IP addresses to VMs to be able for access. For example, the most convenient way is to change the default input filter policy to 'accept' by importing the following nft table rules.
-
+## Configuration on hypervisors running Gardenlinux
+If the two Servers, that host VMs in tests, run Gardenlinux, and they require extra configurations so that provisioning and benchmarking tests can work.
```
-command: sudo nft -f filter_table.nft
-filter_table.nft:
- table inet filter {
- chain input {
- type filter hook input priority filter; policy accept;
- counter packets 1458372 bytes 242766426
- iifname "lo" counter packets 713890 bytes 141369289 accept
- ip daddr 127.0.0.1 counter packets 0 bytes 0 accept
- icmp type echo-request limit rate 5/second burst 5 packets accept
- ip6 saddr ::1 ip6 daddr ::1 counter packets 0 bytes 0 accept
- icmpv6 type { echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept
- ct state established,related counter packets 627814 bytes 93897896 accept
- tcp dport 22 ct state new counter packets 362 bytes 23104 accept
- rt type 0 counter packets 0 bytes 0 drop
- meta l4proto ipv6-icmp counter packets 0 bytes 0 accept
- }
-
- chain forward {
- type filter hook forward priority filter; policy accept;
- }
-
- chain output {
- type filter hook output priority filter; policy accept;
- }
- }
+sudo nft add chain inet filter input '{ policy accept; }'
```
Additionally, if the used hypervisors are running Gardenlinux, it is needed to remount `/tmp` to allow execute binary files being uploaded to it, due to the strict security policy. Simply execute `sudo mount -o remount,exec /tmp`.
-## Interface configuration in VMs
-To ssh into VMs, QEMU's default networking needs to be activated, and VMs need to be configured to have two interfaces, one using NIC's VF and one connecting to qemu's network bridge. Here is an example of the libvirt default networking configuration file.
+## Enable QEMU's default networking
+To ssh into VMs, QEMU's default networking needs to be activated and configured to support IP address assignment via DHCP. Enter the libvirt's default network editing mode by running `sudo virsh net-edit default`, copy the configureation and restart libvirt service by running `sudo systemctl restart libvirtd`.
```
@@ -70,20 +47,21 @@ To ssh into VMs, QEMU's default networking needs to be activated, and VMs need t
```
-In order to add one extra interface dedicated for ssh connection, please modify the VM's libvirt configuration file in the format of XML and add the following section to setup an interface.
+The above steps are needed on hypervsiors to support automated provision of VMs and benchmark testing.
-```
-
-
-
-
-
-```
+# Provision VMs
+The script, `provision.py`, is able to create needed VMs according to the test_configuration.json file. This configuration file is copied into a newly created directory `/test/benchmark_test/provision_templates` and updated with VM's accessing IP address during the provision process. Right now, it provisions three VMs to meet the setup requirement of running benchmark tests.
+
+## Prepare Gardenlinux image (.raw)
+This step is manual, as the compilation of the kernel is time consuming and once it is done, it can be reused for quite some time. Two steps are needed to prepare the gardenlinux VM image.
-# Configuration file for test environment
-The configuration file, `/test/benchmark_test/test_configurations.json` for the test environment provides machine access information and the most of test configurations to the execution script. The following fields need to be double-checked and therefore changed according to the actual environment setup.
+1. Clone [Gardenlinux](https://github.com/gardenlinux/gardenlinux) source code using git.
+2. Inside the cloned repo, run `./build kvm-amd64`. The built image can be found under `./build/kvm-amd64-today-local.raw`, and remember the absolute path of this image file.
-1. "host_address", "user_name" and "port" fields in "hypervisors" and "vm" sections. They are needed to remotely access machines which are the foundations for the following operations.
+## Configuration file for test environment
+The configuration file, `/test/benchmark_test/config_templates/test_configurations.json` for the test environment provides machine access information and the most of test configurations to the execution script. The following fields need to be double-checked and therefore changed according to the actual environment setup.
+
+1. "host_address", "user_name" and "port" fields in "hypervisors" sections. They are needed to remotely access machines which are the foundations for the following operations.
2. "expected_throughput" values need to adapted to the actual environment, as depending on the hardware capability, e.g., CPU speed and cabling specification, the maximum achievable throughput can be different. If these values are too high, tests will always fail.
@@ -91,9 +69,19 @@ The configuration file, `/test/benchmark_test/test_configurations.json` for the
4. "machine_name" field is NOT expected to be changed.
+## Ignition file
+To have a complete ignition file template, `./benchmark_test/config_templates/provision_tmpl.ign`, please contact the maintainers for a proper hashed password to fill in.
+
+
+## Run the provision script
+The most commonly used commands to run the provision script are as follows.
+1. `./provision.py --disk-template `. For example, , e.g., `./provision.py --disk-template /home/gardenlinux/.build/kvm-amd64-today-local.raw`. It is expected that the defined VMs are provisioned on two hypervsiors, and their access IPs are updated in the `test_configurations.json` file.
+
+2. `./provision.py --clean-up`. It is expected that the provisioned VMs are destroyed and undefined.
+
# Execution of test script
-This test suite is invoked by executing the script `runtest.py` under the repository `/test/benchmark_test`.
+This test suite is invoked by executing the script `runtest.py` under the repository `/test/benchmark_test`. In oder to run dpservice either natively or via container, please make sure that a valid dp_service.conf file is created under `/tmp`.
## dpservice-cli
The testing script assumes that dpservice-cli exists under '/tmp' on hypervisors. If you have never run this test suite before, please first compile your local dpservice project by using `meson` and `ninja` commands. Because dpservice-cli is already included in the dpservice repository, the compiled dpservice-cli binary will be transferred to hypervisors automatically.
@@ -101,7 +89,6 @@ The testing script assumes that dpservice-cli exists under '/tmp' on hypervisors
## Test script's parameters
This script accepts several parameters, which are explained as follows.
-
1. `--mode`. This option specifies which operation mode of dpservice needs to be tested. Select from 'offload', 'non-offload' and 'both'. It must be specified.
2. `--stage`. This option specifies which testing stage needs to be used. Choose its value from 'dev' and 'cicd'. The stage of 'dev' is intended for carrying out tests during the development. If this option is set to 'dev', a docker image will be generated from the local repository of dpservice, and this image will be transferred to the hypervisors and executed. For example, a command like `./runtest.py --mode non-offloading --stage deploy -v` will achieve this purpose.
@@ -109,7 +96,7 @@ Alternatively, if this option is set as 'cicd', the above described docker image
3. `--docker-image`. This option specifies the container image to be deployed to hypervisors. It is optional but required for the 'cicd' stage.
-4. `--reboot`. This option specifies if a reboot process needs to be performed on VMs. It is needed if test configurations have changed, e.g., private IP addresses of VMs, and VMs need to obtain new configurations. If you want to ensure a fresh start of VMs, this option can also be enabled. It is optional.
+4. `--reboot`. This option specifies if a reboot process needs to be performed on VMs. It is needed if test configurations have changed, e.g., private IP addresses of VMs, and VMs need to obtain new configurations. If you want to ensure a fresh start of VMs, this option can also be enabled. It is optional, but it is recommended to set this flag so that each machine is able to receive newest interface configurations.
5. `--env-config-file` and `--env-config-name`. They provide information of the above described `test_configurations.json`. It is possible this file is renamed or located somewhere else. And it is also possible to create several configurations within this file and specify one of them for the tests.
@@ -117,7 +104,7 @@ Alternatively, if this option is set as 'cicd', the above described docker image
# Examplary command to invoke tests
```
-./run_benchmarktest.py --mode offload --stage cicd --docker-image ghcr.io/ironcore-dev/dpservice:sha-e9b4272 -v
+./run_benchmarktest.py --mode offload --stage cicd --docker-image ghcr.io/ironcore-dev/dpservice:sha-e9b4272 --reboot -v
-./run_benchmarktest.py --mode both --stage dev -v
+./run_benchmarktest.py --mode both --stage dev --reboot -v
```
diff --git a/hack/connectivity_test/prepare_hypervisor.sh b/hack/connectivity_test/prepare_hypervisor.sh
new file mode 100755
index 000000000..3394c0dae
--- /dev/null
+++ b/hack/connectivity_test/prepare_hypervisor.sh
@@ -0,0 +1,68 @@
+#!/bin/bash
+
+# Ensure the script is run as root
+if [ "$EUID" -ne 0 ]; then
+ echo "Please run as root"
+ exit
+fi
+
+echo "Installing required Python libraries..."
+apt update
+apt install -y python3-termcolor python3-psutil python3-paramiko python3-jinja2
+
+echo "Checking for Gardenlinux-specific configuration..."
+
+# Check if the system is running Gardenlinux
+if grep -qi "gardenlinux" /etc/os-release; then
+ echo "Gardenlinux detected. Configuring firewall and remounting /tmp..."
+
+ # Apply the nft rules -- temporarily allow input traffics
+ sudo nft add chain inet filter input '{ policy accept; }'
+
+ # Remount /tmp with exec option
+ sudo mount -o remount,exec /tmp
+
+ sudo sysctl -w net.ipv4.ip_forward=1
+
+ echo "Gardenlinux-specific configuration completed."
+else
+ echo "Non-Gardenlinux system detected. Skipping Gardenlinux-specific configuration."
+fi
+
+# Define the XML configuration for the default network
+NETWORK_XML=$(cat <
+ default
+ $(uuidgen)
+
+
+
+
+
+
+
+
+
+
+
+
+
+EOF
+)
+
+# Backup the existing network configuration
+sudo cp /etc/libvirt/qemu/networks/default.xml /etc/libvirt/qemu/networks/default.xml.backup
+
+# Apply the new network configuration
+echo "$NETWORK_XML" | sudo tee /etc/libvirt/qemu/networks/default.xml > /dev/null
+
+# Restart the libvirt service
+sudo systemctl restart libvirtd
+
+# Start net default
+sudo virsh net-start default
+
+# Confirm the default network is active
+sudo virsh net-list --all
+
+echo "Script execution completed."
diff --git a/test/benchmark_test/benchmark_test_config.py b/test/benchmark_test/benchmark_test_config.py
index 04d12ae86..547bf08a6 100644
--- a/test/benchmark_test/benchmark_test_config.py
+++ b/test/benchmark_test/benchmark_test_config.py
@@ -88,6 +88,7 @@ def init_vms(env_config, reboot_vm):
remote_machine_op_reboot(vm_info["machine_name"])
remote_machine_op_vm_config_rm_default_route(
vm_info["machine_name"])
+ remote_machine_op_vm_config_nft_default_accept(vm_info["machine_name"])
remote_machine_op_vm_config_tmp_dir(vm_info["machine_name"])
remote_machine_op_terminate_processes(vm_info["machine_name"])
remote_machine_op_upload(
diff --git a/test/benchmark_test/config_templates/provision_tmpl.ign b/test/benchmark_test/config_templates/provision_tmpl.ign
new file mode 100644
index 000000000..2d7ebed3a
--- /dev/null
+++ b/test/benchmark_test/config_templates/provision_tmpl.ign
@@ -0,0 +1 @@
+{"ignition":{"version":"3.2.0"},"passwd":{"users":[{"name":"root","passwordHash":"PLEASE ASK MAINTAINER FOR A PROPER HASHEDPASS","sshAuthorizedKeys":["{{ pub_rsa_key }}"],"shell":"/bin/bash"}]},"systemd":{"units":[{"contents":"[Unit]\nDescription=Allow SSH Root Login\nAfter=network.target\n\n[Service]\nType=oneshot\nExecStart=/bin/bash -c \"sed -i 's/^#\\?PermitRootLogin .*/PermitRootLogin yes/' /etc/ssh/sshd_config\"\nExecStartPost=/bin/systemctl restart sshd\n\n[Install]\nWantedBy=multi-user.target\n","enabled":true,"name":"allowrootssh.service"},{"contents":"[Unit]\nDescription=Setup iperf3 on Debian\nAfter=network-online.target\nWants=network-online.target\n\n[Service]\nType=oneshot\nExecStart=/bin/bash -c 'echo \"deb http://deb.debian.org/debian/ bookworm main\" \u003e /etc/apt/sources.list.d/bookworm.list'\nExecStart=/usr/bin/apt-get update\nExecStart=/usr/bin/apt-get install -y iperf3 nftables\nExecStart=/bin/rm /etc/apt/sources.list.d/bookworm.list\nExecStart=/usr/bin/apt-get update\nRemainAfterExit=true\n\n[Install]\nWantedBy=multi-user.target\n","enabled":true,"name":"setup-iperf3.service"}]}}
diff --git a/test/benchmark_test/config_templates/test_configurations.json b/test/benchmark_test/config_templates/test_configurations.json
new file mode 100644
index 000000000..81cf4408b
--- /dev/null
+++ b/test/benchmark_test/config_templates/test_configurations.json
@@ -0,0 +1,77 @@
+{
+ "key_file": "~/.ssh/id_rsa",
+ "public_key_file": "~/.ssh/id_rsa.pub",
+ "default_dpservice_image": "ghcr.io/ironcore-dev/dpservice:sha-e9b4272",
+ "concurrent_flow_count": 3,
+ "expected_throughput": {
+ "sw": {
+ "local_vm2vm": 10,
+ "remote_vm2vm": 8,
+ "lb": 5
+ },
+ "hw": {
+ "local_vm2vm": 20,
+ "remote_vm2vm": 20,
+ "lb": 12
+ }
+ },
+ "hypervisors": [
+ {
+ "machine_name": "hypervisor-1",
+ "host_address": "192.168.23.166",
+ "user_name": "",
+ "port": 22,
+ "vms": [
+ {
+ "machine_name": "vm1",
+ "if_config":{
+ "ipv4": "192.168.129.5",
+ "ipv6": "2002::123",
+ "vni": 66,
+ "pci_addr": "0000:8a:00.0_representor_vf2"
+ }
+ },
+ {
+ "machine_name": "vm2",
+ "if_config":{
+ "ipv4": "192.168.129.6",
+ "ipv6": "2002::124",
+ "vni": 66,
+ "pci_addr": "0000:8a:00.0_representor_vf1"
+ }
+ }
+ ]
+ },
+ {
+ "machine_name": "hypervisor-2",
+ "role": "local",
+ "host_address": "192.168.23.86",
+ "user_name": "",
+ "port": 22,
+ "vms": [
+ {
+ "machine_name": "vm3",
+ "if_config":{
+ "ipv4": "172.32.4.9",
+ "ipv6": "2003::123",
+ "vni": 66,
+ "pci_addr": "0000:3b:00.0_representor_vf0"
+ },
+ "nat": {
+ "ip": "10.10.20.20",
+ "ports": [10240, 10360]
+ }
+ }
+ ]
+ }
+ ],
+ "lb": {
+ "name": "test_lb",
+ "ip": "10.20.30.30",
+ "ports": "TCP/5201,TCP/50007",
+ "vni": 66,
+ "lb_nodes": ["hypervisor-2"],
+ "lb_machines": ["vm3"]
+ }
+}
+
diff --git a/test/benchmark_test/config_templates/vm_tmpl.xml b/test/benchmark_test/config_templates/vm_tmpl.xml
new file mode 100755
index 000000000..96c7df74a
--- /dev/null
+++ b/test/benchmark_test/config_templates/vm_tmpl.xml
@@ -0,0 +1,78 @@
+
+ {{ VM_NAME }}
+
+
+
+
+
+ 8
+ 8
+ 4
+
+ hvm
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ destroy
+ restart
+ destroy
+
+
+
+
+
+ /usr/bin/qemu-system-x86_64
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ /dev/urandom
+
+
+
+
diff --git a/test/benchmark_test/conftest.py b/test/benchmark_test/conftest.py
index 6dcbb98f9..329194837 100644
--- a/test/benchmark_test/conftest.py
+++ b/test/benchmark_test/conftest.py
@@ -27,10 +27,7 @@ def pytest_addoption(parser):
"--reboot", action="store_true", help="Reboot VMs to obtain new configurations such as IPs"
)
parser.addoption(
- "--env-config-file", action="store", default="./test_configurations.json", help="Specify the file containing setup information"
- )
- parser.addoption(
- "--env-config-name", action="store", default="regular_setup", help="Specify the name of environment configuration that fits to hardware and VM setup. "
+ "--env-config-file", action="store", default="./provision_tmpls/output/test_configurations.json", help="Specify the file containing setup information"
)
parser.addoption(
"--dpservice-build-path", action="store", default=f"{script_dir}/../../build", help="Path to the root build directory"
@@ -45,18 +42,11 @@ def signal_handler(sig, frame):
def test_config(request):
config = request.config
test_config_file = config.getoption("--env-config-file")
- test_config_name = config.getoption("--env-config-name")
with open(test_config_file, 'r') as file:
config = json.load(file)
- env_config = next(
- (env for env in config['environments'] if env['name'] == test_config_name), None)
- if not env_config:
- raise RuntimeError(
- f"Failed to get config info for environment with name {test_config_name}")
-
- return env_config
+ return config
@pytest.fixture(scope="package", autouse=True)
diff --git a/test/benchmark_test/provision.py b/test/benchmark_test/provision.py
new file mode 100755
index 000000000..cd3dec72e
--- /dev/null
+++ b/test/benchmark_test/provision.py
@@ -0,0 +1,455 @@
+#!/usr/bin/env python3
+
+# SPDX-FileCopyrightText: 2023 SAP SE or an SAP affiliate company and IronCore contributors
+# SPDX-License-Identifier: Apache-2.0
+
+import argparse, json, shutil
+import re
+import os, time
+import random
+from jinja2 import Environment, FileSystemLoader
+
+from remote_machine_management import (
+ RemoteMachine,
+)
+
+tmpls_output_path = ''
+working_test_config_file = ''
+provision_tmpls_compressed = ''
+
+current_script_directory = os.getcwd()
+local_provision_tmpls_path = current_script_directory + '/provision_tmpls'
+target_provision_tmpls_output_path = '/tmp/provision_tmpls'
+
+hypervisor_machines = []
+
+def underscore_convert(text):
+ return re.sub("_", "-", text)
+
+
+def get_test_config(args, init):
+ if init:
+ os.system(f"cp ./config_templates/test_configurations.json {working_test_config_file}")
+
+ with open(working_test_config_file, 'r') as file:
+ config = json.load(file)
+
+ return config
+
+
+def get_working_test_config_file_path():
+ return os.path.join(tmpls_output_path, 'test_configurations.json')
+
+
+def prepare_ignition_file(env_config, template_path, ignition_template_name):
+ # Read public RSA key
+ pub_rsa_key_path = os.path.expanduser(env_config["public_key_file"])
+ with open(pub_rsa_key_path, 'r') as key_file:
+ pub_rsa_key = key_file.read().strip()
+
+ # Load the Jinja2 template for the ignition file
+ env = Environment(loader=FileSystemLoader(os.path.expanduser(template_path)))
+ template = env.get_template(ignition_template_name)
+
+ ignition_config = template.render(pub_rsa_key=pub_rsa_key)
+
+ # Save the ignition file
+ output_ignition_path = os.path.join(tmpls_output_path, "provision.ign")
+ with open(output_ignition_path, 'w') as output_file:
+ output_file.write(ignition_config)
+
+
+def create_path_on_local(path):
+ # If the directory does not exist, create it
+ if not os.path.exists(path):
+ os.makedirs(path)
+ print(f"Created directory: {path}")
+ else:
+ print(f"Directory already exists: {path}")
+
+
+def remove_path_on_local(path):
+ try:
+ if os.path.exists(path):
+ if os.path.isfile(path) or os.path.islink(path):
+ os.remove(path) # Remove the file or symbolic link
+ print(f"File or symlink removed: {path}")
+ elif os.path.isdir(path):
+ shutil.rmtree(path) # Remove the directory and its contents
+ print(f"Directory removed: {path}")
+ else:
+ print(f"Unknown file type: {path}")
+ else:
+ print(f"Path does not exist: {path}")
+ except Exception as e:
+ print(f"Error removing {path}: {e}")
+
+
+def create_local_provision_tmpls_path():
+ create_path_on_local(local_provision_tmpls_path)
+ os.system(f"cp ./config_templates/test_configurations.json {local_provision_tmpls_path}/test_configurations.json")
+ os.system(f"cp ./config_templates/provision_tmpl.ign {local_provision_tmpls_path}/provision_tmpl.ign")
+ os.system(f"cp ./config_templates/vm_tmpl.xml {local_provision_tmpls_path}/vm_tmpl.xml")
+
+
+def remove_local_provision_tmpls_path():
+ try:
+ remove_path_on_local(local_provision_tmpls_path)
+ except Exception as e:
+ print(f"Failed to remove local provision templates path: {e}")
+
+
+def create_tmpls_output_path():
+ # Append '/output' to the provided path
+ output_path = os.path.join(local_provision_tmpls_path, 'output')
+
+ # Expand user (~) and create the full path if it doesn't exist
+ expanded_output_path = os.path.expanduser(output_path)
+ create_path_on_local(expanded_output_path)
+
+ return expanded_output_path
+
+def generate_vm_iface_mac_addr():
+ mac_addr = [0x02, 0x00, 0x00]
+
+ for _ in range(3):
+ mac_addr.append(random.randint(0x00, 0xFF))
+
+ mac_addr_str = ":".join(map(lambda x: f"{x:02x}", mac_addr))
+
+ return mac_addr_str
+
+
+
+def update_vm_mac_addresses(config):
+ for hypervisor in config['hypervisors']:
+ for vm in hypervisor['vms']:
+ # Generate MAC address
+ mac_address = generate_vm_iface_mac_addr()
+ # Add the MAC address to the VM's if_config
+ vm['mac_address'] = mac_address
+ print(f"Generated MAC address for {vm['machine_name']}: {mac_address}")
+
+ # Write updated configuration back to the JSON file
+ with open(working_test_config_file, 'w') as file:
+ json.dump(config, file, indent=4)
+ print(f"Updated configuration written to {working_test_config_file}")
+
+
+def convert_pci_address(pci_addr):
+ # Extract the domain, bus, slot, function, and vf using regex
+ match = re.match(r"(?:(?P[0-9a-fA-F]{4}):)?(?P[0-9a-fA-F]{2}):(?P[0-9a-fA-F]{2})\.(?P[0-9a-fA-F])_representor_vf(?P\d+)", pci_addr)
+
+ if match:
+ # If domain is missing, default it to '0000'
+ domain = match.group('domain') if match.group('domain') else '0000'
+ bus = match.group('bus')
+ slot = match.group('slot')
+ vf = match.group('vf')
+
+ # Compute the function value as the VF value plus 2
+ function = hex(int(vf) + 2)[2:]
+
+ return f'domain="0x{domain}" bus="0x{bus}" slot="0x{slot}" function="0x{function}"'
+ else:
+ raise ValueError("Invalid PCI address format")
+
+
+def generate_vm_domain_xml(config):
+ ignition_file = os.path.join(target_provision_tmpls_output_path, "provision.ign")
+ env = Environment(loader=FileSystemLoader(os.path.expanduser('./config_templates')))
+ template = env.get_template('vm_tmpl.xml')
+
+ for hypervisor in config['hypervisors']:
+ hypervisor_name = hypervisor['machine_name']
+ xml_repo_per_hypervisor = tmpls_output_path + '/' + hypervisor_name
+ create_path_on_local(xml_repo_per_hypervisor)
+
+ for vm in hypervisor['vms']:
+ vm_name = vm['machine_name']
+ vm_xml_name = xml_repo_per_hypervisor + '/' + f'{vm_name}.xml'
+ pci_address = convert_pci_address(vm['if_config']['pci_addr'])
+ print(pci_address)
+ mac_address = vm['mac_address']
+
+ disk_image = os.path.expanduser(f'{target_provision_tmpls_output_path}/{hypervisor_name}/{vm_name}.raw')
+
+ vm_tmpl = template
+ vm_xml = vm_tmpl.render(VM_NAME=vm_name, VF_PCI_ADDRESS=pci_address, BRIDGE_IFACE_MAC=mac_address, DISK_IMAGE=disk_image, IGNITION_FILE=ignition_file)
+
+ # Save the xml file
+ with open(vm_xml_name, 'w') as output_file:
+ output_file.write(vm_xml)
+
+
+def add_remote_machine(machine):
+ hypervisor_machines.append(machine)
+
+def cleanup_remote_machine():
+ hypervisor_machines.clear()
+
+def get_remote_machine_by_name(name):
+ machine = next((machine for machine in hypervisor_machines if machine.machine_name == name), None)
+ if not machine:
+ raise ValueError(f"Failed to get machine for {name}")
+ return machine
+
+
+def compress_provision_tmpls():
+ global provision_tmpls_compressed
+ provision_tmpls_compressed = os.path.join(local_provision_tmpls_path, "provision_tmpls.tar.gz")
+ shutil.make_archive(provision_tmpls_compressed.replace('.tar.gz', ''), 'gztar', tmpls_output_path)
+ print(f"Compressed provision templates directory to {provision_tmpls_compressed}")
+
+
+def upload_and_decompress_provision_tmpls(machine):
+ try:
+ # Remove existing repo
+ task = [
+ {"command": "rm -r", "parameters": f"{target_provision_tmpls_output_path}"}
+ ]
+ machine.exec_task(task)
+
+ # Upload the compressed file to the remote machine
+ provision_tmpls_dst = '/tmp/provision_tmpls.tar.gz'
+ machine.upload("sftp", provision_tmpls_compressed, provision_tmpls_dst)
+ task = [
+ {"command": f'test -e {provision_tmpls_dst} && echo "exists" || echo "not exists"'}
+ ]
+ result = machine.exec_task(task)
+ if result != "exists":
+ machine.upload("scp", provision_tmpls_compressed, provision_tmpls_dst)
+ print(f"Uploaded {provision_tmpls_compressed} to remote machine at {provision_tmpls_compressed}")
+
+ # Decompress the file on the remote machine
+ task = [
+ {"command": "mkdir -p", "parameters": f"{target_provision_tmpls_output_path}"}
+ ]
+ machine.exec_task(task)
+
+ task = [
+ {"command": "tar -xzf", "parameters": f"{provision_tmpls_dst} -C {target_provision_tmpls_output_path}"}
+ ]
+ machine.exec_task(task)
+ print(f"Decompressed {provision_tmpls_dst} to {target_provision_tmpls_output_path}")
+
+ # Clean up the compressed file on the remote machine
+ task = [
+ {"command": "rm", "parameters": f"{provision_tmpls_dst}"}
+ ]
+ machine.exec_task(task)
+ print(f"Removed compressed file {provision_tmpls_compressed} from remote machine")
+ except Exception as e:
+ print(f"Failed to transfer provision templates: {e}")
+ raise e
+
+
+def upload_vm_disk_image(machine, hypervisor_info, src_disk_image_path):
+ try:
+ for vm in hypervisor_info['vms']:
+ vm_name = vm['machine_name']
+ print(f"Uploading vm disk image for remote machine {vm_name}")
+ dst_disk_image_path = os.path.expanduser(f"{target_provision_tmpls_output_path}/{hypervisor_info['machine_name']}/{vm_name}.raw")
+ machine.upload("sftp", src_disk_image_path, dst_disk_image_path)
+ task = [
+ {"command": f'test -e {dst_disk_image_path} && echo "exists" || echo "not exists"'}
+ ]
+ result = machine.exec_task(task)
+ if result != "exists":
+ machine.upload("scp", src_disk_image_path, dst_disk_image_path)
+ print(f"Uploaded {src_disk_image_path} to remote machine at {dst_disk_image_path}")
+ except Exception as e:
+ print("Cannot transfer disk image")
+
+
+def start_vm_on_hypervisor(machine, hypervisor_info):
+ try:
+ for vm in hypervisor_info['vms']:
+ vm_name = vm['machine_name']
+ vm_xml_path = os.path.expanduser(f'{target_provision_tmpls_output_path}/{hypervisor_info["machine_name"]}/{vm_name}.xml')
+ print(f"Starting VM {vm_name} on hypervisor {hypervisor_info['machine_name']}")
+ task = [
+ {"command": "virsh define", "parameters": vm_xml_path, "sudo":True},
+ ]
+ machine.exec_task(task)
+ task = [
+ {"command": "virsh start", "parameters": vm_name, "sudo":True, "delay": 3}
+ ]
+ machine.exec_task(task)
+ print(f"VM {vm_name} started successfully on {hypervisor_info['machine_name']}")
+ except Exception as e:
+ print(f"Failed to start VM {vm_name}: {e}")
+ raise e
+
+def stop_and_remove_vm_on_hypervisor(machine, hypervisor_info):
+ try:
+ for vm in hypervisor_info['vms']:
+ vm_name = vm['machine_name']
+ print(f"Stopping and removing VM {vm_name} on hypervisor {hypervisor_info['machine_name']}")
+ task = [
+ {"command": "virsh destroy", "parameters": vm_name, "sudo": True},
+ ]
+ machine.exec_task(task)
+ task = [
+ {"command": "virsh undefine", "parameters": vm_name, "sudo": True, "delay": 10}
+ ]
+ machine.exec_task(task)
+ print(f"VM {vm_name} stopped and removed successfully on {hypervisor_info['machine_name']}")
+ except Exception as e:
+ print(f"Failed to stop and remove VM {vm_name}: {e}")
+ raise e
+
+
+def provision_vms_on_hypervisor(config, args):
+ try:
+ for hypervisor_info in config["hypervisors"]:
+ hypervisor_machine = get_remote_machine_by_name(hypervisor_info["machine_name"])
+ upload_and_decompress_provision_tmpls(hypervisor_machine)
+ upload_vm_disk_image(hypervisor_machine, hypervisor_info, args['disk_template'])
+ start_vm_on_hypervisor(hypervisor_machine, hypervisor_info)
+ except Exception as e:
+ print("Cannot transfer provision tmpls")
+
+
+def clean_up_on_hypervisors(config, args):
+ try:
+ for hypervisor_info in config['hypervisors']:
+ hypervisor_machine = get_remote_machine_by_name(hypervisor_info['machine_name'])
+ # Stop and remove all VMs on the hypervisor
+ stop_and_remove_vm_on_hypervisor(hypervisor_machine, hypervisor_info)
+
+ # Remove the /mnt/provision_tmpls directory on the hypervisor
+ task = [
+ {"command": "rm -rf", "parameters": f"{target_provision_tmpls_output_path}"}
+ ]
+ hypervisor_machine.exec_task(task)
+ print(f"Removed {target_provision_tmpls_output_path} from {hypervisor_info['machine_name']}")
+
+ except Exception as e:
+ print(f"Failed to clean up on hypervisors: {e}")
+
+
+def update_vm_ip_addresses(config, args):
+ try:
+ for hypervisor_info in config['hypervisors']:
+ print(f"Retrieving IP addresses from hypervisor {hypervisor_info['machine_name']}")
+
+ hypervisor_machine = get_remote_machine_by_name(hypervisor_info['machine_name'])
+ # Attempt to retrieve IP address up to 10 times, with a 3-second interval
+ for attempt in range(10):
+ print(f"Attempt {attempt + 1}/10 to retrieve IP addresses...")
+ task = [
+ {"command": "virsh net-dhcp-leases", "parameters": 'default', "sudo": True}
+ ]
+ result = hypervisor_machine.exec_task(task)
+
+ for vm in hypervisor_info['vms']:
+ mac_address = vm['mac_address']
+ vm_ip = None
+
+ # Parse each line in the virsh net-dhcp-leases result
+ for line in result.splitlines():
+ if mac_address in line:
+ match = re.search(r"(\d{1,3}\.){3}\d{1,3}", line)
+ if match:
+ vm_ip = match.group(0)
+ break
+
+ if vm_ip:
+ print(f"Found IP address {vm_ip} for VM {vm['machine_name']} with MAC {mac_address}")
+ vm['host_address'] = vm_ip
+ else:
+ print(f"No IP address found yet for VM {vm['machine_name']} with MAC {mac_address}")
+
+ # Break out of the loop if all VMs have been assigned an IP address
+ all_vms_have_ips = all('host_address' in vm for vm in hypervisor_info['vms'])
+ if all_vms_have_ips:
+ break
+
+ # Wait for 3 seconds before trying again
+ time.sleep(3)
+
+
+ # Write the updated configuration with IP addresses back to the JSON file
+ with open(working_test_config_file, 'w') as file:
+ json.dump(config, file, indent=4)
+ print(f"Updated configuration written to {working_test_config_file}")
+
+ except Exception as e:
+ print(f"Failed to update VM IP addresses: {e}")
+ raise e
+
+
+def connect_with_hypervisors(config):
+ try:
+ for hypervisor_info in config["hypervisors"]:
+ hypervisor_machine = RemoteMachine(hypervisor_info, os.path.expanduser(config["key_file"]))
+ add_remote_machine(hypervisor_machine)
+ hypervisor_machine.start()
+ except Exception as e:
+ print("Cannot connect with hypervisor")
+ remove_local_provision_tmpls_path()
+ exit(1)
+
+def disconnect_from_hypervisors():
+ for machine in hypervisor_machines:
+ machine.disconnect()
+ print(f"Disconnected from {machine.machine_name}")
+ cleanup_remote_machine()
+
+def provision_benchmark_environment(args):
+ global tmpls_output_path
+ global working_test_config_file
+
+ args = vars(args)
+
+ create_local_provision_tmpls_path()
+ tmpls_output_path = create_tmpls_output_path()
+ working_test_config_file = get_working_test_config_file_path()
+ config = get_test_config(args, True)
+ connect_with_hypervisors(config)
+
+ if args["clean_up"]:
+ clean_up_on_hypervisors(config, args)
+ disconnect_from_hypervisors()
+ remove_local_provision_tmpls_path()
+ return
+ if args['disk_template'] == '':
+ print("Disk template must be provided")
+ exit(1)
+
+ prepare_ignition_file(config, local_provision_tmpls_path, args["ignition_template_name"])
+ update_vm_mac_addresses(config)
+ config = get_test_config(args, False)
+ generate_vm_domain_xml(config)
+
+ compress_provision_tmpls()
+ provision_vms_on_hypervisor(config, args)
+
+ update_vm_ip_addresses(config, args)
+
+ disconnect_from_hypervisors()
+
+def add_arg_parser():
+ script_path = os.path.dirname(os.path.abspath(__file__))
+ parser = argparse.ArgumentParser(
+ formatter_class=lambda prog: argparse.HelpFormatter(prog, max_help_position=32))
+
+ parser.add_argument("--disk-template", action="store",
+ default='', help="Disk image template location and name")
+ parser.add_argument("--ignition-template-name", action="store", default="provision_tmpl.ign",
+ help="Ignition file template's name")
+ parser.add_argument("--xml-template-name", action="store", default="vm.xml",
+ help="VM xml definition template's name")
+ parser.add_argument("--clean-up", action="store_true", default=False, help="Flag to clean up VM and provision files on hypervisors via tunnels.")
+ parser.set_defaults(func=provision_benchmark_environment)
+
+ args = parser.parse_args()
+ if hasattr(args, 'func'):
+ args.func(args)
+ else:
+ parser.print_help()
+
+
+if __name__ == '__main__':
+ add_arg_parser()
diff --git a/test/benchmark_test/remote_machine_management.py b/test/benchmark_test/remote_machine_management.py
index 12d639248..88390163d 100644
--- a/test/benchmark_test/remote_machine_management.py
+++ b/test/benchmark_test/remote_machine_management.py
@@ -105,7 +105,6 @@ def get_all_pids(self):
def disconnect(self):
"""Close the SSH connection."""
self.logger.info("Disconnecting...")
- self.stop_event.set()
self.client.close()
self.logger.info("Disconnected.")
@@ -278,8 +277,9 @@ class RemoteMachine:
def __init__(self, config, key_file, parent_machine=None):
self.machine_name = config["machine_name"]
self.parent_machine = parent_machine
- self.ssh_manager = SSHManager(self.machine_name, config["host_address"], config["port"], config["user_name"],
- key_file=key_file, proxy=None if not parent_machine else self.parent_machine.get_connection())
+ port, user_name, proxy = (config["port"], config["user_name"], None) if not parent_machine else (22, "root", self.parent_machine.get_connection())
+ self.ssh_manager = SSHManager(self.machine_name, config["host_address"], port, user_name,
+ key_file=key_file, proxy=proxy)
self.logger = self.ssh_manager.logger
def start(self):
@@ -303,6 +303,9 @@ def stop(self):
self.ssh_manager.terminate_all_containers()
self.ssh_manager.disconnect()
+ def disconnect(self):
+ self.ssh_manager.disconnect()
+
def get_machine_name(self):
return self.machine_name
diff --git a/test/benchmark_test/remote_machine_operations.py b/test/benchmark_test/remote_machine_operations.py
index e49508fd0..2f63dda31 100644
--- a/test/benchmark_test/remote_machine_operations.py
+++ b/test/benchmark_test/remote_machine_operations.py
@@ -431,6 +431,21 @@ def remote_machine_op_vm_config_rm_default_route(machine_name, default_gw="192.1
if machine:
machine.logger.error(f"Failed to remove the default route: {e}")
+def remote_machine_op_vm_config_nft_default_accept(machine_name):
+ try:
+ machine = get_remote_machine(machine_name)
+ if not machine.parent_machine:
+ raise NotImplementedError(
+ f"Cannot rm default route on a non-vm machine {machine_name}")
+
+ vm_task = [
+ {"command": f"nft add chain inet filter input '{{ policy accept; }}'", "sudo": True}
+ ]
+ machine.exec_task(vm_task)
+ except Exception as e:
+ if machine:
+ machine.logger.error(f"Failed to remove the default route: {e}")
+
def remote_machine_op_vm_config_tmp_dir(machine_name):
machine = None
diff --git a/test/benchmark_test/run_benchmarktest.py b/test/benchmark_test/run_benchmarktest.py
index 97dcd9b31..38348cfee 100755
--- a/test/benchmark_test/run_benchmarktest.py
+++ b/test/benchmark_test/run_benchmarktest.py
@@ -51,10 +51,8 @@ def add_arg_parser():
default=f"{script_path}/../../build", help="Path to dpservice-bin build directory")
parser.add_argument("--reboot", action="store_true", default=False,
help="Reboot VMs to obtain new configurations such as IPs")
- parser.add_argument("--env-config-file", action="store", default="./test_configurations.json",
+ parser.add_argument("--env-config-file", action="store", default="./provision_tmpls/output/test_configurations.json",
help="Specify the file containing setup information")
- parser.add_argument("--env-config-name", action="store", default="regular_setup",
- help="Specify the name of environment configuration that fits to hardware and VM setup. ")
parser.add_argument("-v", "--verbose", action="store_true", default=False,
help="Allow to output debug information during pytest execution")
parser.set_defaults(func=execute_benchmark_pytest)
diff --git a/test/benchmark_test/test_configurations.json b/test/benchmark_test/test_configurations.json
deleted file mode 100644
index e8b581880..000000000
--- a/test/benchmark_test/test_configurations.json
+++ /dev/null
@@ -1,90 +0,0 @@
-{
- "environments": [
- {
- "name": "regular_setup",
- "key_file": "~/.ssh/id_rsa",
- "default_dpservice_image": "ghcr.io/ironcore-dev/dpservice:sha-e9b4272",
- "concurrent_flow_count": 3,
- "expected_throughput": {
- "sw": {
- "local_vm2vm": 10,
- "remote_vm2vm": 8,
- "lb": 5
- },
- "hw": {
- "local_vm2vm": 20,
- "remote_vm2vm": 20,
- "lb": 12
- }
- },
- "hypervisors": [
- {
- "machine_name": "hypervisor-1",
- "host_address": "192.168.23.166",
- "user_name": "tli",
- "port": 22,
- "vms": [
- {
- "machine_name": "vm1",
- "host_address": "192.168.122.47",
- "user_name": "root",
- "port": 22,
- "if_config":{
- "ipv4": "192.168.129.5",
- "ipv6": "2002::123",
- "vni": 66,
- "pci_addr": "8a:00.0_representor_vf2"
- }
- },
- {
- "machine_name": "vm2",
- "host_address": "192.168.122.49",
- "user_name": "root",
- "port": 22,
- "if_config":{
- "ipv4": "192.168.129.6",
- "ipv6": "2002::124",
- "vni": 66,
- "pci_addr": "8a:00.0_representor_vf1"
- }
- }
- ]
- },
- {
- "machine_name": "hypervisor-2",
- "role": "local",
- "host_address": "192.168.23.86",
- "user_name": "tli",
- "port": 22,
- "vms": [
- {
- "machine_name": "vm3",
- "host_address": "192.168.122.47",
- "user_name": "root",
- "port": 22,
- "if_config":{
- "ipv4": "172.32.4.9",
- "ipv6": "2003::123",
- "vni": 66,
- "pci_addr": "0000:3b:00.0_representor_vf0"
- },
- "nat": {
- "ip": "10.10.20.20",
- "ports": [10240, 10360]
- }
- }
- ]
- }
- ],
- "lb": {
- "name": "test_lb",
- "ip": "10.20.30.30",
- "ports": "TCP/5201,TCP/50007",
- "vni": 66,
- "lb_nodes": ["hypervisor-2"],
- "lb_machines": ["vm3"]
- }
- }
- ]
-}
-