design rational #114

hujun-open · 2023-12-19T23:15:03Z

hujun-open
Dec 19, 2023

Hi,
I wonder what is the reason behind architecture choice of using a device plugin + a CNI plugin? where device plugin create the macvtap interface, and then CNI plugin move the pre-created interface into the pod namespace
why not just let CNI plugin create the macvtap interface directly?

Answered by hujun-open

Dec 20, 2023

after some research, my guess is libvirtd expect to see the tap device (/dev/tapxx) for pre-created macvtap interface; however if the CNI plugin create the macvtap interface, then tap device won't show up in k8s pod, it is only on host; while a k8s device plugin will allow k8s to mount host's /dev/tapxxx to the pod;

it would be nice for dev to confirm if my theory is correct

View full answer

hujun-open · 2023-12-20T01:25:32Z

hujun-open
Dec 20, 2023
Author

after some research, my guess is libvirtd expect to see the tap device (/dev/tapxx) for pre-created macvtap interface; however if the CNI plugin create the macvtap interface, then tap device won't show up in k8s pod, it is only on host; while a k8s device plugin will allow k8s to mount host's /dev/tapxxx to the pod;

it would be nice for dev to confirm if my theory is correct

7 replies

maiqueb Dec 21, 2023
Maintainer

Unless I'm seriously wrong (it's been years since we've chosen this route) that doesn't work - libvirt requires the /dev/tapXYZ device to work properly.

But I don't remember the exact error it throws tbh. I don't even remember if traffic was broken, or if libvirt fails to create the VM.

hujun-open Dec 21, 2023
Author

I did a test like following:

I wrote a simple CNI plugin that creates a tap interface with configurable interface name (e.g. tap999)
I also wrote kubevirt hook sidecar that replace the interface target dev in original libvirt XML to the name of the tap interface created by CNI above (e.g. tap999), e.g. attach the tap interface pre-created by the CNI plugin
create a VM with single multus interface that use the CNI in #1, VM is running without issue even though there is no tap device within the pod; although I also noticed there is no /dev/tapxx on the host as well; it seems linux does not create /dev/tapxx for tap interface
following are from within compute container of the virt-launch pod

bash-5.1# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0@if132: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
    link/ether ba:01:ef:ab:7f:66 brd ff:ff:ff:ff:ff:ff link-netnsid 0
3: 6c270ef2f25-nic: <BROADCAST,NOARP> mtu 1500 qdisc noop master k6t-6c270ef2f25 state DOWN mode DEFAULT group default
    link/ether 02:db:09:76:72:57 brd ff:ff:ff:ff:ff:ff
4: tap999: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 7a:c5:2f:ed:2f:f7 brd ff:ff:ff:ff:ff:ff
5: k6t-6c270ef2f25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:db:09:76:72:57 brd ff:ff:ff:ff:ff:ff
6: tap6c270ef2f25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master k6t-6c270ef2f25 state DOWN mode DEFAULT group default qlen 1000
    link/ether 96:ec:14:f2:60:a0 brd ff:ff:ff:ff:ff:ff
7: pod6c270ef2f25: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether da:a6:d7:1d:7f:9d brd ff:ff:ff:ff:ff:ff
bash-5.1# ip -d link show tap999
4: tap999: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 7a:c5:2f:ed:2f:f7 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65521
    tun type tap pi off vnet_hdr on persist on user root group root addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

bash-5.1# ls /dev/tap*
ls: cannot access '/dev/tap*': No such file or directory

bash-5.1# virsh list
Authorization not available. Check if polkit service is running or see debug message for more information.
 Id   Name             State
--------------------------------
 1    default_testvm   running

bash-5.1# virsh dumpxml default_testvm
Authorization not available. Check if polkit service is running or see debug message for more information.
<domain type='kvm' id='1'>
  <name>default_testvm</name>
  <uuid>5a9fc181-957e-5c32-9e5a-2de5e9673531</uuid>
  <metadata>
    <kubevirt xmlns="http://kubevirt.io">
      <uid/>
    </kubevirt>
  </metadata>
  <memory unit='KiB'>63488</memory>
  <currentMemory unit='KiB'>62500</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <iothreads>1</iothreads>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>KubeVirt</entry>
      <entry name='product'>None</entry>
      <entry name='uuid'>5a9fc181-957e-5c32-9e5a-2de5e9673531</entry>
      <entry name='family'>KubeVirt</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Cooperlake</model>
    <vendor>Intel</vendor>
    <topology sockets='1' dies='1' cores='1' threads='1'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='avx512ifma'/>
    <feature policy='require' name='sha-ni'/>
    <feature policy='require' name='avx512vbmi'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='avx512vbmi2'/>
    <feature policy='require' name='gfni'/>
    <feature policy='require' name='vaes'/>
    <feature policy='require' name='vpclmulqdq'/>
    <feature policy='require' name='avx512bitalg'/>
    <feature policy='require' name='avx512-vpopcntdq'/>
    <feature policy='require' name='rdpid'/>
    <feature policy='require' name='fsrm'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='ibpb'/>
    <feature policy='require' name='ibrs'/>
    <feature policy='require' name='amd-stibp'/>
    <feature policy='require' name='amd-ssbd'/>
    <feature policy='require' name='rsba'/>
    <feature policy='disable' name='hle'/>
    <feature policy='disable' name='rtm'/>
    <feature policy='disable' name='pku'/>
    <feature policy='disable' name='avx512-bf16'/>
    <feature policy='disable' name='taa-no'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk' model='virtio-non-transitional'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' discard='unmap'/>
      <source file='/var/run/kubevirt-ephemeral-disks/disk-data/containerdisk/disk.qcow2' index='2'/>
      <backingStore type='file' index='3'>
        <format type='qcow2'/>
        <source file='/var/run/kubevirt/container-disks/disk_0.img'/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='ua-containerdisk'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk' model='virtio-non-transitional'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' discard='unmap'/>
      <source file='/var/run/kubevirt-ephemeral-disks/cloud-init-data/default/testvm/noCloud.iso' index='1'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='ua-cloudinitdisk'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-non-transitional'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0' model='virtio-non-transitional'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <alias name='pci.10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <interface type='ethernet'>
      <mac address='76:27:df:d4:19:2e'/>
      <target dev='tap999' managed='no'/>
      <model type='virtio-non-transitional'/>
      <mtu size='1500'/>
      <alias name='ua-net1'/>
      <rom enabled='no'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='unix'>
      <source mode='bind' path='/var/run/kubevirt-private/3b5c44ad-6ebf-4af5-933b-d8346e6299be/virt-serial0'/>
      <log file='/var/run/kubevirt-private/3b5c44ad-6ebf-4af5-933b-d8346e6299be/virt-serial0-log' append='on'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='unix'>
      <source mode='bind' path='/var/run/kubevirt-private/3b5c44ad-6ebf-4af5-933b-d8346e6299be/virt-serial0'/>
      <log file='/var/run/kubevirt-private/3b5c44ad-6ebf-4af5-933b-d8346e6299be/virt-serial0-log' append='on'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-default_testvm/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' socket='/var/run/kubevirt-private/3b5c44ad-6ebf-4af5-933b-d8346e6299be/virt-vnc'>
      <listen type='socket' socket='/var/run/kubevirt-private/3b5c44ad-6ebf-4af5-933b-d8346e6299be/virt-vnc'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'>
      <alias name='watchdog0'/>
    </watchdog>
    <memballoon model='virtio-non-transitional' freePageReporting='on'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

following is from the host, no /dev/tapxxx as well

root@k8svm:~/kvirt/hook# ls /dev/tap*
ls: cannot access '/dev/tap*': No such file or directory

maiqueb Dec 21, 2023
Maintainer

Well, what does your simple CNI plugin do ? To which is the tap device attached ?

Like, IIUC, we're creating a macvtap interface on top of a host interface. I don't know how your approach is networked.

hujun-open Dec 21, 2023
Author

my CNI only creates a detached tap interface with a user specified name, it also creates a dummy interface with runtime assigned interface name so that kubevirt won't complain.
this pre-created tap interface doesn't attach to anything, just want to test if libvirt could use a pre-created tap interface inside a container when there is no /dev/tapxxx; since libvirt does require /dev/tapxx for pre-created macvtap interface.

the source code of the CNI plugin below:

/*
This is a CNI plugin to create a tap interface with name specified in TapIfName,
also with a dummy interface with runtime assigned name.
*/
package main

import (
	"encoding/json"
	"fmt"
	"log"

	"github.com/containernetworking/cni/pkg/skel"
	"github.com/containernetworking/cni/pkg/types"
	current "github.com/containernetworking/cni/pkg/types/100"
	"github.com/containernetworking/cni/pkg/version"
	"github.com/containernetworking/plugins/pkg/ns"
	bv "github.com/containernetworking/plugins/pkg/utils/buildversion"
	"github.com/vishvananda/netlink"
)

type Net struct {
	Name          string `json:"name"`
	CNIVersion    string `json:"cniVersion"`
	RuntimeConfig struct {
		IPs []string `json:"ips,omitempty"`
	} `json:"runtimeConfig,omitempty"`
	TapIfName string `json:"tapname"`
}

func parseConfig(stdin []byte) (*Net, error) {
	conf := Net{}

	if err := json.Unmarshal(stdin, &conf); err != nil {
		return nil, fmt.Errorf("failed to parse network configuration: %v", err)
	}

	return &conf, nil
}

func cmdAdd(args *skel.CmdArgs) error {
	conf, err := parseConfig(args.StdinData)
	if err != nil {
		log.Println(err)
		return err
	}
	netns, err := ns.GetNS(args.Netns)
	if err != nil {
		log.Println(err)
		return fmt.Errorf("failed to open netns %q: %v", netns, err)
	}
	//create dummy interface with runtime assigned ifname, this is to satisfy kubevirt
	dm := &netlink.Dummy{
		LinkAttrs: netlink.LinkAttrs{
			Name:      args.IfName,
			Namespace: netlink.NsFd(int(netns.Fd())),
		},
	}
	if err := netlink.LinkAdd(dm); err != nil {
		return fmt.Errorf("failed to create dummy, %v: %v", args.IfName, err)
	}

	err = netns.Do(func(_ ns.NetNS) error {
		return AddTAPIf(conf.TapIfName)
	})
	if err != nil {
		log.Println(err)
		return err
	}

	result := &current.Result{CNIVersion: current.ImplementedSpecVersion}
	result.Interfaces = []*current.Interface{
		{
			Name:    args.IfName,
			Sandbox: args.Netns,
		},
	}
	result.IPs = []*current.IPConfig{}
	return types.PrintResult(result, conf.CNIVersion)
}

func AddTAPIf(name string) error {

	newif := &netlink.Tuntap{
		LinkAttrs: netlink.LinkAttrs{
			Name:   name,
			MTU:    1500,
			TxQLen: -1,
		},
		NonPersist: false,
		Mode:       netlink.TUNTAP_MODE_TAP,
	}
	err := netlink.LinkAdd(newif)
	if err != nil {
		return fmt.Errorf("failed to create %v, %w", name, err)
	}
	err = netlink.LinkSetUp(newif)
	if err != nil {
		return fmt.Errorf("failed to bring up %v, %w", name, err)
	}

	return nil
}

func cmdDel(args *skel.CmdArgs) error {

	return nil
}

func cmdCheck(args *skel.CmdArgs) error {
	return nil
}
func main() {
	skel.PluginMain(cmdAdd, cmdCheck, cmdDel, version.All, bv.BuildString("mytap"))
}

maiqueb Dec 22, 2023
Maintainer

Sure.

Your CNI does not relate to macvtap, which removes the need for the bridge in the pod.

Anyways, we're discussing on a closed answer, and I am unsure what's the direction / overall motive.

Feel free to open other issues / questions.

Again, than you for your interest / curiosity on the project.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

design rational #114

{{title}}

Replies: 1 comment 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

design rational #114

hujun-open Dec 19, 2023

Replies: 1 comment · 7 replies

hujun-open Dec 20, 2023 Author

maiqueb Dec 21, 2023 Maintainer

hujun-open Dec 21, 2023 Author

maiqueb Dec 21, 2023 Maintainer

hujun-open Dec 21, 2023 Author

maiqueb Dec 22, 2023 Maintainer

hujun-open
Dec 19, 2023

Replies: 1 comment 7 replies

hujun-open
Dec 20, 2023
Author

maiqueb Dec 21, 2023
Maintainer

hujun-open Dec 21, 2023
Author

maiqueb Dec 21, 2023
Maintainer

hujun-open Dec 21, 2023
Author

maiqueb Dec 22, 2023
Maintainer