Node Capacity/Allocatable of VF num NOT correct

[qiaoning]

What happened?

Node Capacity/Allocatable of VF num NOT correct

Capacity:
  cpu:                        128
  ephemeral-storage:          256981444Ki
  hugepages-1Gi:              0
  hugepages-2Mi:              0
  memory:                     1056587000Ki
  openshift.io/ibcx6vfnuma0:  7
  openshift.io/ibcx6vfnuma1:  7
  pods:                       110
Allocatable:
  cpu:                        127600m
  ephemeral-storage:          256981444Ki
  hugepages-1Gi:              0
  hugepages-2Mi:              0
  memory:                     1027323544794
  openshift.io/ibcx6vfnuma0:  7
  openshift.io/ibcx6vfnuma1:  7
  pods:                       110

$ kubectl get SriovNetworkNodePolicy ibcx6vfnuma0 -n sriov-network-operator -oyaml
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  annotations:
  generation: 1
  name: ibcx6vfnuma0
  namespace: sriov-network-operator
spec:
  deviceType: netdevice
  isRdma: true
  linkType: IB
  nicSelector:
    deviceID: 101b
    rootDevices:
    - "0000:12:00.0"
    vendor: 15b3
  nodeSelector:
    feature.node.kubernetes.io/ib-network-sriov.capable: "true"
  numVfs: 8
  priority: 10
  resourceName: ibcx6vfnuma0

What did you expect to happen?

Expected numVfs is : 8

What are the minimal steps needed to reproduce the bug?

Step 1: Deploy sriov-network-operator 1.2.1
Step 2: Deploy a payload:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: myubt
  name: myubt
  namespace: test1
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: myubt
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        k8s.v1.cni.cncf.io/networks: sriov-network-operator/ibcx6vfnuma0,sriov-network-operator/ibcx6vfnuma1
      creationTimestamp: null
      labels:
        app: myubt
    spec:
      containers:
      - args: [ "while true; do sleep 30; done;" ]
        command: [ "/bin/bash", "-c", "--" ]
        image: ubuntu:20.04
        imagePullPolicy: IfNotPresent
        name: myubt
        resources:
          limits:
            cpu: "1"
            memory: 4Gi
            openshift.io/ibcx6vfnuma0: "1"
            openshift.io/ibcx6vfnuma1: "1"
          requests:
            cpu: "1"
            memory: 4Gi
            openshift.io/ibcx6vfnuma0: "1"
            openshift.io/ibcx6vfnuma1: "1"
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      hostPID: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30

Step 3: Delete the sriov-device-plugin-7wkm8 pod of node on which step 2 payload is running on.

kubectl delete po sriov-device-plugin-7wkm8 -n sriov-network-operator

Then, the sriov-device-plugin daemonset will auto launch a new pod you just deleted.

Step 4: Now the node Capacity/Allocatable of VF num NOT correct, show me 7, NOT 8. Why?

Anything else we need to know?

Yeah!
I add some debug code to sriov-network-device-plugin,
It's filterd by linkTypes list, code location: pkg/netdevice/netDeviceProvider.go#GetFilteredDevices()
and, pkg/resources/deviceSelectors.go#func (s *linkTypeSelector) Filter, linkType := dev.(types.PciNetDevice).GetLinkType(), linkType is empty.
And, the final root cause is pkg/netdevice/pciNetDevice.go#func NewPciNetDevice(...)

	pciAddr := pciDev.GetPciAddr()
	glog.Infof("...pciAddr: %s", pciAddr)
	netDevs, _ := utils.GetNetNames(pciAddr)
	if len(netDevs) == 0 {
		ifName = ""
	} else {
		ifName = netDevs[0]
	}

The len(netDevs) == 0, because the /sys/bus/pci/devices/0000:12:00.5/net directory is empty.

# pwd
/sys/bus/pci/devices/0000:12:00.5/net
# ls -al
total 0
drwxr-xr-x  3 root root 0 Dec 10 22:06 .
drwxr-xr-x 12 root root 0 Dec 10 22:06 ..

and other vf director is seems OK, like below:

# pwd
/sys/bus/pci/devices/0000:12:00.4/net
# ls 
ibs2v3
# ls ibs2v3/
addr_assign_type  broadcast        carrier_down_count  delete_child  device   flags              ifindex    mode              netdev_group  phys_port_name  power       set_mac     subsystem     uevent
addr_len          carrier          carrier_up_count    dev_id        dormant  gro_flush_timeout  iflink     mtu               operstate     phys_switch_id  proto_down  speed       tx_queue_len  umcast
address           carrier_changes  create_child        dev_port      duplex   ifalias            link_mode  name_assign_type  phys_port_id  pkey            queues      statistics  type

And if I removed the spec.template.metadata.annotations

k8s.v1.cni.cncf.io/networks: sriov-network-operator/ibcx6vfnuma0,sriov-network-operator/ibcx6vfnuma1

And repeat the step 1,2,3,4..., Everything is OK, the vf number is 8, NOT 7.

So, My Question is, what's the difference between add k8s.v1.cni.cncf.io/networks to annotations and NOT.
So, My Question is, what's the difference between add k8s.v1.cni.cncf.io/networks to annotations and NOT.
So, My Question is, what's the difference between add k8s.v1.cni.cncf.io/networks to annotations and NOT.

Component Versions

Please fill in the below table with the version numbers of components used.

Component	Version
SR-IOV Network Device Plugin	3.5.1
SR-IOV CNI Plugin	sriovCni:v2.6.3; ibSriovCni: v1.0.2
Multus	Not sure
Kubernetes	v1.23.10
OS	Ubuntu 20.04 Kernel 5.4.0-42-generic
sriov-operator	v1.2.0

Config Files

Config file locations may be config dependent.

Device pool config file location (Try '/etc/pcidp/config.json')

{
  "resourceList": [
    {
      "resourceName": "ibcx6vfnuma0",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "devices": [
          "101c"
        ],
        "rootDevices": [
          "0000:12:00.0"
        ],
        "linkTypes": [
          "infiniband"
        ],
        "IsRdma": true,
        "NeedVhostNet": false
      },
      "SelectorObj": null
    },
    {
      "resourceName": "ibcx6vfnuma1",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "devices": [
          "101c"
        ],
        "rootDevices": [
          "0000:33:00.0"
        ],
        "linkTypes": [
          "infiniband"
        ],
        "IsRdma": true,
        "NeedVhostNet": false
      },
      "SelectorObj": null
    }
  ]
}

Multus config (Try '/etc/cni/multus/net.d')

CNI config (Try '/etc/cni/net.d/')

Kubernetes deployment type ( Bare Metal, Kubeadm etc.)

Kubeconfig file

SR-IOV Network Custom Resource Definition

Logs

SR-IOV Network Device Plugin Logs (use kubectl logs $PODNAME)

Multus logs (If enabled. Try '/var/log/multus.log' )

Kubelet logs (journalctl -u kubelet)

[qiaoning]

aa43530

Seems bug fixed on this commit.

#479

[SchSeba]

yep that was what I was going to comment that we fix that image around the link type

[SchSeba]

can I close this issue?

[qiaoning]

can I close this issue?

Yeah, thanks a lot.