vdpa/vfe: Add new vdpa driver for virtual full emulation device #1

kailiangz1 · 2022-04-21T02:13:22Z

Virtio full emulation will have 2 driver in DPDK, one is oring eth
device driver, the other is vdpa driver. To distinguish this 2 dirver
add vdpa=1 in command,like "-a 0000:3d:00.2,vdpa=1"

This vdpa_vfe dirver only supoort 1 queue pair now.

Doorbell will use software relay if vfe notify area is not 4K aligned

Interrupt will map to callfd through vfio ioctl
VFE's Msix number should bigger than queue number. Interrupt map will
fail if Msix number is less.

Signed-off-by: Kailiang Zhou kailiangz@nvidia.com

GavinLi-NV

Mixed tab and space?

paravmellanox · 2022-04-21T12:20:56Z

Virtio full emulation will have 2 driver in DPDK, one is oring eth device driver, the other is vdpa driver. To distinguish this 2 dirver add vdpa=1 in command,like "-a 0000:3d:00.2,vdpa=1"

Dont say virtio full emulation device. It is internal term. Just say virtio device.
Also this extra knob of vdpa is needed when both the drivers are compiled.
Can you disable the compilation of ether device driver over virtio device?
if not, can you submit the patch upstream to do so? So that we have single driver that can work as default?

This vdpa_vfe dirver only supoort 1 queue pair now.

Please correct all spelling mistakes.
Install "codespell" tool on your system and run them on all the patches before you post.

Doorbell will use software relay if vfe notify area is not 4K aligned

Interrupt will map to callfd through vfio ioctl VFE's Msix number should bigger than queue number. Interrupt map will fail if Msix number is less.
This is not clear. This is internal and doesnt belong to commit message. Move it to code comment.

Signed-off-by: Kailiang Zhou kailiangz@nvidia.com

paravmellanox

I stopped half way.
We need to see the code split into multiple patches.
patch-1 for virtio driver.
patch-2 for vdpa driver that uses virtio driver.
Fix all existing comments and apply them to rest of the code.

paravmellanox · 2022-04-21T14:12:34Z

drivers/vdpa/meson.build

@@ -9,6 +9,7 @@ drivers = [
        'ifc',
        'mlx5',
        'sfc',
+        'vfe',


instead of vfe all over, just say virtio

paravmellanox · 2022-04-21T14:12:46Z

drivers/vdpa/vfe/meson.build

+# Copyright(c) 2018 Intel Corporation
+
+deps += 'vhost'
+sources = files('vfe_vdpa.c','virtio.c','virtio_pci.c')


file name to be virtio_vdpa.c

paravmellanox · 2022-04-21T14:13:35Z

drivers/vdpa/vfe/rte_vhost.h

+ */
+
+#ifndef _RTE_VHOST_H_
+#define _RTE_VHOST_H_


this file should not be added. use the existing vhost header file.

if we use existing vhost header file, there will be a lot of redefine.
because virtio_pci.h also define some virtio macro, and vhost header include linux header file which also define virtio macro.

In file included from ../drivers/vdpa/vfe/virtio_pci.h:15,
from ../drivers/vdpa/vfe/vfe_vdpa.c:23:
../drivers/vdpa/vfe/virtio.h:48: warning: "VIRTIO_F_IOMMU_PLATFORM" redefined
48 | #define VIRTIO_F_IOMMU_PLATFORM 33
|
In file included from /usr/include/linux/vhost_types.h:16,
from /usr/include/linux/vhost.h:14,
from ../drivers/vdpa/vfe/rte_vhost.h:26,
from ../drivers/vdpa/vfe/vfe_vdpa.c:17:

paravmellanox · 2022-04-21T14:14:52Z

drivers/vdpa/vfe/vfe_vdpa.c

@@ -0,0 +1,1196 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2019 Mellanox Technologies, Ltd


use the year 2022.
There is new copyright format of nvidia. use that one.
above copied is no longer allowed.
refer to some latest added dpdk pmd files and use that.

paravmellanox · 2022-04-21T14:15:30Z

drivers/vdpa/vfe/vfe_vdpa.c

+RTE_LOG_REGISTER(vfe_vdpa_logtype, pmd.vdpa.vfe, NOTICE);
+#define DRV_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ## level, vfe_vdpa_logtype, \
+		"VFE %s(): " fmt "\n", __func__, ##args)


virtio_vdpa instead of VFE

paravmellanox · 2022-04-21T14:33:26Z

drivers/vdpa/vfe/vfe_vdpa.c

+
+	do {
+		nbytes = read(rte_intr_fd_get(virtq->intr_handle), &buf,
+				  8);


all to be in single line for 80 characters

paravmellanox · 2022-04-21T14:33:45Z

drivers/vdpa/vfe/vfe_vdpa.c

+	struct rte_intr_handle *intr_handle;
+	int retries = VFE_VDPA_INTR_RETRIES;
+
+


remove extra lines.
audit whole code and remove.

paravmellanox · 2022-04-21T14:34:40Z

drivers/vdpa/vfe/vfe_vdpa.c

+	hw_vq = hw->vqs[index];
+
+
+	if(enable==0) {


white space add

paravmellanox · 2022-04-21T14:34:57Z

drivers/vdpa/vfe/vfe_vdpa.c

+	int vid;
+	struct rte_vhost_vring vq;
+	uint64_t gpa;
+	struct virtio_hw *hw = NULL;


no need for null initialize

paravmellanox · 2022-04-21T14:35:19Z

drivers/vdpa/vfe/vfe_vdpa.c

+
+	gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.desc);
+	if (gpa == 0) {
+		DRV_LOG(ERR, "Fail to get GPA for descriptor ring.");


print device name

In rte_vhost_driver_unregister which is called from vdpa-rpc thread, vsocket should be removed from reconn_list again after remove vsocket from conn_list. Because vhost_user_read_cb which is called in vhost-events thread can add vsocket to reconn_list again. When qemu close domain socket server, vhost_user_read_cb will be called to clean up vhost device. vsocket->path is NULL #0 0x00007f07665834d1 in __strnlen_sse2 () from /lib64/libc.so.6 Mellanox#1 0x00007f076aee79da in vhost_user_add_connection (fd=160, vsocket=0x7f070406d160) at ../lib/vhost/socket.c:226 Mellanox#2 0x00007f076aee7d63 in vhost_user_client_reconnect (arg=<optimized out>) at ../lib/vhost/socket.c:481 Mellanox#3 0x00007f07668cbdd5 in start_thread () from /lib64/libpthread.so.0 Mellanox#4 0x00007f07665f4ead in clone () from /lib64/libc.so.6 RM: 3585558 Signed-off-by: Yajun Wu <yajunw@nvidia.com>

In rte_vhost_driver_unregister which is called from vdpa-rpc thread, vsocket should be removed from reconn_list again after remove vsocket from conn_list. Because vhost_user_read_cb which is called in vhost-events thread can add vsocket to reconn_list again. When qemu close domain socket server, vhost_user_read_cb will be called to clean up vhost device. vsocket->path is NULL #0 0x00007f07665834d1 in __strnlen_sse2 () from /lib64/libc.so.6 #1 0x00007f076aee79da in vhost_user_add_connection (fd=160, vsocket=0x7f070406d160) at ../lib/vhost/socket.c:226 #2 0x00007f076aee7d63 in vhost_user_client_reconnect (arg=<optimized out>) at ../lib/vhost/socket.c:481 #3 0x00007f07668cbdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f07665f4ead in clone () from /lib64/libc.so.6 RM: 3585558 Signed-off-by: Yajun Wu <yajunw@nvidia.com>

[ upstream commit 1c80a40 ] The net/vhost pmd currently provides a -1 vid when disabling interrupt after a virtio port got disconnected. This can be caught when running with ASan. First, start dpdk-l3fwd-power in interrupt mode with a net/vhost port. $ ./build-clang/examples/dpdk-l3fwd-power -l0,1 --in-memory \ -a 0000:00:00.0 \ --vdev net_vhost0,iface=plop.sock,client=1\ -- \ -p 0x1 \ --interrupt-only \ --config '(0,0,1)' \ --parse-ptype 0 Then start testpmd with virtio-user. $ ./build-clang/app/dpdk-testpmd -l0,2 --single-file-segment --in-memory \ -a 0000:00:00.0 \ --vdev net_virtio_user0,path=plop.sock,server=1 \ -- \ -i Finally stop testpmd. ASan then splats in dpdk-l3fwd-power: ================================================================= ==3641005==ERROR: AddressSanitizer: global-buffer-overflow on address 0x000005ed0778 at pc 0x000001270f81 bp 0x7fddbd2eee20 sp 0x7fddbd2eee18 READ of size 8 at 0x000005ed0778 thread T2 #0 0x1270f80 in get_device .../lib/vhost/vhost.h:801:27 Mellanox#1 0x1270f80 in rte_vhost_get_vhost_vring .../lib/vhost/vhost.c:951:8 Mellanox#2 0x3ac95cb in eth_rxq_intr_disable .../drivers/net/vhost/rte_eth_vhost.c:647:8 Mellanox#3 0x170e0bf in rte_eth_dev_rx_intr_disable .../lib/ethdev/rte_ethdev.c:5443:25 Mellanox#4 0xf72ba7 in turn_on_off_intr .../examples/l3fwd-power/main.c:881:4 Mellanox#5 0xf71045 in main_intr_loop .../examples/l3fwd-power/main.c:1061:6 Mellanox#6 0x17f9292 in eal_thread_loop .../lib/eal/common/eal_common_thread.c:210:9 Mellanox#7 0x18373f5 in eal_worker_thread_loop .../lib/eal/linux/eal.c:915:2 Mellanox#8 0x7fddc16ae12c in start_thread (/lib64/libc.so.6+0x8b12c) (BuildId: 81daba31ee66dbd63efdc4252a872949d874d136) Mellanox#9 0x7fddc172fbbf in __GI___clone3 (/lib64/libc.so.6+0x10cbbf) (BuildId: 81daba31ee66dbd63efdc4252a872949d874d136) 0x000005ed0778 is located 8 bytes to the left of global variable 'vhost_devices' defined in '.../lib/vhost/vhost.c:24' (0x5ed0780) of size 8192 0x000005ed0778 is located 20 bytes to the right of global variable 'vhost_config_log_level' defined in '.../lib/vhost/vhost.c:2174' (0x5ed0760) of size 4 SUMMARY: AddressSanitizer: global-buffer-overflow .../lib/vhost/vhost.h:801:27 in get_device Shadow bytes around the buggy address: 0x000080bd2090: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 0x000080bd20a0: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 0x000080bd20b0: f9 f9 f9 f9 00 f9 f9 f9 00 f9 f9 f9 00 f9 f9 f9 0x000080bd20c0: 00 00 00 00 00 00 00 f9 f9 f9 f9 f9 04 f9 f9 f9 0x000080bd20d0: 00 00 00 00 00 f9 f9 f9 f9 f9 f9 f9 00 00 00 00 =>0x000080bd20e0: 00 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9 04 f9 f9[f9] 0x000080bd20f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x000080bd2100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x000080bd2110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x000080bd2120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x000080bd2130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Thread T2 created by T0 here: #0 0xe98996 in __interceptor_pthread_create (.examples/dpdk-l3fwd-power+0xe98996) (BuildId: d0b984a3b0287b9e0f301b73426fa921aeecca3a) Mellanox#1 0x1836767 in eal_worker_thread_create .../lib/eal/linux/eal.c:952:6 Mellanox#2 0x1834b83 in rte_eal_init .../lib/eal/linux/eal.c:1257:9 Mellanox#3 0xf68902 in main .../examples/l3fwd-power/main.c:2496:8 Mellanox#4 0x7fddc164a50f in __libc_start_call_main (/lib64/libc.so.6+0x2750f) (BuildId: 81daba31ee66dbd63efdc4252a872949d874d136) ==3641005==ABORTING More generally, any application passing an incorrect vid would trigger such an OOB access. Fixes: 4796ad6 ("examples/vhost: import userspace vhost application") Signed-off-by: David Marchand <david.marchand@redhat.com> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

GavinLi-NV suggested changes Apr 21, 2022

View reviewed changes

kailiangz1 force-pushed the main branch from b3332dc to 2d929cd Compare April 21, 2022 03:27

paravmellanox reviewed Apr 21, 2022

View reviewed changes

kailiangz1 closed this Jul 5, 2022

kailiangz1 force-pushed the main branch from 2d929cd to 7607634 Compare July 5, 2022 08:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vdpa/vfe: Add new vdpa driver for virtual full emulation device #1

vdpa/vfe: Add new vdpa driver for virtual full emulation device #1

kailiangz1 commented Apr 21, 2022

GavinLi-NV left a comment

paravmellanox commented Apr 21, 2022

paravmellanox left a comment

paravmellanox Apr 21, 2022

paravmellanox Apr 21, 2022

paravmellanox Apr 21, 2022

kailiangz1 Apr 22, 2022

paravmellanox Apr 21, 2022

paravmellanox Apr 21, 2022

paravmellanox Apr 21, 2022

paravmellanox Apr 21, 2022

paravmellanox Apr 21, 2022

paravmellanox Apr 21, 2022

paravmellanox Apr 21, 2022

@@ @@ -9,6 +9,7 @@ drivers = [ @@
                       'ifc',
                       'mlx5',
                       'sfc',
+                      'vfe',

		@@ -0,0 +1,1196 @@
		/* SPDX-License-Identifier: BSD-3-Clause
		* Copyright 2019 Mellanox Technologies, Ltd

		struct rte_intr_handle *intr_handle;
		int retries = VFE_VDPA_INTR_RETRIES;

vdpa/vfe: Add new vdpa driver for virtual full emulation device #1

vdpa/vfe: Add new vdpa driver for virtual full emulation device #1

Conversation

kailiangz1 commented Apr 21, 2022

GavinLi-NV left a comment

Choose a reason for hiding this comment

paravmellanox commented Apr 21, 2022

paravmellanox left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment