Skip to content

Commit

Permalink
vhost: fix deadlock during vDPA SW live migration
Browse files Browse the repository at this point in the history
In a nested virtualization environment, running dpdk-vdpa
in QEMU-L1 for software live migration will result in a
deadlock between dpdk-vdpa and QEMU-L2 processes.
'rte_vdpa_relay_vring_used'->
'__vhost_iova_to_vva'->
'vhost_user_iotlb_rd_unlock(vq)'->
'vhost_user_iotlb_miss'-> send vhost message
'VHOST_USER_SLAVE_IOTLB_MSG' to QEMU-L2's vdpa socket, then
call 'vhost_user_iotlb_rd_lock(vq)' to hold the read lock
`iotlb_lock`.

But there is no place to release this read lock.

QEMU-L2 get the 'VHOST_USER_SLAVE_IOTLB_MSG', then call
'vhost_user_send_device_iotlb_msg' to send
'VHOST_USER_IOTLB_MSG' messages to dpdk-vdpa.

dpdk-vdpa will call vhost_user_iotlb_cache_insert and will
obtain the write lock `iotlb_lock`, but the read lock
`iotlb_lock` has not been released and will block here.

This patch add lock and unlock function to fix the deadlock.

Fixes: b13ad2d ("vhost: provide helpers for virtio ring relay")
Cc: stable@dpdk.org

Signed-off-by: Hao Chen <chenh@yusur.tech>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
  • Loading branch information
Hao Chen authored and mcoquelin committed Feb 6, 2024
1 parent 4b0a1fd commit 19639c3
Showing 1 changed file with 9 additions and 2 deletions.
11 changes: 9 additions & 2 deletions lib/vhost/vdpa.c
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
#include "rte_vdpa.h"
#include "vdpa_driver.h"
#include "vhost.h"
#include "iotlb.h"

/** Double linked list of vDPA devices. */
TAILQ_HEAD(vdpa_device_list, rte_vdpa_device);
Expand Down Expand Up @@ -147,7 +148,6 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)

int
rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m)
__rte_no_thread_safety_analysis /* FIXME: requires iotlb_lock? */
{
struct virtio_net *dev = get_device(vid);
uint16_t idx, idx_m, desc_id;
Expand Down Expand Up @@ -193,17 +193,21 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m)
if (unlikely(nr_descs > vq->size))
return -1;

vhost_user_iotlb_rd_lock(vq);
desc_ring = (struct vring_desc *)(uintptr_t)
vhost_iova_to_vva(dev, vq,
vq->desc[desc_id].addr, &dlen,
VHOST_ACCESS_RO);
vhost_user_iotlb_rd_unlock(vq);
if (unlikely(!desc_ring))
return -1;

if (unlikely(dlen < vq->desc[desc_id].len)) {
vhost_user_iotlb_rd_lock(vq);
idesc = vhost_alloc_copy_ind_table(dev, vq,
vq->desc[desc_id].addr,
vq->desc[desc_id].len);
vhost_user_iotlb_rd_unlock(vq);
if (unlikely(!idesc))
return -1;

Expand All @@ -220,9 +224,12 @@ rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m)
if (unlikely(nr_descs-- == 0))
goto fail;
desc = desc_ring[desc_id];
if (desc.flags & VRING_DESC_F_WRITE)
if (desc.flags & VRING_DESC_F_WRITE) {
vhost_user_iotlb_rd_lock(vq);
vhost_log_write_iova(dev, vq, desc.addr,
desc.len);
vhost_user_iotlb_rd_unlock(vq);
}
desc_id = desc.next;
} while (desc.flags & VRING_DESC_F_NEXT);

Expand Down

0 comments on commit 19639c3

Please sign in to comment.