Skip to content

Commit

Permalink
Make io_uring_for_each_cqe() only read the tail once
Browse files Browse the repository at this point in the history
There's no need to read the tail with acquire semantics for each loop,
it can just be done at the start of the loop. Iterating CQEs is
inherently racy anyway, if the ring is setup with signal notifications
for CQE postings.

Not the prettiest code in the world, but some hoops have to get jumped
through to avoid the compiler noticing that the passed in 'head' is now
effectively unused.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
  • Loading branch information
axboe committed Sep 30, 2024
1 parent 2bfc0d5 commit c16017c
Showing 1 changed file with 11 additions and 4 deletions.
15 changes: 11 additions & 4 deletions src/include/liburing.h
Original file line number Diff line number Diff line change
Expand Up @@ -305,15 +305,22 @@ int __io_uring_get_cqe(struct io_uring *ring,
#define io_uring_cqe_index(ring,ptr,mask) \
(((ptr) & (mask)) << io_uring_cqe_shift(ring))

/*
* NOTE: we should just get rid of the 'head' being passed in here, it doesn't
* serve a purpose anymore. The below is a bit of a work-around to ensure that
* the compiler doesn't complain about 'head' being unused (or only written,
* never read), as we use a local iterator for both the head and tail tracking.
*/
#define io_uring_for_each_cqe(ring, head, cqe) \
/* \
* io_uring_smp_load_acquire() enforces the order of tail \
* and CQE reads. \
*/ \
for (head = *(ring)->cq.khead; \
(cqe = (head != io_uring_smp_load_acquire((ring)->cq.ktail) ? \
&(ring)->cq.cqes[io_uring_cqe_index(ring, head, (ring)->cq.ring_mask)] : NULL)); \
head++) \
for (__u32 __HEAD__ = (head) = *(ring)->cq.khead, \
__TAIL__ = io_uring_smp_load_acquire((ring)->cq.ktail); \
(cqe = ((head) != __TAIL__ ? \
&(ring)->cq.cqes[io_uring_cqe_index(ring, __HEAD__, (ring)->cq.ring_mask)] : NULL)); \
(head) = ++__HEAD__)

/*
* Must be called after io_uring_for_each_cqe()
Expand Down

0 comments on commit c16017c

Please sign in to comment.