CUDA RNDV protocols

Jump to bottom Edit New page

Devendar Bureddy edited this page May 16, 2020 · 2 revisions

The following are default rndv protocols with default UCX_RNDV_SCHEME=auto

Inter-node:

GPUDIrectRDMA enabled

D-D
Sender stages cuda fragments and PUTs to destination Device buffer
D-H
Sender stages cuda fragments and PUTs to destination Host buffer
H-D
put_zcopy protocol
H-H
get-zcopy protocol

GPUDIrectRDMA disabled

D-D
Receiver requests fragments from Sender (frag rtr) Sender stages fragments and PUTs to the receiver staging buffer and notifies (frag ATP) Receiver PUTs fragment to Device buffer.
D-H
Sender pipelines cuda fragments and PUTs to destination Host buffer
H-D
Receiver GETs host fragments and PUTs to destination Device buffer
H-H
get-zcopy protocol

Intra-node

CUDA-IPC

D-D
put_zcopy protocol
D-H
Sender stages cuda fragments and PUTs to destination Host buffer
H-D
put_zcopy protocol
H-H
get-zcopy protocol

No CUDA_IPC

D-D
Sender stages cuda fragments and PUTs(IB)to destination Device buffer
D-H
Sender pipelines cuda fragments and PUTs to destination Host buffer
H-D
put_zcopy protocol (IB).
H-H
get-zcopy protocol

UCX - best communication library ever