-
Notifications
You must be signed in to change notification settings - Fork 55
rxe dev: Home
alokprasad edited this page Dec 24, 2017
·
13 revisions
This repository contains a full kernel source tree, with the RoCE driver code located in the directory drivers/infiniband/hw/rxe
.
Kernel Space Driver
- Github: https://github.com/SoftRoCE/rxe-dev.git (this repository)
- Active Branch:
rxe_submission_v18
User Space Library
- Github: https://github.com/SoftRoCE/librxe-dev.git
- Current Version:
librxe-1.0.0
- Clone kernel git:
0.
git clone https://github.com/SoftRoCE/rxe-dev.git
0. Checkout branch 'rxe_submission_v18' - Compile kernel:
0. Enter the source directory
cd rxe-dev
0.cp /boot/config-$(uname –r) .config
0.make menuconfig
0. Need to enableCONFIG_RDMA_RXE
0. Need to enableCONFIG_INFINIBAND_ADDR_TRANS=y
andCONFIG_INFINIBAND_ADDR_TRANS_CONFIGFS=y
in new config file.config
0.make –j 32
0.make modules_install
0.make install
0.make headers_install INSTALL_HDR_PATH=/usr
- Verify that the new kernel entry is added (e.g. to grub); if not, need to add it manually.
- Boot with new kernel.
- Install the following package (example shown using RedHat):
0.
yum install perl-Switch
(name might vary according to distribution) - Make sure that the following upstream user space libraries are installed:
0.
libibverbs
0.libibverbs-devel
0.libibverbs-utils
0.librdmacm
0.librdmacm-devel
0.librdmacm-utils
( For ubuntu libibverbs-dev libswitch-perl rdmacm-utils ibverbs-utils ) - Compile and install user space library
librxe
: 0.git clone https://github.com/SoftRoCE/librxe-dev.git
0.cd librxe-dev
0../configure --libdir=/usr/lib64/ --prefix=
0.make
0.make install
- Load
ib_rxe
kernel module using therxe_cfg
script included in thelibrxe
RPM: 0.rxe_cfg start
(this might requiresudo
or root privileges) - Create RXE device over network interface (e.g.
eth0
): 0.rxe_cfg add eth0
- Use the
status
command to display the current configuration: 0.rxe_cfg status
- If configured successfully, you should see output similar to the following:
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
eth0 yes mlx4_en rxe0 1024 (3)
- If you are using a Mellanox HCA: Need to make sure that the
mlx4_ib
kernel module is not loaded (modprobe –rv mlx4_ib
) in the soft-RoCE machine. - Now you have an Infiniband device called “rxe0” that can be used to run any RoCE app.