Skip to content

VMA over Ubuntu 16.04 and inbox driver

Jump to bottom
Liran Oz edited this page Nov 13, 2018 · 3 revisions

This page will help you get VMA installed and running on a clean Ubuntu 16.04 with the inbox drivers coming from Ubuntu.
Meaning, you will not need any additional MLNX_OFED driver besides what you get from Redhat and the latest VMA copy. Please login as root for the below instructions.

1. Install Ubuntu 16.04 (beta)

Follow the installation guide from Ubuntu. Make sure Kernel is newer than 4.4. Upgrade if needed.

$ uname -r
4.4.0-2-generic
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial  
DISTRIB_DESCRIPTION="Ubuntu Xenial Xerus (development branch)"

2. Verify Mellanox NIC is connected properly

$ lspci | grep Mellanox
15:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
15:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
1a:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
1a:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]

3. Install Prerequisites:

Basic drivers for user space access for Ethernet and Infiniband:

$ apt-get install dkms infiniband-diags libibverbs* ibacm librdmacm* libmlx4* libmlx5* mstflint libibcm.* libibmad.* libibumad* opensm srptools libmlx4-dev librdmacm-dev rdmacm-utils ibverbs-utils perftest vlan ibutils

Additional build tools:

$ apt-get install libtool autoconf automake build-essential ibutils ibverbs-utils rdmacm-utils infiniband-diags perftest librdmacm-dev libibverbs-dev libmlx4-1 numactl libnuma-dev autoconf automake gcc g++ git libtool pkg-config

Important! use libnl3:

$ apt-get install libnl-3-200 libnl-route-3-200 libnl-route-3-dev libnl-utils

4. Run the related drivers

Do once after install: Add modules to Kernel

$ echo mlx4_ib >> /etc/modules
$ echo ib_umad >> /etc/modules
$ echo ib_cm >> /etc/modules
$ echo ib_ucm >> /etc/modules
$ echo rdma_ucm >> /etc/modules

Load related drivers in system:

$ modprobe mlx4_ib; modprobe ib_umad; modprobe ib_cm; modprobe ib_ucm; modprobe rdma_ucm

5. Configure NIC port type as Ethernet

Install MFT Tools
Start mst:
$ mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success

Get PCI details:

$ mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded


MST devices:
------------
/dev/mst/mt4103_pciconf0         - PCI configuration cycles access.
                                 domain:bus:dev.fn=0000:02:00.0 addr.reg=88 data.reg=92
                                 Chip revision is: 00
/dev/mst/mt4103_pci_cr0          - PCI direct access.
                                 domain:bus:dev.fn=0000:02:00.0 bar=0xc7200000 size=0x100000
                                 Chip revision is: 00
Change the port to Eth:
$ mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
$ reboot

Verify drivers are up

$ ibstat
CA 'mlx4_0'  
    CA type: MT4103
    Number of ports: 2
    Firmware version: 2.34.5000
    Hardware version: 0
    Node GUID: 0xe41d2d03001d6690
    System image GUID: 0xe41d2d03001d6693
    Port 1:
            State: Down
            Physical state: Polling
            Rate: 10
            Base lid: 0
            LMC: 0
            SM lid: 0
            Capability mask: 0x02504868
            Port GUID: 0xe41d2d03001d6691
            Link layer: InfiniBand
    Port 2:
            State: Down
            Physical state: Polling
            Rate: 10
            Base lid: 0
            LMC: 0
            SM lid: 0
            Capability mask: 0x02504868
            Port GUID: 0xe41d2d03001d6692
            Link layer: InfiniBand

$ ibv_devinfo  
hca_id: mlx4_0
    transport:                      InfiniBand (0)
    fw_ver:                         2.34.5000
    node_guid:                      e41d:2d03:001d:6690
    sys_image_guid:                 e41d:2d03:001d:6693
    vendor_id:                      0x02c9
    vendor_part_id:                 4103 
    hw_ver:                         0x0  
    board_id:                       MT_1090111019
    phys_port_cnt:                  2
            port:   1
                    state:                  PORT_DOWN (1)
                    max_mtu:                4096 (5)
                    active_mtu:             4096 (5)
                    sm_lid:                 0
                    port_lid:               0
                    port_lmc:               0x00
                    link_layer:             InfiniBand

            port:   2
                    state:                  PORT_DOWN (1)
                    max_mtu:                4096 (5)
                    active_mtu:             4096 (5)
                    sm_lid:                 0
                    port_lid:               0
                    port_lmc:               0x00
                    link_layer:             InfiniBand

6. Download, build and install libvma from source:

$ git clone https://github.com/Mellanox/libvma.git
$ cd libvma
$ ./autogen.sh
$ ./configure --prefix=/usr --libdir=/usr/lib64
$ make -j
$ make install
$ ldconfig

7. Configure and load driver modules for ConnectX-3:

$ echo options mlx4_core log_num_mgm_entry_size=-1 >> /etc/modprobe.d/mlnx.conf
$ modprobe -rv mlx4_en mlx4_ib mlx4_core
$ modprobe -v mlx4_en mlx4_ib mlx4_core rdma_ucm

8. Memory settings for high performance and RDMA

Configure huge-pages for better libvma performance:

$ echo 1000000000 > /proc/sys/kernel/shmmax
$ echo 800 > /proc/sys/vm/nr_hugepages

Relax memlock restrictions for users allowing more direct memory mapping for HW.
Create file /etc/security/limits.d/rdma.conf with the following:

 # configuration for rdma tuning
 *       soft    memlock         unlimited
 *       hard    memlock         unlimited
 # rdma tuning end

Enable new limit for the user.
You will need to logout and back in and then run:

$ ulimit -l unlimited

9. Load libvma and run the app (as root):

$ LD_PRELOAD=libvma.so <test_app>

10. For running as user (as non root user):

Set cap_net_raw for the executable. For example to use sockperf:

$ setcap cap_net_raw=ep /usr/bin/sockperf

Set special permission for library (SET_UID) and place in standard location (which libvma is now already). These steps are required when using LD_PRELOAD with capabilities set on executable.

$ chmod u+s /usr/lib64/libvma.so.8*
Clone this wiki locally