Soft-RoCE (RXE)

In order to study the coding of RDMA, I need build the corresponding environment. Due to the lack of hardware, I found Soft-RoCE would be the first choice after some research work.

All the information focus on these two repositories: [rxe-dev] and [librxe-dev]. And others aim to help us how to build this virtual RDMA device which named RXE and how to use it.

Struggle Against RXE

Internet shows that the RXE need kernel support and user space codes.

Firstly, we git clone the [rxe-dev] to compile and install the new kernel to support RoCE. Then restart with the new kernel. Finally, compile the [librxe-dev] to get the utilities of Soft-Roce.

When I switched to the compiled new kernel, but it failed to start the system. When I configured the [librxe-dev], it showed “configure: error: <infiniband/driver.h> not found. librxe requires libibverbs”.

This page also shows the same issue, and several people also hangs on this error.

Sudden Inspiration

MosesAlexander’s comment of “I just found that the rxe functionality is all in rdma-core now.” gives me sudden inspiration. The rdma-core has already contained all the RXE utilities, and it looks like that there’s no maintenance for the two repositories [rxe-dev] and [librxe-dev].

Solution

Just “yum -y install libibverbs libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils” will be OK !

Notice:  I blog this post at 2018/09/03, based on CentOS 7 (3.10.0-862.el7.x86_64), whose kernel support RDMA-related technologies originally.

Verification

Run some commands to verify:

[root@localhost ~]# rxe_cfg start
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU 
ens33 yes e1000 
virbr0 no bridge 
virbr0-nic no tun 
[root@localhost ~]# rxe_cfg add ens33
[root@localhost ~]# rxe_cfg status
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU 
ens33 yes e1000 rxe0 1024 (3) 
virbr0 no bridge 
virbr0-nic no tun 
[root@localhost ~]# ibv_devices 
device node GUID
------ ----------------
rxe0 020c29fffe495c4d

Also, you could run example codes from the-geek-in-the-corner.