How to build the development environment of Software RDMA over Converged Ethernet (RoCE) ?

Soft-RoCE (RXE)

In order to study the coding of RDMA, I need build the corresponding environment. Due to the lack of hardware, I found Soft-RoCE would be the first choice after some research work.

All the information focus on these two repositories: [rxe-dev] and [librxe-dev]. And others aim to help us how to build this virtual RDMA device which named RXE and how to use it.

Struggle Against RXE

Internet shows that the RXE need kernel support and user space codes.

Firstly, we git clone the [rxe-dev] to compile and install the new kernel to support RoCE. Then restart with the new kernel. Finally, compile the [librxe-dev] to get the utilities of Soft-Roce.

When I switched to the compiled new kernel, but it failed to start the system. When I configured the [librxe-dev], it showed “configure: error: <infiniband/driver.h> not found. librxe requires libibverbs”.

This page also shows the same issue, and several people also hangs on this error.

Sudden Inspiration

MosesAlexander’s comment of “I just found that the rxe functionality is all in rdma-core now.” gives me sudden inspiration. The rdma-core has already contained all the RXE utilities, and it looks like that there’s no maintenance for the two repositories [rxe-dev] and [librxe-dev].


Just “yum -y install libibverbs libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils” will be OK !

Notice:  I blog this post at 2018/09/03, based on CentOS 7 (3.10.0-862.el7.x86_64), whose kernel support RDMA-related technologies originally.


Run some commands to verify:

[root@localhost ~]# rxe_cfg start
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU 
ens33 yes e1000 
virbr0 no bridge 
virbr0-nic no tun 
[root@localhost ~]# rxe_cfg add ens33
[root@localhost ~]# rxe_cfg status
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU 
ens33 yes e1000 rxe0 1024 (3) 
virbr0 no bridge 
virbr0-nic no tun 
[root@localhost ~]# ibv_devices 
device node GUID
------ ----------------
rxe0 020c29fffe495c4d

Also, you could run example codes from the-geek-in-the-corner.




Tagged as: , , ,

2 replies

  1. ibv_rc_pingpong -d rxe0 -g 0

    local address: LID 0x0000, QPN 0x000011, PSN 0x9f9e68, GID fe80::9657:a5ff:fe5e:66ac

    No responding, Why?


Fill in your details below or click an icon to log in: 徽标

You are commenting using your account. Log Out /  更改 )

Google photo

You are commenting using your Google account. Log Out /  更改 )

Twitter picture

You are commenting using your Twitter account. Log Out /  更改 )

Facebook photo

You are commenting using your Facebook account. Log Out /  更改 )

Connecting to %s

这个站点使用 Akismet 来减少垃圾评论。了解你的评论数据如何被处理