Soft-RoCE (RXE)
In order to study the coding of RDMA, I need build the corresponding environment. Due to the lack of hardware, I found Soft-RoCE would be the first choice after some research work.
All the information focus on these two repositories: [rxe-dev] and [librxe-dev]. And others aim to help us how to build this virtual RDMA device which named RXE and how to use it.
Struggle Against RXE
Internet shows that the RXE need kernel support and user space codes.
Firstly, we git clone the [rxe-dev] to compile and install the new kernel to support RoCE. Then restart with the new kernel. Finally, compile the [librxe-dev] to get the utilities of Soft-Roce.
When I switched to the compiled new kernel, but it failed to start the system. When I configured the [librxe-dev], it showed “configure: error: <infiniband/driver.h> not found. librxe requires libibverbs”.
This page also shows the same issue, and several people also hangs on this error.
Sudden Inspiration
MosesAlexander’s comment of “I just found that the rxe functionality is all in rdma-core now.” gives me sudden inspiration. The rdma-core has already contained all the RXE utilities, and it looks like that there’s no maintenance for the two repositories [rxe-dev] and [librxe-dev].
Solution
Just “yum -y install libibverbs libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils” will be OK !
Notice: I blog this post at 2018/09/03, based on CentOS 7 (3.10.0-862.el7.x86_64), whose kernel support RDMA-related technologies originally.
Verification
Run some commands to verify:
[root@localhost ~]# rxe_cfg start Name Link Driver Speed NMTU IPv4_addr RDEV RMTU ens33 yes e1000 virbr0 no bridge virbr0-nic no tun [root@localhost ~]# rxe_cfg add ens33 [root@localhost ~]# rxe_cfg status Name Link Driver Speed NMTU IPv4_addr RDEV RMTU ens33 yes e1000 rxe0 1024 (3) virbr0 no bridge virbr0-nic no tun [root@localhost ~]# ibv_devices device node GUID ------ ---------------- rxe0 020c29fffe495c4d
Also, you could run example codes from the-geek-in-the-corner.
分类:RDMA
ibv_rc_pingpong -d rxe0 -g 0
local address: LID 0x0000, QPN 0x000011, PSN 0x9f9e68, GID fe80::9657:a5ff:fe5e:66ac
No responding, Why?
赞赞
No ideas without more related information, but you’d like to check this post: https://www.rdmamojo.com/2015/01/24/verify-rdma-working/.
PS: Dotan Barak wrote lots of RDMA-related blogs, maybe you’ll get the answer from his site.
赞赞