In the Linux kernel, the following vulnerability has been resolved:
RDMA/mlx5: Fix a race for an ODP MR which leads to CQE with error
This patch addresses a race condition for an ODP MR that can result in a CQE with an error on the UMR QP.
During the _mlx5ibderegmr() flow, the following sequence of calls occurs:
mlx5revokemr() mlx5rumrrevokemr() mlx5rumrpostsend_wait()
At this point, the lkey is freed from the hardware's perspective.
However, concurrently, mlx5ibinvalidate_range() might be triggered by another task attempting to invalidate a range for the same freed lkey.
This task will: - Acquire the umemodp->umemmutex lock. - Call mlx5rumrupdate_xlt() on the UMR QP. - Since the lkey has already been freed, this can lead to a CQE error, causing the UMR QP to enter an error state [1].
To resolve this race condition, the umemodp->umemmutex lock is now also acquired as part of the mlx5revokemr() scope. Upon successful revoke, we set umem_odp->private which points to that MR to NULL, preventing any further invalidation attempts on its lkey.
[1] From dmesg:
infiniband rocep8s0f0: dumpcqe:277:(pid 0): WC error: 6, Message: memory bind operation error cqedump: 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 cqedump: 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 cqedump: 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 cqe_dump: 00000030: 00 00 00 00 08 00 78 06 25 00 11 b9 00 0e dd d2
WARNING: CPU: 15 PID: 1506 at drivers/infiniband/hw/mlx5/umr.c:394 mlx5rumrpostsendwait+0x15a/0x2b0 [mlx5ib] Modules linked in: ip6tablemangle ip6tablenatip6tablefilter ip6tables iptablemangle xtconntrack xtMASQUERADE nfconntracknetlink nfnetlink xtaddrtype iptablenat nfnat brnetfilter rpcsecgsskrb5 authrpcgss oidregistry overlay rpcrdma rdmaucm ibiser libiscsi scsitransportiscsi rdmacm iwcm ibumad ibipoib ibcm mlx5ib ibuverbs ibcore fuse mlx5core CPU: 15 UID: 0 PID: 1506 Comm: ibvrcpingpong Not tainted 6.12.0-rc7+ #1626 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014 RIP: 0010:mlx5rumrpostsendwait+0x15a/0x2b0 [mlx5ib] [..] Call Trace: <TASK> mlx5rumrupdatexlt+0x23c/0x3e0 [mlx5ib] mlx5ibinvalidaterange+0x2e1/0x330 [mlx5ib] _mmunotifierinvalidaterangestart+0x1e1/0x240 zappagerangesingle+0xf1/0x1a0 madvisevmabehavior+0x677/0x6e0 domadvise+0x1a2/0x4b0 _x64sysmadvise+0x25/0x30 dosyscall64+0x6b/0x140 entrySYSCALL64afterhwframe+0x76/0x7e