US What RDMA is
RDMA (remote direct memory access) is a networking technology that lets one system read from or write to another system’s memory without routing the data through the CPU at either end, reducing latency and freeing processor resources for the real workload.
In a conventional network stack, every packet is handled by the CPU: copied between buffers, checksummed and queued by the operating system before it reaches the application. On a busy edge AI node, that overhead competes directly with inference for the same processor time, and it adds latency that varies with load.
RDMA bypasses most of that work. The network card moves data straight between the memory of the two systems, with the CPU only setting up the transfer and being told when it is done.
How RDMA works
Three ideas make RDMA different from ordinary TCP/IP networking.
-
//
Zero-copy. Data goes straight from application memory to the network card and into the far system’s memory, with no intermediate copies in the operating system.
-
//
Kernel bypass. Applications post work directly to the network card through queue pairs, using the RDMA verbs interface, instead of making a system call for each packet.
-
//
One-sided operations. An RDMA read or write completes without the remote CPU taking part at all, once the memory has been registered and access granted.
-
//
Transport offload. Reliable delivery, ordering and retransmission run in the network card hardware rather than in software.
RDMA transports compared
RDMA is a capability, not a single network. Three transports deliver it.
| Transport | Runs over | Routable | Network needs | Typical use |
|---|---|---|---|---|
| InfiniBand | Dedicated InfiniBand fabric | Yes, within the fabric | InfiniBand adapters and switches | AI training clusters, supercomputers |
| RoCEv2 | Ethernet, inside UDP/IP | Yes | Switches with PFC and ECN, or a well-tuned lossy setup | Data centres, AI and storage over Ethernet |
| RoCEv1 | Ethernet, Layer 2 only | No | One broadcast domain | Largely replaced by RoCEv2 |
| iWARP | Ethernet, over TCP | Yes | Standard Ethernet switches | Storage and Windows SMB Direct |
Our InfiniBand FAQ covers the dedicated fabric. At the industrial edge, Ethernet is already in place, so RoCEv2 is usually the practical route to RDMA.
Why RDMA matters for edge AI
The practical effects for edge AI workloads are direct and measurable.
-
//
Lower latency. Data reaches the inference engine faster and with more predictable timing, which matters in vision-guided robotics, machine vision and closed-loop control.
-
//
Freed CPU time. Processor cores stay available for pre-processing, tracking and application logic instead of the network stack.
-
//
Better multi-node scaling. Clusters of edge devices behave more like one tightly coupled system, which becomes essential once a deployment moves beyond a single inference node.
-
//
GPU-to-GPU transfers. With NVIDIA GPUDirect RDMA, the network card reads and writes GPU memory directly, so frames and tensors skip system memory altogether.
GPUDirect RDMA
GPUDirect RDMA is NVIDIA’s technology that lets a third-party PCIe device, most often an RDMA network card, read and write GPU memory directly. Without it, data arriving from the network lands in system memory and is then copied to the GPU, which costs a second trip across PCIe, CPU time and latency. With it, the network card writes straight into GPU memory, and results can leave the same way.
It is one of a family. GPUDirect Peer to Peer moves data between GPUs in the same system, GPUDirect Storage reads NVMe drives straight into GPU memory, and GPUDirect for Video does the same for capture cards and frame grabbers, so camera frames reach the GPU without a stop in system memory.
Start with the GPU. GPUDirect RDMA is supported on NVIDIA’s data centre and professional RTX GPUs, not on consumer GeForce cards, so check the datasheet. Among the cards we list, the Advantech SKY-MXM-2000B (RTX PRO 2000 Blackwell) and SKY-MXM-A2000 (RTX A2000) MXM modules list both GPUDirect RDMA and GPUDirect for Video, as do the RTX Ada modules in our MXM GPU modules range. For full-size slots, RTX Ada professional cards such as the SKY-QUAD-4000EA-20 list GPUDirect RDMA too. Beyond the GPU, three more things have to line up.
-
//
Network card and driver. An RDMA network card whose driver can map GPU memory, such as NVIDIA ConnectX adapters with the nvidia-peermem module on Linux.
-
//
PCIe layout. Throughput is best when the GPU and the network card share a PCIe switch or root port. Paths that cross between processor sockets can be much slower.
-
//
Software. The application or library has to use it, as NCCL, CUDA-aware MPI and sensor frameworks such as NVIDIA Holoscan do.
Where RDMA is used
RDMA began in high-performance computing and is now the default for AI training clusters, where GPUs in different servers exchange data constantly. It also carries fast storage: NVMe over Fabrics and Windows SMB Direct both use RDMA to reach remote flash with close to local latency.
At the edge, the same patterns appear on a smaller scale. A multi-GPU edge AI server sharing work across nodes, a vision system streaming many uncompressed cameras to GPUs, or an inference cluster on a vehicle all benefit when data movement stops consuming CPU time. Our industrial GPU computers take PCIe network cards for this kind of build.
When RDMA is worth it at the edge
RDMA adds network cards, configuration and testing, so it earns its place only where data movement is a real cost.
-
//
Multi-node and multi-GPU systems. Worth it: several inference nodes sharing models or results, GPUs fed by many uncompressed cameras, and edge servers reaching shared NVMe storage.
-
//
Latency-critical control. Worth it: control loops where predictable latency matters more than average speed.
-
//
Single-node inference. Rarely needed: a single edge computer running inference on compressed camera streams, where the network carries a small fraction of its capacity.
-
//
Data gateways. Rarely needed: gateways that forward sensor data and summaries upstream, where TCP is simpler and more than fast enough.
What you need for RDMA
RDMA needs support at every layer. Both systems need RDMA-capable network cards, often called RNICs, with drivers for the operating system; on Linux that means the RDMA core stack and the verbs libraries. The application, or the framework it uses, has to use RDMA, as AI communication libraries, MPI and NVMe over Fabrics do. For RoCEv2, the switches must be configured to keep packet loss low.
Speed matters too. RDMA is most useful at 10GbE and above, where the CPU cost of moving data at full line rate becomes significant. Our article on what 10GbE changes for edge AI covers how link speed and RDMA work together at the edge.
Quick answers on RDMA
RDMA: common questions
Is RDMA faster than TCP?
Usually, yes. RDMA has lower latency and uses far less CPU at the same link speed, because the network card does the work the TCP stack would do in software.
Is RDMA the same as InfiniBand?
No. InfiniBand is one network that supports RDMA natively. RoCEv2 and iWARP bring RDMA to Ethernet.
Does RDMA need special switches?
It depends on the transport. InfiniBand needs InfiniBand switches. RoCEv2 runs on Ethernet switches, ideally with PFC and ECN configured. iWARP runs on any Ethernet switch.
What does RDMA mean in networking?
Direct memory-to-memory transfer between systems. The network card moves the data, so neither CPU copies it and neither operating system handles each packet.
Is RDMA used for storage?
Yes. NVMe over Fabrics, SMB Direct and many parallel file systems use RDMA to reach remote storage with low latency.
What is GPUDirect RDMA?
Direct transfers between the network and the GPU. It is NVIDIA’s technology that lets an RDMA network card read and write GPU memory directly, so data skips system memory and the CPU.
We build edge AI computers and servers as standard, specifying RDMA-capable network cards, PCIe layout and cooling where the workload needs them, and checking driver support for the operating system before the system ships.