What Is InfiniBand?

InfiniBand is a networking standard for connecting servers, storage and accelerators inside a data centre with very high bandwidth and very low latency. It was created in 1999 when two competing server I/O projects merged, and it is maintained by the InfiniBand Trade Association (IBTA). It is an open specification, but NVIDIA, through its acquisition of Mellanox, now supplies most InfiniBand hardware.

Each server connects through a host channel adapter (HCA), a network card that plugs into PCIe. HCAs connect to InfiniBand switches, which form a switched fabric. A subnet manager, running on a switch or a server, discovers the fabric, assigns addresses and calculates the routes. Applications then exchange data using remote direct memory access (RDMA): the HCA reads from or writes to memory on another machine without involving either operating system, which removes copies and keeps CPU load and latency low.

InfiniBand is also lossless by design. Credit-based flow control means a sender only transmits when the receiver has buffer space, so the fabric does not drop packets under congestion, unlike standard Ethernet.

InfiniBand generations are named by their per-lane data rate. Most links use four lanes (4x), which is the figure usually quoted for a port.

Rates are signalling rates. Earlier generations lost around a fifth to 8b/10b encoding, while FDR and later use more efficient encoding. Ports use QSFP-family or OSFP connectors with copper direct-attach cables for short runs inside a rack and optical transceivers between racks.

Ethernet and InfiniBand now reach similar raw speeds, so the difference lies in how the fabric behaves under load and how it is managed.

Applications rarely talk to InfiniBand hardware directly. They use libraries that map their communication onto RDMA verbs, the low-level interface the HCA exposes.

• MPI. The message passing interface used by most scientific and engineering simulation codes on supercomputers.
• NCCL and GPUDirect RDMA. NVIDIA’s collective communication library moves data between GPUs in different servers, and GPUDirect RDMA lets the HCA read GPU memory without staging it in system memory.
• Storage protocols. NVMe over Fabrics, iSER and parallel file systems use RDMA to give servers fast access to shared flash storage.
• IP over InfiniBand. Carries ordinary TCP/IP traffic for tools and services that are not RDMA-aware.

Because the same verbs interface also works on RoCE, much of this software runs on either fabric, which is why the choice between InfiniBand and Ethernet is often made on scale, latency and operations rather than application support.

InfiniBand is a back-end network: it links the machines inside a cluster to each other, while Ethernet carries user and management traffic. Its main uses are supercomputers, which make up a large share of the TOP500 list, large AI training clusters where hundreds or thousands of GPUs exchange data on every training step, and high-performance parallel storage.

At the edge, workloads look different. A single edge AI server runs inference on one to a few GPUs, so traffic flows in from cameras and sensors rather than between servers. Ethernet at 10 Gb/s or faster, sometimes with RoCE, is the usual choice there. Our explainer on 10GbE networking covers that tier. The same data centre GPUs used in training clusters also run in edge servers, connected over PCIe rather than an InfiniBand fabric, and our overview of AMD EPYC Embedded processors for edge servers covers the host side.

We build edge AI servers with the GPUs, PCIe expansion and high-speed Ethernet each workload needs as standard, and validate them under load before they ship. Tell us your models, camera count and data rates, and we will specify the system to match.