US What separates the H100 and H200
The H100 vs H200 question comes down to memory. Both are NVIDIA data centre GPUs built on the Hopper architecture, using the same GH100 chip with the same fourth-generation Tensor Cores, Transformer Engine and FP8 support. On the SXM versions the peak compute figures are identical.
NVIDIA launched the H100 in 2022. It announced the H200 in November 2023 as the first GPU with HBM3e, and it reached customers during 2024. The H200 raises memory from 80 GB to 141 GB and bandwidth from 3.35 TB/s to 4.8 TB/s, about 1.76 times as much capacity and 1.4 times the bandwidth.
That matters because much of today’s AI work, especially large language model inference, is limited by how fast the GPU can read its memory rather than by how fast it can calculate.
H100 and H200 versions compared
Each GPU comes in more than one form, and the versions differ in power and memory as well as in how they connect to other GPUs. These are NVIDIA’s published figures.
| Version | Memory | Bandwidth | Max power | Form factor | GPU-to-GPU link | MIG instances |
|---|---|---|---|---|---|---|
| H100 PCIe | 80 GB HBM2e | 2.0 TB/s | 300–350 W | Dual-slot PCIe card | NVLink bridge, 600 GB/s | Up to 7 × 10 GB |
| H100 NVL | 94 GB HBM3 | 3.9 TB/s | 350–400 W | Dual-slot PCIe card | NVLink bridge, 600 GB/s | Up to 7 × 12 GB |
| H100 SXM | 80 GB HBM3 | 3.35 TB/s | Up to 700 W | SXM5 module on HGX board | NVSwitch, 900 GB/s | Up to 7 × 10 GB |
| H200 NVL | 141 GB HBM3e | 4.8 TB/s | Up to 600 W | Dual-slot PCIe card | 2- or 4-way bridge, 900 GB/s | Up to 7 × 16.5 GB |
| H200 SXM | 141 GB HBM3e | 4.8 TB/s | Up to 700 W | SXM module on HGX board | NVSwitch, 900 GB/s | Up to 7 × 18 GB |
The PCIe and NVL cards fit standard air-cooled rack servers. The H100 PCIe 80 GB card, the H100 NVL 94 GB card and the H200 NVL 141 GB card are all passive dual-slot cards that rely on the server’s fans. SXM modules only come on HGX baseboards inside dedicated eight-GPU servers.
Why the extra memory matters
A GPU running a large model spends much of its time fetching weights and cached data from memory. More capacity and more bandwidth help in three ways.
-
//
Bigger models on fewer GPUs. A 70-billion-parameter model stored at 8-bit precision needs roughly 70 GB for its weights. That leaves an H200 with around 70 GB for everything else, against about 10 GB on an 80 GB H100.
-
//
Longer context and more users. The key-value (KV) cache that stores conversation context grows with every token and every concurrent user. Spare memory lets one GPU serve longer prompts or more requests.
-
//
Faster token generation. Generating each token means reading the model’s weights, so 4.8 TB/s against 3.35 TB/s translates fairly directly into higher throughput. NVIDIA quotes up to around 1.9 times the H100’s Llama 2 70B inference speed for the H200.
-
//
Larger MIG slices. Split into seven Multi-Instance GPU slices, an H200 gives each one 16.5–18 GB instead of 10 GB, enough for a small language model per slice.
HBM2e, HBM3 and HBM3e
High-bandwidth memory (HBM) stacks DRAM dies vertically and places the stacks beside the GPU on the same package, connected through a very wide interface. It is the reason data centre GPUs reach terabytes per second where graphics cards with GDDR memory reach hundreds of gigabytes per second. JEDEC publishes the HBM standards.
The H100 PCIe uses HBM2e, the H100 SXM and NVL use HBM3, and the H200 uses HBM3e, the faster extension of HBM3 with taller stacks. The H200 has six HBM3e stacks, which accounts for both its higher capacity and its higher bandwidth.
Because the stacks sit on the GPU package, HBM cannot be upgraded or added to later. The memory that comes with the card is the memory it keeps for life, so it is worth sizing it for the largest model you expect to run over the system’s working life.
Which one to choose
For training, where compute usually sets the pace, the two GPUs perform much alike, and the H100 remains a sound choice where the model and batch fit in 80 or 94 GB. The H200 pays off for inference on models of roughly 30 billion parameters and up, for long-context work, and wherever it lets you serve a model from one GPU instead of two. Both draw most of their value from software, so frameworks built on CUDA and TensorRT run on either without changes.
In an air-cooled rack, the NVL cards are the practical route. A four-GPU chassis such as the SKY-622G4 GPU server takes H200 NVL cards with NVLink bridges between them.
Quick answers on H100 vs H200
The comparison raises a handful of recurring questions, mainly about generations and compatibility.
H100 and H200: common questions
Does the H200 use the Blackwell architecture?
No, it is a Hopper GPU. The H200 uses the same Hopper architecture as the H100 with faster, larger memory. Blackwell is the next generation, used in the B200 and GB200.
Is the H200 better than the H100?
For memory-bound work, yes. It has more memory and more bandwidth with the same compute cores, so it is faster for large-model inference and roughly equal for compute-bound training.
Can an H100 and an H200 be linked with NVLink?
Not as a matched pair. NVLink bridges are specified for identical cards, and the H100 NVL and H200 NVL use different bridges. Mixed GPUs can share a server and communicate over PCIe.
When was the H200 released?
Announced in November 2023. It reached customers in 2024 as an SXM module, and NVIDIA added the H200 NVL PCIe card in late 2024.
Do the H100 and H200 run the same software?
Yes. Both use the same CUDA, cuDNN and TensorRT stack, so models and containers move between them without changes beyond memory sizing.
We fit H100 and H200 cards in the GPU servers we build as standard, with power, airflow and any NVLink bridges set up for the card count, and we can help you size the memory to the models you plan to run.