CARDINALGPU memory bandwidth, GPU-to-GPU links and server networking describe different data paths. A useful hardware proposal identifies each path and its topology. Choose the interconnect around how the application distributes work, rather than the largest bandwidth number in a specification.
Start with the path the data takes
Local memory bandwidth describes transfers between an accelerator and its attached memory. PCIe connects devices within a host system. NVLink provides another connection between supported NVIDIA devices. InfiniBand or Ethernet may carry traffic between servers. These technologies can appear in one deployment because they serve different paths.
Draw the proposed server before comparing specifications. Include accelerators, CPU sockets, PCIe switches and network adapters. Then draw the connections between servers. Label each connection with its purpose rather than assuming that every line represents the same interface.
Scroll across the diagram to read every label
Download diagramScale-up commonly describes expanding a tightly connected compute domain. Scale-out commonly describes distributing work across additional systems. The physical boundary is not always one chassis. Ask the supplier to identify the actual domain and links instead of relying on those labels alone.
Scaling down means reducing allocated resources. It is not the name of a different interconnect. If a proposal uses an unfamiliar scaling term, ask which devices communicate and what happens to the application's work when the configuration changes.
Bandwidth numbers need a direction and a denominator
Write bits and bytes explicitly. Eight bits make one byte. A nominal 400 Gb/s network rate converts arithmetically to 50 GB/s before accounting for protocol overhead. It does not promise 50 GB/s of application payload.
Also distinguish one direction from simultaneous traffic in both directions. A specification may add send and receive bandwidth together. Comparing that aggregate with another product's one-direction rate can create a misleading twofold difference.
| Published label | What to establish before comparison |
|---|---|
| GB/s or TB/s memory bandwidth | Which accelerator and memory configuration; peak or measured |
| PCIe generation and lane width | Negotiated generation, active lanes and one-direction or aggregate convention |
| NVLink bandwidth | Per link or total per GPU; direction convention and installed topology |
| Ethernet or InfiniBand rate | Per port or aggregate, port mode and usable payload measurement |
| Collective benchmark bandwidth | Operation, rank count and the benchmark's normalization formula |
PCIe 5.0 uses 32 GT/s signaling with 128b/130b encoding. For an x16 link, the encoding-adjusted arithmetic is approximately 63.0 GB/s in each direction before further protocol overhead. Vendor tables often round an aggregate bidirectional figure to 128 GB/s. These conventions should not be mixed without explanation. Source: PCI-SIG implementation overview
GT/s describes transfers, not useful application bytes. The lane count also matters. A device installed in a physically long slot may operate with a different electrical width. Request the supported slot configuration and the observed negotiated link information from the actual server.
Two identical GPUs can have different communication paths
Trace a transfer from its source GPU to its destination. Does it remain behind one PCIe switch, cross a host bridge or cross between CPU domains? If NVLink is present, which pairs can use it? The answer is part of the hardware configuration.
NVIDIA's multi-GPU programming documentation makes peer access dependent on the system's PCIe or NVLink topology. It provides a capability query rather than treating all device pairs as equivalent. A procurement specification should therefore identify the topology that the application expects. Source: CUDA multi-GPU systems
For NVIDIA systems, a seller can supply the read-only output below as one topology record. Request the driver version and a mapping from GPU identifiers to physical units alongside it.
nvidia-smi topo -mThe matrix distinguishes paths through PCIe switches, host bridges and CPU interconnects, as well as NVLink connections. It also reports CPU and memory affinity. It is useful evidence of the reported layout, but it is not a throughput test. Source: NVIDIA System Management Interface topology documentation
A topology requirement can change the choice between otherwise similar offers. For example, a distributed job may need a specific pair of GPUs near one network adapter. A server with the same accelerator count but a different placement should be evaluated as a different configuration.
Bridges and switches are part of the offer
The H100 NVL 94GB and H100 PCIe 80GB references should not be read as proof that an offered pair includes bridges. Ask for their part numbers and quantities. Check that the spacing and installation match the proposed server.
An L40S has no NVLink support. That does not prevent every multi-GPU workload, but it rules out an NVLink-dependent proposal. Independent replicas and a model split across devices can have very different communication requirements. Source: L40S interconnect specifications
InfiniBand and Ethernet are fabric choices, not GPU memory
RDMA allows supported devices to transfer data using a memory-oriented transport. InfiniBand and RoCE provide different networking environments for such communication. They should not be treated as interchangeable names for a cable speed.
InfiniBand uses fabric management, including a subnet manager. RoCE carries RDMA over Ethernet; RoCEv2 includes IP and UDP encapsulation. Those differences affect the equipment and configuration to review. They do not establish a universal performance winner for every cluster. Source: NVIDIA RDMA programming manual
A RoCE proposal should specify the network design and congestion behavior expected by the selected stack. Existing Ethernet connectivity alone is not sufficient evidence. Request the supported switch configuration and a test at the intended scale. Source: NVIDIA RoCE documentation
Keep management, application traffic and storage traffic visible in the design. They may share infrastructure or use separate networks, but that is a design decision. Ask what other traffic is present during the performance demonstration and during normal operation.
GPUDirect RDMA is a supported path, not an automatic feature of every installation
NVIDIA describes GPUDirect RDMA as a direct exchange path between GPU memory and a peer device such as a network adapter. Its documentation also describes platform constraints. The GPU, adapter, PCIe arrangement and software must be evaluated together. Source: GPUDirect RDMA overview
Ask the supplier to identify the path actually used by the application. Do not accept a feature name as proof that transfers avoid an unintended staging path. Record the relevant versions and test configuration so the result can be reproduced.
Match the network investment to the work
Independent inference replicas
If each replica holds its own model and handles separate requests, begin with the service's input, output and model-loading needs. Do not buy a tightly coupled GPU fabric solely because another cluster uses one. Establish whether communication is actually a limiting stage.
Test the deployment pattern, including restarts and model loading where those operations matter. A steady-state benchmark can miss a slow recovery process. Keep that operational requirement separate from the throughput of a single request.
One model distributed across accelerators
Identify the chosen parallelism strategy and communication pattern. Ask how frequently data moves and between which ranks. A memory-only device count does not specify the topology needed to execute efficiently.
Compare alternative arrangements that complete the same workload. Fewer GPUs with more memory may change the distribution strategy; more GPUs may add compute and communication together. Measure the resulting application rather than assuming either direction is better.
A training job spanning servers
Use the intended node count and effective batch in the evaluation. A single-node result does not establish scaling behavior. Record time spent waiting for communication and any change in useful progress per unit time.
Ask how the fabric behaves when several jobs run together. The proposal should explain resource sharing and the test conditions. An isolated demonstration may not represent the operating environment you plan to use.
Collective tests need interpretation
A collective operation coordinates data across participating processes, often called ranks. The operation, message size and rank count influence how a benchmark should be read. Keep those parameters beside the reported bandwidth.
NVIDIA's NCCL tests distinguish algorithm bandwidth from a normalized bus-bandwidth metric. Their documentation explains that the conversion depends on the collective operation. The bus figure is not a direct measurement of one physical cable's payload rate. Source: NCCL test performance definitions
Request the full benchmark command, software versions and topology record. Include both small and large messages relevant to the application. Then run the actual workload. A communication test helps diagnose a system; it does not establish end-to-end model performance.
Retain failed results and configuration changes in the evaluation record. If a test only passes after moving devices or changing firmware, that final configuration belongs in the accepted bill of materials. Do not leave the change as an undocumented setup detail.
Request a complete network proposal
Ask for the accelerator and server models, NIC part numbers, switch models, optics, cable types and lengths. Identify port modes and any required software entitlement. Keep spares and optional expansion items separate from the initial deployment.
For a used offer, ask the seller to identify the actual adapters and switches, their firmware state and included accessories. Define the required evidence and acceptance process before shipment. Cardinal reviews seller-supplied documents; it does not claim to test the proposed fabric.
Attach a diagram to the sourcing request when topology matters. A precise diagram can resolve ambiguity that a long list of component names leaves open. Use the server compatibility guide to connect the network proposal to the host platform.
Questions buyers ask about GPU networking
Is NVLink the same as InfiniBand?
No. They describe different interconnect technologies and can serve different paths in one installation. Identify which devices each link connects before comparing bandwidth.
Does 400 Gb/s mean 400 GB/s?
No. Dividing by eight gives 50 GB/s before protocol overhead. Also check whether a figure describes one direction, one port or an aggregate.
Will two GPUs always communicate directly over PCIe?
No. Peer access depends on the devices, platform and software configuration. Request the topology and capability evidence for the exact system.
Do I need InfiniBand for every multi-GPU deployment?
No. The requirement depends on where communication occurs and how the application distributes work. Evaluate a supported fabric against a defined workload and operating target.
Can a good collective benchmark replace an application test?
No. It answers a narrower communication question. Use it alongside the actual application test, with both configurations documented.