HOW TO READ A DATACENTER GPU SPECIFICATION

A GPU specification describes a component under stated conditions. A purchase decision needs one more step: connecting those numbers to your software, server and workload. Start with the exact variant, calculate memory needs, then check the system around it.

Calculations are illustrative, not measured benchmarks

Start with the exact variant

A model name is a useful search term. It is not a complete order description. Ask for the manufacturer part number, memory configuration and form factor. Record whether an offer includes a bare module, an add-in card or a complete server.

For example, NVIDIA lists H100 SXM with 80 GB of memory and H100 NVL with 94 GB per GPU. The NVL pair has 188 GB in aggregate. That does not turn either card into a single 188 GB device. NVIDIA also lists different power and interconnect specifications for these variants. Source: NVIDIA H100 specifications

Use a two-level shortlist. Put the model in the first column and the exact variant in the second. Keep the seller's part number beside both. This makes differences visible before you compare prices.

Do not assume a familiar connector means a compatible installation. A module needs its intended platform. A PCIe card still needs the correct slot space, power delivery and cooling. Ask the server manufacturer to confirm the exact combination.

Memory capacity answers only the first question

Memory capacity tells you how much data can reside on an accelerator. For language-model inference, budget separately for model weights, the key-value cache and runtime allocations. A weights-only calculation is a lower bound, not a configuration recommendation.

Start with this arithmetic: parameter count multiplied by bytes per parameter. A hypothetical model with 70 billion parameters needs 140 billion bytes for BF16 weights at two bytes each. That is 140 decimal GB, or approximately 130.4 GiB.

These values describe idealized weight storage. Quantized formats can require scales, metadata and some higher-precision tensors. The framework can reserve memory beyond live tensors. Test the actual model artifact and serving configuration before buying against a narrow margin.

Account for the KV cache

During autoregressive generation, the key-value cache preserves attention data from earlier tokens. Longer sequences can require more cache memory. Cache implementations also make different memory and speed tradeoffs. Hugging Face documents dynamic, static, offloaded and quantized cache strategies. Source: Transformers cache strategies

For a conventional full-attention cache, a useful starting calculation is:

KV bytes = 2 × layers × KV heads × head dimension
           × cached tokens × bytes per element
           × concurrent sequences

The factor of two represents keys and values. Use the model's KV-head count, which can differ from its query-head count. Treat cached tokens as the full retained context, including generated tokens.

Consider a hypothetical architecture with 80 layers, eight KV heads and a head dimension of 128. At 8,192 cached tokens and two bytes per element, one sequence requires 2.5 GiB. Four equal-length sequences require 10 GiB. These are explicit example assumptions, not specifications for every 70B model.

Sliding-window attention, shared prefixes, cache quantization and allocation strategies can change the result. Add runtime and workspace needs separately. A fixed percentage can help early budgeting, but it cannot guarantee that a model will fit.

Aggregate memory is not automatically usable memory

Two accelerators provide two pools of physical memory. Software must partition or distribute the workload to use both. The partitioning method introduces communication and can leave uneven allocations. Confirm support in the intended runtime instead of dividing total memory by card capacity and rounding up.

Training needs a different estimate. Gradients, optimizer states and saved activations can dominate memory beyond weights. The result depends on optimizer, precision, batch size and checkpointing. Do not use an inference weights calculation as a training purchase specification.

Separate three kinds of bandwidth

A memory bandwidth figure describes movement between an accelerator and its local memory. GPU interconnect bandwidth concerns communication between accelerators. Network bandwidth concerns communication across the wider system. These numbers measure different paths.

Read the units carefully. A byte contains eight bits, so 400 Gb/s is 50 GB/s before protocol overhead. That arithmetic does not establish delivered application bandwidth. Also check whether a published number combines directions or represents one direction.

For a multi-GPU offer, request a topology diagram. A count of eight GPUs says little about which devices communicate directly. Ask which bridges, baseboards, adapters and switches are included. Those details belong in the offer, not in a later assumption.

Read the footnotes below the performance number

Peak compute describes a theoretical arithmetic rate. It does not tell you how many requests your application can serve. Precision, operation type and sparsity assumptions must match before two headline numbers are comparable.

NVIDIA marks several H100 Tensor Core figures as using sparsity. A sparse figure should not be compared directly with a dense figure as though they describe the same work. Source: H100 specification footnotes

Make a small comparison sheet with separate columns for precision, dense or sparse operation, power limit and software version. Leave unsupported values empty. A blank field is more useful than a confident comparison built from incompatible numbers.

For inference, define a latency target alongside throughput. Record time to first token, output-token speed and the distribution of request lengths. Measure at the intended concurrency. A throughput result with an unacceptable response time does not satisfy the same requirement.

For training, compare time to a defined result using the same data and quality target. Record the full machine configuration. A benchmark using a different model, framework or communication setup answers a different question.

Check what surrounds the accelerator

A viable configuration combines the accelerator with a supported host, adequate cooling and a working software stack. Ask for the server model, CPU configuration, system memory, firmware and installed accelerator software.

Power arithmetic helps expose missing assumptions. Eight accelerators configured at 700 W total 5.6 kW for the accelerators alone. CPUs, memory, drives, fans and conversion losses add to the system requirement. This is a component calculation, not a facility power estimate.

Ask your hosting provider to confirm rack power and cooling for the complete server. Ask the integrator to confirm the hardware bill of materials. Ask your software team to run the actual workload on a representative configuration.

Before requesting offers, turn your shortlist into a short acceptance brief. State the application, expected load, required capacity and acceptable evidence. This makes sellers answer the same question and makes the resulting offers easier to compare.

Questions buyers ask

Can I choose a GPU by memory capacity alone?

No. Memory capacity is an initial constraint. You still need compatible software, sufficient communication bandwidth and a supported server. Validate the workload with room for its runtime allocations.

Does four-bit mean any model becomes four times smaller?

The idealized weight payload uses one quarter of BF16 storage. Complete memory use includes other tensors, metadata, cache and runtime allocations. Model quality and supported kernels also need validation.

What should I send with a quote request?

Include the exact variant if known, quantity, workload, server environment, delivery location and timing. If the variant is uncertain, share the constraints you know. The used GPU buying guide includes a copyable request brief.