CARDINALInference precision is a configuration, not a single number. A model can store four-bit weights, process activations in BF16, accumulate selected operations at higher precision and keep its attention cache in another format. Buyers should qualify that complete configuration before using quantization to choose a GPU.
Four precision decisions sit inside one deployment
The stored weight format describes how the model's learned parameters are represented. Quantizing those parameters can reduce their storage footprint, but the saved checkpoint also needs information that explains how to interpret its compressed values. Scales, grouping and the quantization method matter. A filename containing “4bit” is not a complete deployment specification. Hugging Face explains the underlying quantization concepts.
Activation precision describes values produced while processing a request. Weight-only quantization can leave these values at a different precision. The accumulator is another choice: the intermediate sum inside an operation need not use the same representation as its inputs. Finally, the KV cache stores attention state across tokens. Changing weight storage does not automatically change cache storage.
Hugging Face's bitsandbytes integration makes the distinction concrete: its four-bit configuration has a separate compute-dtype setting. That setting should be recorded alongside the quantization type. Do not describe an entire deployment as four-bit arithmetic merely because its linear-layer weights use a four-bit representation. See the bitsandbytes configuration documentation.
Scroll across the diagram to read every label
Download diagramWhat BF16, FP16, FP8, INT8 and four-bit actually tell you
BF16 and FP16 both use sixteen bits per represented value, but their numerical properties differ. BF16 has a wider exponent range; FP16 allocates more bits to fractional precision. Neither is universally more accurate for every operation. Overflow, rounding and sensitive reductions are reasons to validate the actual model rather than selecting a format by its name. NVIDIA's accuracy guidance also explains why selected operations may need higher precision. See TensorRT accuracy considerations.
| Label | What it describes | What to ask next |
|---|---|---|
| BF16 / FP16 | Different sixteen-bit floating-point representations | Which operations, accumulation settings and GPU kernels use each? |
| FP8 | A family of eight-bit floating-point formats | Which encoding, scaling recipe and native or weight-only execution path? |
| INT8 | Eight-bit integer quantization | Are both weights and activations quantized, and how are scales determined? |
| INT4 / four-bit | A storage width or named quantized representation | Which method, grouping, metadata and compute dtype? |
| W4A16 | Four-bit weights with sixteen-bit activations | Does the chosen engine support this checkpoint on the exact GPU? |
| FP8 KV cache | Quantized attention-state storage | Which cache encoding, scales and attention backend are used? |
FP8 is not one interchangeable encoding. E4M3 and E5M2 divide their exponent and fraction bits differently. Scaling is part of using these narrow formats effectively. NVIDIA's Transformer Engine documentation discusses recipes and hardware support; a recipe from one environment should not be assumed to describe another engine's checkpoint. Read the Transformer Engine low-precision primer.
Likewise, INT8 and FP8 have the same nominal bit width but different numerical representations. Neither label by itself states whether activations are quantized. Four-bit integer weights, NF4 and newer floating-point formats are also different configurations. In a sourcing brief, preserve the method's exact name instead of compressing everything into “low precision.”
vLLM's W4A16 workflow provides a practical example: its recipe quantizes selected linear weights to four-bit integers and includes calibration and evaluation steps. The example also leaves a named layer outside the quantization target. This illustrates why the average model footprint can differ from a simple bits-per-parameter calculation. See the official W4A16 workflow.
An FP8 checkpoint does not prove native FP8 arithmetic
Separate “the engine can load this model” from “this GPU performs its matrix operations natively in FP8.” Current vLLM documentation describes W8A8 execution on supported Ada, Hopper and Blackwell GPUs, while supported Turing and Ampere paths can use weight-only W8A16 with Marlin. Those are different execution paths for an FP8-labeled checkpoint. Compatibility must be checked against the actual release and model. See vLLM's FP8 support and kernel-selection notes.
That distinction matters when comparing an A100 PCIe 80GB with an H100 PCIe 80GB. Loading success on both is not evidence of identical arithmetic. Conversely, a lack of native support for one format does not establish that no compressed checkpoint can run through an alternative supported path.
Ask for the engine version, GPU architecture, checkpoint revision and selected backend. Keep startup logs where they identify the kernel. For models with experts, confirm the expert-layer backend separately when the engine exposes that choice. A successful linear-layer path does not settle every operator used by a model.
The general vLLM quantization matrix is a useful starting point because it separates methods and hardware. It is not a lifetime guarantee or a substitute for the method-specific page. Pin the version used for a qualification run; an unversioned “latest” documentation link can change after a purchasing decision. Review the quantization support matrix.
Half the weight bytes does not mean twice the tokens per second
Reducing weight traffic can help a workload constrained by memory movement. A different workload may spend more time on arithmetic, attention, communication or overhead. Matrix dimensions and batch shape also affect utilization. NVIDIA's matrix-multiplication guide explains the distinction between memory-limited and arithmetic-limited operations, which is why a storage ratio cannot be used as a universal speed multiplier. See the matrix-multiplication performance guide.
Compare the measurements that match your service. Record prompt length, output length, concurrent requests, time to first token and generation latency. A configuration that improves aggregate throughput may still miss an interactive response target. Keep the baseline and quantized runs on the same workload and report their memory footprints separately.
The same caution applies to capacity. A hypothetical sixteen-billion-parameter set contains 32 billion raw bytes at two bytes per weight, or 8 billion raw bytes at half a byte per weight. Those are arithmetic illustrations, not complete memory requirements. Metadata, unquantized tensors, cache and runtime allocations remain. Use the inference VRAM guide to build a complete budget.
Quantizing the KV cache is a separate optimization. vLLM documents cache-specific types and scaling controls; changing a weight-quantization setting is not the same action. Validate the attention backend and the effect on output quality with the context lengths you will actually serve. The KV-cache memory guide explains the capacity calculation. See vLLM's cache-quantization documentation.
Dense and structured-sparse TFLOPS belong in separate comparison columns. A sparse peak assumes eligible data and execution; it does not describe arbitrary dense model weights. Nor does a tensor-core peak describe every operation in a service. NVIDIA explicitly footnotes its H100 low-precision peaks with sparsity. Read that condition before comparing an H100 SXM5 80GB against another card's dense figure. See the H100 specification footnote.
Make output quality an acceptance criterion
Choose the quality test before choosing the quantized candidate. For a support assistant, include retrieval-grounded answers, misleading documents and requests requiring abstention. For code generation, use executable tests. For extraction, check required fields and schema compliance. These are suggested evaluation designs, not claims that a particular quantization method passes them.
Keep the reference checkpoint, tokenizer, chat template and generation settings identifiable. Compare representative requests, including the difficult cases your users bring. A short general benchmark can be useful, but it should not replace the domain-specific acceptance test. Record failures as well as aggregate scores.
Calibration data and evaluation data should serve different purposes. Calibration helps construct a quantized representation; evaluation judges whether the resulting deployment meets requirements. Preserve the recipe and the calibration dataset identity so another engineer can reproduce the candidate. If a checkpoint supplier cannot explain its recipe, that uncertainty belongs in the deployment decision.
Turn the qualified configuration into a purchasing brief
Give a hardware supplier a workload specification rather than only a model name. Include the exact checkpoint, quantization method, weight and activation formats, cache dtype, engine version, required context, concurrency and latency target. State whether you have already validated the configuration or are seeking hardware for a qualification trial.
A buyer considering L40S PCIe 48GB for a smaller serving deployment should establish the complete memory footprint and execution path first. A buyer considering H200 NVL 141GB for greater memory headroom still needs the same software evidence. More capacity does not resolve an unsupported kernel or an unacceptable quality change.
Separate must-haves from options. “Serve the approved checkpoint at the tested settings” is a requirement. “Try a smaller quantized checkpoint later” is an optimization candidate. Buying against the second statement before validating it can turn an assumed memory saving into a deployment constraint.
Buyer questions
Does a four-bit model use only half a byte per parameter in GPU memory?
That is the raw storage width for four-bit values. It excludes scales, other metadata, higher-precision tensors and runtime allocations. Measure the loaded model and complete serving configuration before deciding the GPU count.
Can BF16 and FP16 be substituted freely?
No. They use the same number of bits but represent values differently. Check hardware and engine support, numerical behavior and application quality before changing either setting.
Does an FP8 model require an H100?
Not universally. A supported weight-only execution path can differ from native FP8 arithmetic, and other architectures have supported paths. Verify the exact method, GPU and engine release instead of making a purchase from the filename.
Should every layer and the KV cache use the lowest precision?
No universal rule establishes that. Treat each change as a candidate to validate. Keeping sensitive operations or cache storage at a different precision may be appropriate for the approved model and workload.
What should accompany a claimed quantization speedup?
The reference configuration, exact hardware, software versions, model revisions, request distribution, latency and throughput measurements, memory usage and quality results. A percentage without that context is insufficient for sizing a deployment.