HOW MUCH VRAM DOES LLM INFERENCE NEED

Plan LLM inference memory by adding model weights, the expected KV cache and a documented runtime allowance. Then check how the serving software distributes that memory across devices. A memory estimate narrows the hardware shortlist; it does not predict token throughput or guarantee that a deployment fits.

Memory planning with a hypothetical 70B workload

Define the workload before choosing memory

Write down the exact model revision, serving framework and intended precision. Record the longest supported request and the number of active sequences you need to serve. A request for “a GPU for a 70B model” leaves most of the memory plan unspecified.

Use the LLM memory planning worksheet to separate verified inputs from assumptions. Keep the source of each input beside its value. If the model or framework changes, the worksheet makes the affected assumptions visible.

Describe the service requirement in user terms as well. State how long someone can wait for the first token and how quickly the answer must continue. Record expected prompt and response lengths. Those targets determine which measurements you will need after checking capacity.

This guide covers decoder-style inference with a conventional KV cache. It does not estimate full training, optimizer state or every multimodal architecture. Hugging Face distinguishes inference caching from training use; a training plan needs different memory accounting. Source: Hugging Face caching overview

Calculate the weight-storage starting point

The basic arithmetic is parameter count multiplied by storage bytes per parameter. It is a useful lower-level component of the budget, not a complete deployment requirement. Hugging Face's optimization guide discusses weight precision as one contributor to inference memory. Source: Hugging Face inference optimization

weight_bytes = parameter_count × weight_bytes_per_parameter
weight_GB = weight_bytes ÷ 1,000,000,000
weight_GiB = weight_bytes ÷ 1,073,741,824

For a hypothetical model with exactly 70 billion parameters, two-byte storage gives 140 GB. One-byte storage gives 70 GB, while half-byte storage gives 35 GB. The table below reports the same arithmetic in binary gibibytes.

These are original storage calculations, rounded to two decimal places where needed. They do not claim that a named checkpoint contains exactly 70 billion stored parameters. Confirm the actual checkpoint and its loaded representation before relying on the result.

Quantization needs an implementation check

A “4-bit model” label does not mean every byte in the deployment scales to one quarter of a 16-bit installation. Quantization can affect selected weight modules while other data uses different types. The supported backend and hardware also matter. Source: Hugging Face bitsandbytes integration

Ask which modules are quantized, how scales and other metadata are stored, and whether the exact format works in the intended engine. Record any quality evaluation required by your application. A smaller representation is useful only if it still meets the workload's functional requirements.

Do not apply the weight-storage setting automatically to the cache. Weight precision and cache precision are separate configuration choices. vLLM exposes an explicit cache data-type setting with its own supported formats and calibration considerations. Source: vLLM quantized KV cache

Add the cache for active sequences

A conventional KV cache stores key and value tensors for processed tokens. Its logical size depends on layer count, KV heads, head dimension, cached token count and element size. The KV cache reference explains the formula and architecture exceptions.

For this worked plan, assume 80 identical full-attention layers, eight KV heads and a head dimension of 128. Assume two-byte cache elements, 8,192 cached tokens per sequence and four independent active sequences. These are hypothetical inputs, not a claim about a named model.

KV_bytes = 2 × layers × KV_heads × head_dimension
           × cached_tokens_per_sequence × bytes_per_element
           × active_sequences

KV_bytes = 2 × 80 × 8 × 128 × 8,192 × 2 × 4
         = 10,737,418,240 bytes
         = 10 GiB
         ≈ 10.737 GB

The factor of two counts keys and values. This calculation follows the ordinary cache tensor dimensions described by Hugging Face. It excludes sharing, windowed layers, allocation rounding and quantization metadata. Source: cache tensor layout

Use the total tokens retained at the point you are sizing. For a simple full-attention scenario, include the prompt and generated continuation. Do not enter only the expected output length while omitting the prompt already in the cache.

Also distinguish active sequences from registered users. A user population is not a simultaneous cache allocation. Document the concurrency you actually intend to support, together with the workload distribution and any admission limits.

Add an explicit runtime allowance

Runtime behavior adds memory beyond the simple weights-and-cache sum. For example, vLLM documents additional GPU memory used by CUDA graphs. The framework configuration can therefore change the memory requirement even when the checkpoint is unchanged. Source: vLLM memory conservation guide

For illustration, apply a 20% planning allowance to weights plus logical cache. This matches the adjustable allowance in Cardinal's memory estimator. It is not a measured runtime value or a universal safety margin. Replace it with configuration-specific measurements when available.

base_GB = weight_GB + KV_GB
allowance_GB = base_GB × assumed_allowance_fraction
planned_GB = base_GB + allowance_GB

planned_GB = (140 + 10.73741824) × 1.20
           = 180.884901888 GB
           ≈ 168.46 GiB

The model does not separately calculate quantization metadata or every runtime buffer. The percentage is a placeholder for planning uncertainty, not proof that those omitted allocations are covered. Keep this limitation attached when sharing a result.

Test the assumptions with a sensitivity table

Change one variable at a time. In this hypothetical case, doubling active sequences from four to eight doubles logical cache from 10 to 20 GiB. Keeping the same 16-bit weights and allowance gives approximately 193.770 GB.

That comparison shows which input changed the result. It does not mean every framework allocates memory identically. Static cache allocation and dynamic growth differ, and supported sliding-window layers can stop retaining additional tokens beyond their window. Source: Hugging Face cache strategies

Check how memory lands on each device

Aggregate capacity is only a first filter. Dividing 180.885 GB by an assumed 80 GB per device gives a mathematical lower bound of three devices. That result assumes perfect distribution and says nothing about a supported three-device deployment.

Tensor parallelism divides model-layer work across accelerators and requires communication between them. The implementation must support the model and intended layout. Hugging Face explicitly notes the importance of fast communication for this approach. Source: Hugging Face tensor parallelism

Pipeline parallelism distributes layers instead. vLLM documents both strategies and configuration-specific tradeoffs. A hardware shortlist should therefore include the supported parallelism plan, not just the sum of all memory labels. Source: vLLM parallelism and scaling

Ask the deployment engineer for the expected allocation on the most heavily loaded device. Record replicated buffers, uneven placement and any separate model replicas. If those details are unknown, label the device count as a memory-only lower bound.

Measure the service requirement separately

Memory fit does not produce a defensible tokens-per-second prediction. Prepare a benchmark that represents the intended prompt lengths, generated lengths and concurrent load. Specify the exact model revision, precision, engine version and hardware configuration.

Report time to first token separately from the rate of subsequent output. Keep an aggregate service throughput figure separate from an individual user's experience. Record the measurement period, request count and treatment of failed or timed-out requests.

Include latency distributions rather than a single best result. For procurement, a reproducible record is more useful than a peak number without its workload. vLLM provides production metrics that can support a version-specific measurement plan. Source: vLLM production metrics

Inspect behavior under cache pressure as well. vLLM documents preemption and recomputation when cache capacity is insufficient, with potential latency effects. A service that starts successfully can still behave poorly at its intended load. Source: vLLM optimization and preemption

Retain both a representative scenario and a stated boundary scenario. For example, use your normal request distribution and a separately labeled longest-request case. Do not present the easier scenario as evidence for the harder one.

Record a result someone else can review

A useful benchmark handoff includes the test input description, configuration file and a dated results summary. Name the person or team responsible for the measurement. Distinguish a measured result from a supplier estimate or a manufacturer reference.

Record whether the run met each acceptance target. If it missed a target, preserve the finding and describe the proposed change. Repeating a measurement after changing precision or concurrency creates a new scenario, not a correction to the old result.

Keep the original memory estimate beside the measured allocation. Explain the difference where possible and mark unresolved differences. That record helps the next purchasing decision without turning one successful run into an unsupported guarantee for a larger workload.

Turn the result into a hardware brief

Include the memory worksheet with the sourcing request. State which values are measured and which remain assumptions. Ask for the exact accelerator variant and a supported server arrangement, with unresolved software questions identified before purchase.

The H100 SXM5 80GB reference and H200 SXM 141GB reference illustrate why variant-level comparison matters. Their capacities are inputs to a review, not automatic deployment recommendations. Confirm the exact variant's documents and platform requirements.

When comparing a larger-memory device with several smaller ones, request complete supported configurations for both options. Include integration, power and cooling requirements. The server power and cooling guide covers that purchasing boundary.

Cardinal can source against the resulting brief and review the documents sellers supply. Keep the application's performance validation with the responsible technical team. A sourced accelerator should satisfy a documented requirement, not an unexplained calculator recommendation.

Common questions

How much VRAM does a 70B model need?

For exactly 70 billion parameters, raw two-byte weights require 140 GB before cache and runtime allocations. The complete requirement depends on the checkpoint, precision, active sequences and serving implementation. Use a stated workload to calculate the next components.

Does 4-bit quantization make a model four times smaller?

Half-byte weight arithmetic is one quarter of two-byte weight arithmetic. That ratio does not apply automatically to the entire deployment. Cache, unquantized modules, metadata and runtime behavior require separate accounting.

Is adding 20% enough headroom?

There is no universal guarantee. In this guide, 20% is an illustrative adjustable allowance. Replace it with measurements from the intended configuration and document the workload envelope those measurements cover.

Can I add GPU memory together?

You can sum capacities as an initial arithmetic check. A working deployment also needs supported distribution, sufficient capacity on each device and suitable communication. The summed number alone does not establish those conditions.