HOW TO CALCULATE KV CACHE MEMORY

A conventional KV cache holds key and value tensors for retained tokens. Its logical memory depends on KV heads, not necessarily query heads. To estimate allocation, also account for layer types, cache precision and the serving engine's storage policy. Logical tensor size is not the same as reserved GPU memory.

Cache arithmetic and architecture reference

Start with the conventional cache shape

For an ordinary decoder cache, each layer retains keys and values for earlier tokens. Reusing those tensors avoids recomputing them at each generation step. Hugging Face describes the per-layer cache structure and its sequence dimension. Source: Hugging Face cache structure

For identical full-attention layers and equal sequence lengths, use this logical storage formula. It assumes both key and value tensors have the stated head dimension and element size. It does not include additional allocator or quantization metadata.

KV_bytes = 2 × L × Hkv × D × T × B × S

L   = number of cached attention layers
Hkv = key/value heads per layer
D   = elements per head
T   = retained tokens per sequence
B   = bytes per stored cache element
S   = independent active sequences

The two counts the key and value tensors. vLLM's documented paged-attention layout likewise separates cache blocks, KV heads, head size and tokens per block. Counting those dimensions gives the logical payload before allocation overhead. Source: vLLM cache tensor layout

If active sequences have different retained lengths, replace T × S with the sum of those lengths. This is a direct extension of the same arithmetic. Count each independently stored token state once; do not assume sharing unless the implementation actually provides it.

Count KV heads rather than query heads

Multi-head attention uses separate key/value heads corresponding to the attention heads. Multi-query attention shares one key/value head across query heads. Grouped-query attention uses an intermediate number of KV heads. The GQA research paper describes this distinction. Source: GQA paper by Ainslie and colleagues

This matters because the cache formula uses the number of stored KV heads. A query-head count can overstate logical cache if you substitute it for a smaller KV-head count. Conversely, assuming every model uses grouped-query attention can understate the requirement.

Inspect the exact model configuration. Transformers' Llama configuration documents num_attention_heads and num_key_value_heads separately. Its class defaults are not evidence for an arbitrary checkpoint. Check the saved configuration and implementation used in your deployment. Source: Llama configuration fields

Do not change the head count in a worksheet as though it were a free runtime compression switch. The architecture and trained weights determine the supported attention structure. A smaller mathematical result does not make an incompatible configuration valid.

Work through a numerical example

Assume 80 identical full-attention layers, a head dimension of 128, two-byte cache elements and 8,192 retained tokens. With eight KV heads, one sequence requires 2,684,354,560 bytes. That equals 2.5 GiB or approximately 2.684 GB.

For four independent sequences at that same length, multiply by four. The logical payload becomes 10 GiB, or approximately 10.737 GB. These are hypothetical values chosen to expose the arithmetic, not measured allocations for a named checkpoint.

one_sequence_bytes = 2 × 80 × 8 × 128 × 8,192 × 2
                   = 2,684,354,560

four_sequence_GiB = (one_sequence_bytes × 4) ÷ 2^30
                  = 10

The comparison below changes only the KV-head count. The 64-head case uses eight times the logical cache of the eight-head case. The one-head case uses one eighth. This isolates an architectural dimension without claiming equal model quality or serving performance.

For the same eight-head case, increasing retained tokens from 8,192 to 32,768 multiplies logical cache by four. At four active sequences, the result is 40 GiB. The weights remain a separate part of the inference budget.

Sum the actual layer requirements

A uniform full-attention formula is inappropriate when some layers retain a bounded window. Hugging Face documents cache growth stopping at the applicable window or chunk size for supported layer types. Mixed architectures need layer-specific accounting. Source: cache behavior for sliding and chunked layers

KV_bytes = sum over layers and independent sequences of:
           2 × KV_heads_in_layer × head_dimension_in_layer
             × retained_tokens_in_layer × bytes_per_element

Consider a second hypothetical configuration. Keep eight KV heads, dimension 128 and two-byte elements. Assume 40 full-attention layers retain 8,192 tokens, while 40 windowed layers retain 4,096. One sequence then contains 1.875 GiB of logical cache.

That result is 25% below the 2.5 GiB uniform example. It is not a 50% reduction because half the layers still retain the longer history. Four independent sequences would total 7.5 GiB before allocation details.

Check the model's actual layer schedule and the engine's supported cache behavior. Do not assume the maximum context length is the retained length in every layer. Architectures using other state representations need their own formula rather than forced substitution into this one.

Separate payload from allocation policy

Logical payload counts values needed for the stated token states. Allocated memory describes the storage the implementation commits. Reserved capacity can include space intended for future tokens or other requests. Keep these labels separate when comparing arithmetic with telemetry.

A static cache can preallocate its maximum size, while a dynamic cache grows with generation. The two may show different allocations for the same short request. That difference does not mean the model configuration or logical token count is wrong. Source: static and dynamic cache strategies

Paging changes allocation rather than tensor arithmetic

Paged attention organizes cache storage into blocks. Its documented layout includes tokens per block and a pool of blocks. Paging does not remove the need to store each retained key and value. Source: vLLM paged-attention design

For a simple illustrative allocation, assume blocks hold 16 tokens and one sequence retains 17 tokens. It needs capacity for two blocks, or 32 token positions. Only 17 positions contain that sequence's logical history. The unused capacity is an allocation effect.

This example excludes block tables, sharing and engine-specific policies. It does not assert that a particular engine allocates every layer in precisely this simplified way. Use it to understand why rounding can matter, then inspect the actual implementation.

Prefix reuse needs a workload assumption

vLLM's automatic prefix caching can reuse previously computed KV state for matching prefixes. Its documentation distinguishes benefits during prompt processing from new-token decoding. A high shared-prefix rate should not be assumed for unrelated requests. Source: automatic prefix caching and its limits

For procurement, retain a baseline without assumed sharing and a separate scenario with measured reuse. State the prefix distribution and eviction behavior relevant to that measurement. Do not reduce every request's cache budget by an invented universal savings percentage.

Check cache precision independently

Replacing two-byte elements with one-byte elements halves the logical element payload in the same formula. That is arithmetic, not a claim that complete GPU allocation halves. Other buffers and metadata remain outside the calculation.

vLLM's quantized-cache documentation describes supported data types and scale calibration. Compatibility and accuracy depend on the chosen implementation. Record the cache setting separately from weight quantization, and validate the intended task. Source: quantized cache configuration

If some layers use a different cache type, calculate their contribution separately. If cache is offloaded, record the placement rather than claiming its storage disappears. The device-resident requirement and the complete host requirement answer different questions.

Keep a reproducible cache record

Record the model revision, cached layer types, KV-head counts and head dimensions. Add cache precision, maximum retained tokens, active sequences and storage policy. Include the engine version because an implementation change can invalidate an earlier allocation assumption.

The memory planning worksheet keeps those inputs beside their sources. It also separates logical cache from measured runtime demand. Use the inference VRAM guide to combine the cache with weight storage and a deployment review.

Once the workload is defined, compare exact hardware variants such as H100 SXM5 80GB and MI300X OAM 192GB. A capacity comparison is a shortlist step. Confirm engine support, device distribution and the complete server before requesting an offer.

Common questions

Does KV cache use query heads or KV heads?

Use the stored KV-head count. Query heads can share key/value heads in grouped-query or multi-query attention. Read the exact configuration rather than assuming the two counts are equal.

Does doubling context double cache memory?

It doubles logical payload for the uniform full-attention formula when every other input stays fixed. Windowed layers, sharing and allocation policies can change the observed result. State which quantity and architecture you mean.

Does paged attention eliminate cache overhead?

No. Block allocation still has capacity rounding and supporting data structures. Paging changes storage management; it does not erase the logical keys and values. Compare measured allocation with a clearly labeled logical estimate.

Why does the engine reserve more memory than this formula predicts?

The formula counts a limited logical payload. The engine may reserve cache capacity beyond current usage and allocate other runtime data. Inspect the engine's settings and reports before treating the difference as an error.