# LLM inference memory planning worksheet

Cardinal Field Guide working template. This estimates memory components, not application throughput or a guaranteed deployment. Record unknown inputs as unknown.

## Workload identity

- Model repository and exact revision:
- Configuration source:
- Verified parameter count:
- Serving engine and version:
- Accelerator software environment:
- Target time to first token:
- Target output rate per active request:
- Aggregate throughput target:
- Typical prompt and generated lengths:
- Longest supported prompt plus continuation:
- Active sequences under the planned load:
- Measurement owner:

## Weight storage

- Weight representation:
- Bytes per parameter assumed:
- Which modules use another representation:
- Quantization scales and metadata accounted for:
- Additional model buffers accounted for:
- Measured loaded weight allocation, if available:
- Source and observation date:

```text
weight_bytes = parameter_count × bytes_per_parameter
weight_GB = weight_bytes / 1,000,000,000
weight_GiB = weight_bytes / 1,073,741,824
```

This arithmetic excludes metadata and other model allocations unless explicitly added.

## KV cache architecture

- Number of cached attention layers:
- Query heads per layer:
- KV heads per layer:
- Head dimension:
- Full-attention, windowed or other layer structure:
- Retained tokens for each layer type:
- Cache element type and bytes per element:
- Independent active sequences:
- Shared prefixes assumed: none / measured scenario
- Cache offloading or other placement:
- Cache allocation policy:
- Source and configuration revision:

For uniform full-attention layers:

```text
logical_KV_bytes = 2 × layers × KV_heads × head_dimension
                   × retained_tokens_per_sequence
                   × cache_bytes_per_element × active_sequences
```

For different sequence lengths, replace tokens times sequences with the sum of independently stored token lengths. For mixed layer types, calculate and sum each layer's contribution.

The factor of two counts keys and values. This conventional formula does not cover every architecture. It excludes allocator metadata, block rounding and replication unless explicitly modeled.

## Runtime plan

- Logical weight storage GB:
- Logical cache storage GB:
- Separately known additional allocations GB:
- Remaining allowance method: measured / explicit assumption
- Assumed allowance fraction, if used:
- What the allowance is intended to cover:
- What remains excluded or unknown:
- Planned total GB:
- Planned total GiB:

```text
base_GB = weights_GB + logical_cache_GB + separately_known_allocations_GB
assumed_allowance_GB = base_GB × allowance_fraction
planned_GB = base_GB + assumed_allowance_GB
```

Do not add the same allocation twice. A percentage allowance is not a universal safety margin. If measured allocations replace an assumption, record the measurement conditions.

## Device distribution

- Candidate exact accelerator variant:
- Capacity per device and stated units:
- Proposed device count:
- Parallelism method and supported configuration:
- Expected allocation on the most heavily loaded device:
- Replicated data or uneven placement:
- Interconnect and host platform:
- Separate model replicas:
- Technical compatibility owner:

```text
memory_only_count_lower_bound = ceiling(planned_GB / assumed_device_capacity_GB)
```

This lower bound assumes perfect distribution. It does not validate the device count, platform, software or throughput.

## Scenario comparison

| Scenario | Precision | Retained tokens | Active sequences | Logical cache | Planned total | Evidence status |
|---|---|---|---|---|---|---|
| Representative workload | | | | | | |
| Longest-request case | | | | | | |
| Higher-concurrency case | | | | | | |

## Measurement record

- Run date and owner:
- Exact hardware and software:
- Input distribution and request count:
- Measurement duration:
- Memory measurement definition and tool:
- Measured allocation per device:
- First-token latency distribution:
- Subsequent-token latency or output rate:
- Aggregate completed throughput:
- Failures, timeouts or preemptions:
- Differences from the planned configuration:
- Acceptance targets met or missed:
- Unresolved questions and next action:

## Reference guides

- https://cardinalsilicon.com/learn/llm-inference-vram/
- https://cardinalsilicon.com/learn/kv-cache-memory/
- https://huggingface.co/docs/transformers/en/cache_explanation
- https://docs.vllm.ai/en/latest/configuration/conserving_memory/

Cardinal sources hardware and reviews seller-supplied documents. The responsible technical team validates application behavior and installation suitability.