Inference Cost Analysis of Prefill and Decode

In large language model (LLM) inference, text generation is typically divided into two distinct phases: Prefill and Deco

In large language model (LLM) inference, text generation is typically divided into two distinct phases: Prefill and Decode. Prefill consumes the entire user prompt in a single forward pass and produces the first output token. Decode then generates subsequent tokens one at a time, conditioned on the cached key-value states. Because these phases differ fundamentally in compute patterns, memory access, and latency behavior, inference cost must be modeled separately rather than as a single monolithic workload.

Computational Cost Estimation

Let P denote the number of model parameters, S the prompt length, and b the number of bytes per floating-point element. Ignoring the quadratic term of attention for a rough estimate:

Prefill compute: FLOPs_prefill ≈ 2 × P × S
Decode per-step compute: FLOPs_decode ≈ 2 × P
Decode per-step memory traffic: Bytes_decode ≈ P × b + KV_cache_bytes

Prefill latency is approximately:

T_prefill ≈ FLOPs_prefill / (GPU_FLOPS × utilization)

Decode latency is approximately:

T_decode ≈ Bytes_decode / memory_bandwidth

Prefill is compute-bound. It performs large matrix multiplications across all prompt tokens in parallel, so its cost scales with prompt length and is limited by GPU peak throughput and matrix-multiplication efficiency. Decode is memory-bound. Each step runs the full Transformer stack but processes only a single token, reducing computation to matrix-vector products with low arithmetic intensity. Its cost is dominated by reading model weights and the KV cache from VRAM.

KV Cache Cost Analysis

During Prefill, the model computes and stores Key and Value tensors for every input token. During Decode, each step reads the historical KV cache and appends the current token's KV pair. The KV cache size can be expressed as:

KV_bytes = 2 × L × n_kv_heads × d_head × S × b

where L is the number of layers, n_kv_heads is the number of KV heads, d_head is the per-head dimension, and b is the precision in bytes.

For Llama-2 7B with L=32, n_kv_heads=32, d_head=128, and FP16 (b=2):

Per-token KV cache: approximately 0.5 MB
1,000-token context: roughly 500 MB

With Grouped Query Attention (GQA), reducing n_kv_heads to 8 brings the per-token cost down to about 0.125 MB, or roughly 125 MB for 1,000 tokens. Multi-Query Attention (MQA) and KV quantization can further shrink this footprint. These techniques directly alleviate the bandwidth pressure during Decode, where the cache must be fetched at every generation step.

Latency and Throughput Metrics

Two metrics capture the user-facing performance of these phases:

Time To First Token (TTFT): driven primarily by Prefill latency. Long prompts increase TTFT because the entire prompt must be processed before the first token appears.
Time Per Output Token (TPOT), also called Inter-Token Latency (ITL): driven primarily by Decode latency. This determines how quickly the rest of the response streams to the user.

System throughput—tokens generated per second—depends on batch size, KV cache capacity, and scheduling policy. Batching affects the two phases differently:

Prefill batching improves GPU utilization but can increase TTFT for individual requests if they wait for a batch to fill.
Decode batching amortizes the cost of reading model weights across multiple sequences, boosting throughput, but KV cache usage grows linearly with batch size.

Advanced scheduling strategies such as Continuous Batching and Chunked Prefill allow Prefill and Decode work to coexist in the same batch. Continuous Batching dynamically inserts new requests as others finish, keeping Decode pipelines busy. Chunked Prefill splits long prompts into smaller chunks, smoothing TTFT variance and preventing a single long prompt from monopolizing the compute cluster.

Example Estimation

Consider an NVIDIA A100 80GB running a 7B parameter model in FP16. The model weights occupy roughly 14 GB. The A100 offers about 2 TB/s of memory bandwidth and roughly 312 TFLOPS of FP16 peak compute.

For a single Decode step, the theoretical lower bound is:

14 GB / 2 TB/s ≈ 7 ms

In practice, the KV cache must also be read, and bandwidth utilization rarely hits the peak, so per-token latency typically falls in the 10–20 ms range.

For Prefill with a 1,000-token prompt:

FLOPs = 2 × 7 × 10^9 × 1,000 = 14 TFLOPs

The theoretical time is:

14 TFLOPs / 312 TFLOPS ≈ 45 ms

Real-world utilization factors push this to roughly 90–150 ms. With longer contexts, attention computation and KV cache reads add further overhead.

Optimization Directions

Several techniques target the bottlenecks of each phase:

Quantization: Reduces the byte size of weights and KV cache, directly easing the Decode bandwidth bottleneck.
PagedAttention: Manages KV cache in non-contiguous pages, reducing fragmentation and improving VRAM utilization.
FlashAttention: Accelerates Prefill and long-context attention by reducing memory reads and writes through tiling and kernel fusion.
Continuous Batching: Dynamically combines requests at the Decode stage to improve throughput without waiting for fixed batch boundaries.
Chunked Prefill: Breaks long prompts into chunks scheduled across iterations, lowering TTFT jitter and improving resource fairness.
Prefix Caching: Reuses KV cache for shared prefixes across requests, eliminating redundant Prefill computation.
Speculative Decoding: Uses a smaller draft model to propose multiple tokens, which the target model verifies in parallel, effectively reducing the number of Decode steps.

Summary

Prefill is a compute-intensive stage whose cost is determined by prompt length, parameter count, and GPU throughput. Decode is a memory-bandwidth-intensive stage whose cost is determined by model weight size, KV cache footprint, and VRAM bandwidth. Production inference systems must optimize both phases independently and balance them through batching, chunked scheduling, and cache reuse to achieve favorable TTFT, TPOT, and overall throughput.