Decoder

At its core, a decoder is an essential information processing component that translates encoded inputs back into a human

At its core, a decoder is an essential information processing component that translates encoded inputs back into a human-readable or target representation. While the concept spans multiple engineering disciplines—such as digital logic circuits mapping binary inputs to specific output lines, or multimedia codecs decompressing raw audio and video streams—its role in modern artificial intelligence has arguably become the most transformative.

In the realm of deep learning, the decoder is the engine responsible for generating output sequences, serving as the architectural backbone for state-of-the-art large language models like GPT and LLaMA.
Originating from the landmark paper Attention Is All You Need, the standard Transformer decoder is built by stacking $N=6$ identical layers. Each layer features three specialized sub-layers, wrapped with residual connections and layer normalization:

  • Masked Multi-Head Self-Attention
  • Encoder-Decoder Cross-Attention
  • Position-wise Feed-Forward Network

Masked Self-Attention

Decoders operate in an autoregressive fashion. When generating the $t$-th token in a sequence, the model must only rely on the preceding $t-1$ tokens, preventing it from "peeking into the future." To enforce this rule, a causal mask is applied during attention calculation. This mask sets future positions to negative infinity, which zeroes out their weights after the softmax operation and ensures training and inference consistency.

Cross-Attention

Acting as a vital bridge between the encoder and the decoder, cross-attention allows the decoder to pull context from the input. The queries come from the previous decoder layer, while the keys and values originate from the encoder's final output. This mechanism empowers the decoder to reference the entire input sequence dynamically, which is crucial for tasks like machine translation and text summarization.

Feed-Forward Networks

Composed of two linear transformations with a non-linear activation (such as ReLU or GELU) in between, the feed-forward network typically expands the feature dimension (e.g., from 512 to 2048) before projecting it back down. It processes each sequence position independently, refining the representations extracted by the attention mechanisms.

Three Main Architectural Paradigms

Depending on how the encoder and decoder are combined, modern transformer models generally fall into three categories:

  • Encoder-Decoder (e.g., T5, BART): Retains both components and full cross-attention, making it ideal for tasks where input and output lengths vary significantly.
  • Decoder-Only (e.g., GPT series, LLaMA): Strips away the encoder and cross-attention, relying entirely on masked self-attention and next-token prediction. Its structural simplicity and high parallelizability make it the dominant choice for modern large language models.
  • Encoder-Only (e.g., BERT): Features no decoder at all, excelling in text understanding and classification tasks.

Practical Implementation in PyTorch

Frameworks like PyTorch offer native modules to construct and test decoder stacks quickly:

import torch
import torch.nn as nn

decoder_layer = nn.TransformerDecoderLayer(
    d_model=512, nhead=8, dim_feedforward=2048, batch_first=True
)
decoder = nn.TransformerDecoder(decoder_layer, num_layers=6)

memory = torch.rand(2, 10, 512)   # Encoder output
tgt = torch.rand(2, 8, 512)       # Target sequence representation
# Generate causal mask to hide future positions
mask = nn.Transformer.generate_square_subsequent_mask(8)

out = decoder(tgt, memory, tgt_mask=mask)
print(out.shape)  # torch.Size([2, 8, 512])

The Autoregressive Generation Loop

During inference, text is generated sequentially through a continuous loop:

  1. The process begins with a special start-of-sequence token (<BOS>).
  2. A forward pass computes a probability distribution across the vocabulary.
  3. The next token is selected using strategies like argmax, top-k, or top-p sampling.
  4. The newly generated token is appended to the input sequence, and the loop repeats until an end-of-sequence token (<EOS>) is produced.

Because recalculating attention for every token gets computationally expensive as sequences grow, production environments heavily rely on KV Caching. By storing the keys and values from previous steps, the model only computes attention for the newest token, drastically reducing redundant computations.

Conclusion

The decoder acts as a powerful bridge transforming compressed representations into coherent target outputs. Whether routing signals in hardware, restoring compressed media, or generating human-like text through masked attention and autoregressive loops, understanding the decoder's mechanics remains an indispensable milestone for mastering modern deep learning.