ALiBi

Transformer self-attention is often described as permutation invariant. The same set of token representations can produce the same attention patterns regardless of their order. That property is helpful in some settings, but language depends heavily on sequence order. Without some notion of position, attention may struggle to capture meaning that comes from word order.

Many models therefore inject positional information. Some use sinusoidal position encodings. Others use learned positional embeddings. These designs often fix a maximum sequence length during training. When inference inputs become longer than that limit, performance can fall apart.

ALiBi's Simple Bias

In 2022, Press and colleagues introduced ALiBi in the ICLR paper "Train Short, Test Long." The method was designed to address both position modeling and long-sequence generalization.

Instead of adding position vectors to token embeddings, ALiBi changes the attention score directly. For each attention head, a linear penalty is applied based on the distance between tokens. The score for positions i and j becomes something like:

A_ij = q_i · k_j - m_h · |i - j|

The value m_h is the slope for head h. A larger distance creates a larger penalty. For example, a slope of 1/2 gives a -1.5 penalty at distance 3.

This bias is static and does not require learning. It also depends only on relative distance, which makes it easy to apply to new sequence lengths.

Slopes and Attention Heads

ALiBi typically uses several attention heads, each with its own slope. The slopes often form a geometric sequence. For eight heads, the slopes may be:

• 1/2
• 1/4
• 1/8
• 1/16
• 1/32
• 1/64
• 1/128
• 1/256

Heads with smaller slopes tend to decay more slowly, so they can capture longer-range dependencies. Heads with larger slopes focus more on nearby tokens.

When the number of heads is not a power of two, the official implementation may generate slopes near the closest power of two and interpolate between adjacent values.

Long Inputs and Practical Use

The main appeal of ALiBi is its ability to handle longer inputs. A model trained on shorter sequences can often process longer ones without retraining. The bias is recalculated for the new sequence length, so no extra position vectors are needed.

This makes long-context tasks more feasible. It also reduces the need for special position interpolation or complex extrapolation methods.

Comparison With Other Position Methods

• Sinusoidal or learned position embeddings: Add absolute position information to embeddings. They may not extrapolate well beyond the training length.
• RoPE: Applies rotations to queries and keys. It has relative position behavior, but direct extrapolation can still cause problems.
• ALiBi: Adds a linear attention bias. It is simple and can support longer sequences.

Compared with RoPE, ALiBi avoids rotation parameters and keeps the positional signal separate from the content representation. Compared with fixed sinusoidal tables, it does not require a predefined maximum length. These differences make