Large language models have increasingly relied on reinforcement learning to improve reasoning, instruction following, and alignment. Among the available algorithms, Proximal Policy Optimization (PPO) has long served as a standard approach, but its dependence on a separate value model creates substantial memory and computation costs. Group Relative Policy Optimization (GRPO) offers a practical alternative by replacing learned value estimates with comparisons among multiple responses generated from the same prompt.
In a conventional PPO setup, two large models are usually required:

• A policy model that generates responses and receives parameter updates.
• A critic model that predicts the expected return of each generated state or token.

For large language models, the critic may be as large as the policy model—or even larger. It also needs its own parameters, optimizer states, activations, and training updates. As models and batch sizes scale, this overhead can dominate the training process.

GRPO removes the critic entirely. Instead, it samples a group of candidate responses for each prompt and uses their rewards as a shared baseline. A response that performs better than the rest of its group receives a positive advantage, while a weaker response receives a negative one. The resulting signal is relative rather than absolute, allowing the policy to learn without a learned value network.

How GRPO Works

Suppose a prompt (q) is sampled and the current policy generates (G) responses:

[
{o_1, o_2, \ldots, o_G}.
]

1. Generate a group of candidates

Each response is sampled from the same rollout policy, typically with different decoding randomness. A larger group creates more opportunities to distinguish high-quality reasoning paths from weak ones, although it also increases generation cost.

2. Assign rewards

Every candidate receives a scalar reward (r_i). The reward may come from:

• A learned reward model
• Rule-based verification
• Exact-answer matching
• Unit-test execution for code
• A combination of automated checks and learned evaluators

For mathematical reasoning and code generation, verifiable rewards are especially useful because correctness can often be evaluated directly and consistently.

3. Normalize rewards within each group

GRPO converts raw rewards into group-relative advantages:

[
A_i =
\frac{r_i-\mu}{\sigma+\varepsilon},
]

where

[
\mu = \frac{1}{G}\sum_{j=1}^{G}r_j
]

and (\sigma) is the standard deviation of the group’s rewards. The small term (\varepsilon) prevents division by zero.

This normalization reduces sensitivity to differences in prompt difficulty. For example, a difficult prompt may produce low absolute rewards for every candidate, while an easy prompt may produce consistently high rewards. Within each group, however, the relative ordering still provides a useful learning signal.

4. Optimize the policy

GRPO retains the clipped policy-update mechanism used by PPO. If (\pi_\theta) is the updated policy and (\pi_{\text{old}}) is the policy that generated the completions, the probability ratio is:

[
\rho_i(\theta)=
\frac{\pi_\theta(o_i\mid q)}
{\pi_{\text{old}}(o_i\mid q)}.
]

The core objective is commonly written as:

[
\mathcal{L}_{\text{policy}}

-\frac{1}{G}\sum_{i=1}^{G}
\min\left(
\rho_i A_i,
\operatorname{clip}(\rho_i,1-\epsilon,1+\epsilon)A_i
\right).
]

A KL-divergence penalty is often added to keep the updated policy close to a reference model:

[
\mathcal{L}

\mathcal{L}{\text{policy}}
+
\beta D{\mathrm{KL}}(\pi_\theta,\pi_{\text{ref}}).
]

This reference model acts as a stability anchor, limiting abrupt changes to the model’s overall behavior.

A Simplified Implementation

The following outline illustrates the main training calculations:

import torch
import torch.nn.functional as F


def grpo_loss(
    new_log_probs,
    old_log_probs,
    reference_log_probs,
    rewards,
    group_size,
    clip_eps=0.2,
    beta=0.01
):
    grouped_rewards = rewards.view(-1, group_size)

    mean = grouped_rewards.mean(dim=-1, keepdim=True)
    std = grouped_rewards.std(dim=-1, keepdim=True, unbiased=False)

    advantages = (grouped_rewards - mean) / (std + 1e-8)
    advantages = advantages.view(-1)

    ratios = torch.exp(new_log_probs - old_log_probs)

    unclipped = ratios * advantages
    clipped = torch.clamp(
        ratios, 1.0 - clip_eps, 1.0 + clip_eps
    ) * advantages

    policy_loss = -torch.min(unclipped, clipped).mean()

    # Sample-based estimate of KL(reference policy || current policy)
    kl_loss = (
        reference_log_probs.exp()
        * (reference_log_probs - new_log_probs)
    ).sum(dim=-1).mean()

    return policy_loss + beta * kl_loss

In a production system, the rollout policy should be frozen while collecting completions and old token log probabilities. The current and reference models can then be evaluated in separate forward passes, often with memory-saving techniques such as sharding or mixed precision.

Engineering Advantages

Lower memory pressure

Removing the critic eliminates the memory required for its parameters, gradients, and optimizer states. This is one of GRPO’s most important practical benefits, particularly when training on a fixed number of accelerators.

The savings are not unlimited, however. A reference model and reward model may still be required, depending on the training setup.

Greater hardware efficiency

GRPO avoids critic forward and backward passes. More memory and computation can therefore be redirected toward policy rollouts, larger effective batches, or multiple candidate generations per prompt.

Simpler infrastructure

A critic-free design reduces synchronization, model-loading, and optimizer-management complexity. It is especially attractive for teams that want to apply reinforcement learning to reasoning models without maintaining two full-scale value networks.

Effective use of verifiable rewards

Group comparison works naturally with tasks where correctness can be checked automatically. If some candidates solve a problem and others do not, normalization turns those outcomes into clear positive and negative learning signals.

Important Limitations

GRPO is not a universal replacement for PPO. Its quality depends heavily on group composition and reward design.

If every candidate in a group receives the same reward, the normalized advantages become effectively zero, providing little useful information. Group diversity is therefore essential. Diverse prompts, sampling temperatures, and candidate-generation strategies can all help.

Relative rewards also do not guarantee absolute improvement. A policy may become better at outperforming its current group without becoming uniformly correct. Combining GRPO with strong verifiers, curriculum design, and careful reward filtering helps mitigate this issue.

Reward quality remains another major concern. An imperfect verifier can reward superficial formatting, exploit test cases, or favor fluent but incorrect reasoning. Diverse evaluation methods and periodic human or model-based audits can reduce these risks.

Finally, larger groups are not free. Increasing (G) improves the resolution of relative comparisons but also raises generation and verification costs. Practical systems must balance group size, sequence length, and available accelerator memory.

Practical Design Choices

Several implementation decisions strongly affect GRPO training stability:

• Use a fixed group size during an update so that normalization remains consistent.
• Apply the KL coefficient (\beta) cautiously; too little regularization may destabilize training, while too much may suppress useful learning.
• Monitor reward variance, clipping rates, KL divergence, and response length rather than relying only on average reward.
• Mask padding tokens when computing token-level losses and log probabilities.
• Keep rollout generation and policy optimization reasonably close in time to reduce the gap between the old and updated policies.
• Validate reward functions against known failure cases before launching long training runs.

Conclusion

GRPO replaces PPO’s learned critic with a simple and powerful idea: evaluate a response by comparing it with other responses generated for the same prompt. This group-relative mechanism lowers memory demands, simplifies training infrastructure, and works particularly well for tasks with automatically verifiable rewards.

Its success, however, depends on meaningful candidate diversity, reliable rewards, and careful regularization. When those conditions are met, GRPO provides an efficient path for improving the reasoning capabilities of large language models while avoiding much of the overhead associated with critic-based reinforcement learning.