Paper Notes

Is OpenAI's Mysterious RL Algorithm Just an STE Approximation?

Zeshi Yang · September 20, 2026

Discussing Score Centering Stabilizes Off-policy Reinforcement Learning, Martin Marek and Max Ryabinin, arXiv:2609.20807.

A recent arXiv paper, Score Centering Stabilizes Off-policy Reinforcement Learning, has attracted attention as a possible account of the reinforcement-learning algorithm used in OpenAI's post-training pipeline. Curious about the method, I took a closer look. My main takeaway is that score centering can be understood as an approximate implementation of a straight-through estimator (STE), a trick that has long been used in sim-to-real reinforcement learning. Nothing is entirely new under the sun. In this post, I will walk through this interpretation in detail.

The Problem Setup for LLM Reinforcement Learning#

One complication in LLM reinforcement learning is that training and inference often use different engines: Megatron for training and vLLM for inference. Even given the same input, the two engines can produce slightly different logits in the forward pass. This train-inference mismatch is a key distinction between LLM RL and conventional RL with smaller models.

Our actual goal is to maximize the performance of the model deployed with vLLM, not the one trained with Megatron.

To formalize the problem, consider it from the perspective of information-geometric optimization (IGO). We use vLLM to generate rollout trajectories, evaluate them with rewards, and obtain an optimal target distribution. The goal is then to optimize the policy by minimizing its KL divergence from this target distribution.

The difficulty is that we cannot backpropagate through the model running in vLLM. We therefore use gradients computed in Megatron as a proxy for the unavailable vLLM gradients. In practice, this changes the RL objective from

$$ \mathbb{E}_{x\sim P_{\mathrm{vLLM}}^{\mathrm{old}}} \left[ \frac{P_{\mathrm{vLLM}}(x)}{P_{\mathrm{vLLM}}^{\mathrm{old}}(x)}R(x) \right] $$

$$ \mathbb{E}_{x\sim P_{\mathrm{vLLM}}^{\mathrm{old}}} \left[ \frac{P_{\mathrm{Megatron}}(x)}{P_{\mathrm{vLLM}}^{\mathrm{old}}(x)}R(x) \right]. $$

Keep this expression in mind, as it captures the central issue in LLM RL.

Analyzing the Objective#

The key observation is that the probability ratio in the second objective is not necessarily 1, even at the beginning of optimization. It therefore cannot be understood in the same way as the ratio used in PPO clipping.

In PPO, the ratio starts at 1 and changes only as the policy is updated. Clipping this ratio keeps the new policy close to the old one and helps stabilize optimization. With a train-inference mismatch between vLLM and Megatron, however, the ratio can already differ from 1 before any update occurs. Clipping does not address this mismatch and may even be harmful. I believe this helps explain why approaches such as MIS and bypass-PPO often fail.

A simple and natural alternative is to use a straight-through estimator. Let $Z_{\mathrm{Megatron}}(x)$ denote the logits computed by Megatron, and let $Z_{\mathrm{vLLM}}^{\mathrm{old}}(x)$ denote those computed by vLLM under the old policy. We can define surrogate logits as

$$ Z_{\mathrm{Megatron}}(x) +\operatorname{sg}\!\left[ Z_{\mathrm{vLLM}}^{\mathrm{old}}(x) -Z_{\mathrm{Megatron}}^{\mathrm{old}}(x) \right], $$

where $\operatorname{sg}[\cdot]$ denotes the stop-gradient operator. The corresponding surrogate distribution is

$$ \hat P_{\mathrm{Megatron}} =\operatorname{Softmax}\!\left( Z_{\mathrm{Megatron}}(x) +\operatorname{sg}\!\left[ Z_{\mathrm{vLLM}}^{\mathrm{old}}(x) -Z_{\mathrm{Megatron}}^{\mathrm{old}}(x) \right] \right). $$

The RL objective can then be written as

$$ \mathbb{E}_{x\sim P_{\mathrm{vLLM}}^{\mathrm{old}}} \left[ \frac{\hat P_{\mathrm{Megatron}}(x)} {P_{\mathrm{vLLM}}^{\mathrm{old}}(x)}R(x) \right]. $$

At the old parameters, the surrogate distribution matches the vLLM distribution in the forward pass, making the ratio equal to 1. The complete gradient derivation is

$$ \begin{aligned} &\left. \nabla_\theta \mathbb E_{x\sim P_{\mathrm{vLLM}}^{\mathrm{old}}} \left[ \frac{\widehat P_{\mathrm{Megatron}}(x;\theta)} {P_{\mathrm{vLLM}}^{\mathrm{old}}(x)} R(x) \right] \right|_{\theta=\theta_{\mathrm{old}}} \\[12pt] &= \mathbb E_{x\sim P_{\mathrm{vLLM}}^{\mathrm{old}}} \left[ \frac{R(x)}{P_{\mathrm{vLLM}}^{\mathrm{old}}(x)} \left. \nabla_\theta\widehat P_{\mathrm{Megatron}}(x;\theta) \right|_{\theta=\theta_{\mathrm{old}}} \right] \\[12pt] &= \mathbb E_{x\sim P_{\mathrm{vLLM}}^{\mathrm{old}}} \left[ \underbrace{ \frac{ \widehat P_{\mathrm{Megatron}}(x;\theta_{\mathrm{old}}) }{ P_{\mathrm{vLLM}}^{\mathrm{old}}(x) } }_{=\,1} R(x) \left. \nabla_\theta \log\widehat P_{\mathrm{Megatron}}(x;\theta) \right|_{\theta=\theta_{\mathrm{old}}} \right] \\[12pt] &= \mathbb E_{x\sim P_{\mathrm{vLLM}}^{\mathrm{old}}} \left[ R(x) \left. \left( \nabla_\theta Z_{\mathrm{Megatron}}(x;\theta) - \sum_v \widehat P_{\mathrm{Megatron}}(v;\theta) \nabla_\theta Z_{\mathrm{Megatron}}(v;\theta) \right) \right|_{\theta=\theta_{\mathrm{old}}} \right] \\[12pt] &= \mathbb E_{x\sim P_{\mathrm{vLLM}}^{\mathrm{old}}} \left[ R(x) \left. \left( \nabla_\theta Z_{\mathrm{Megatron}}(x;\theta) - \sum_v P_{\mathrm{vLLM}}^{\mathrm{old}}(v) \nabla_\theta Z_{\mathrm{Megatron}}(v;\theta) \right) \right|_{\theta=\theta_{\mathrm{old}}} \right] \\[12pt] &= \mathbb E_{x\sim P_{\mathrm{vLLM}}^{\mathrm{old}}} \left[ R(x) \left. \left( \nabla_\theta\log P_{\mathrm{Megatron}}(x;\theta) - \sum_v P_{\mathrm{vLLM}}^{\mathrm{old}}(v) \nabla_\theta\log P_{\mathrm{Megatron}}(v;\theta) \right) \right|_{\theta=\theta_{\mathrm{old}}} \right] \qquad \text{(SC)}. \end{aligned} $$

Key result

This is exactly the same gradient as the one used by the algorithm attributed to OpenAI. In other words, the method can be interpreted as an STE approximation that compensates for the inference mismatch between Megatron and vLLM.

What Changes in the Off-Policy Setting?#

So far, the derivation has assumed an on-policy setting, where each batch of rollout data is used for a single optimizer step. Once we reuse rollout data to improve sample efficiency, the training becomes off-policy, and the two formulations are no longer equivalent.

The gradient of the surrogate-STE version becomes

$$ \begin{aligned} \mathbb{E}_{x\sim P_{\mathrm{vLLM}}^{\mathrm{old}}} \Bigg[ \frac{\hat P_{\mathrm{Megatron}}(x)} {P_{\mathrm{vLLM}}^{\mathrm{old}}(x)}R(x) \Bigg( &\nabla_\theta\log P_{\mathrm{Megatron}}(x;\theta)\\ &-\sum_v \hat P_{\mathrm{Megatron}}(v) \nabla_\theta\log P_{\mathrm{Megatron}}(v;\theta) \Bigg) \Bigg]. \end{aligned} $$

In this case, the ratio is no longer 1. The surrogate-STE estimator effectively treats the samples as if they were drawn from $\hat P_{\mathrm{Megatron}}$, whereas score centering continues to use samples drawn from $P_{\mathrm{vLLM}}^{\mathrm{old}}$. I therefore view the latter estimator as biased because it omits importance sampling. This leads me to expect the surrogate-STE formulation to be more stable in the off-policy setting.

The paper's authors also evaluate score centering combined with truncated importance sampling (TIS). In my view, that formulation is already quite close to the surrogate-STE variant.

Further Thoughts#

Looking back, many RL algorithms for LLMs have not been analyzed systematically. They often appear instead as heuristic modifications of earlier methods such as PPO. Clipping makes sense in the original PPO setting because it implicitly constrains how far the updated policy can move. But when the problem is specifically the train-inference mismatch between vLLM and Megatron, I believe clipping is the wrong tool. It tends to invite additional patches, producing a system that is difficult to reason about and tune.

Starting from the IGO perspective leads to a more direct derivation and a clearer intuition for what is happening. In a sense, this is an exercise in subtraction: removing unnecessary machinery makes the core problem easier to see.

This viewpoint also suggests room for improvement. An STE provides a surrogate, or “fake,” gradient, which is not guaranteed to be a valid subgradient. A natural next step would be to use the actual inference behavior of vLLM to correct that surrogate gradient.

IGO is a powerful mathematical tool. Viewed through this lens, the underlying geometry of many RL algorithms becomes much easier to understand.