Blog

Notes on reinforcement learning, humanoid agents and character animation — mostly derivations I worked through for my own understanding, written up in case they are useful to someone else.

October 27, 2026

Training–Inference Mismatch in Generative RL

Why can reward fine-tuning under classifier-free guidance improve and then give back its gains? The weighted flow-matching gradient hides a term that fits the model to its own guided rollouts. Conditional centering plus a pre-update anchor removes it, a fixed-reference STE gives the same objective, and the identical structure reappears as score centering for LLMs. With Yujie Wang · English / 中文.

Reinforcement Learning Diffusion LLM

September 20, 2026

Is OpenAI's Mysterious RL Algorithm Just an STE Approximation?

Score centering, proposed in a recent arXiv paper as a possible account of OpenAI's post-training algorithm, turns out to be an approximate straight-through estimator that compensates for the Megatron/vLLM train-inference mismatch. A derivation from the information-geometric optimization perspective, and why clipping is the wrong tool here.

Reinforcement Learning LLM Paper Notes

I also write in Chinese on Zhihu.