1. The setup, in one paragraph

A flow-matching text-to-image model (Esser et al., 2024) is trained to imitate data, which makes its samples plausible but says nothing about whether they follow the prompt, count correctly, render legible text, or match human taste. Reinforcement learning closes that gap by optimizing a reward instead of a likelihood – the same move RLHF made for language. Every method here is a different way to solve

$${𝜃}^{\ast }=\underset{𝜃}{argmax}{𝔼}_{𝑐\sim \mathrm{𝒟︀},𝑥\sim {𝜋}_{𝜃}\left(\cdot |𝑐\right)}\left[𝑟\left(𝑥,𝑐\right)\right]-𝛽\cdot KL\left({𝜋}_{𝜃}\left(\cdot |𝑐\right)\parallel {𝜋}_{\text{ref}}\left(\cdot |𝑐\right)\right),$$

where $𝑟\left(𝑥,𝑐\right)$ is the reward – a learned human-preference model (ImageReward (Xu et al., 2023), HPSv2 (Wu et al., 2023), PickScore (Kirstain et al., 2023)) or a rule-based checker (GenEval (Ghosh et al., 2023) object detection, OCR for text) – and the $KL$ term is a leash keeping the tuned policy ${𝜋}_{𝜃}$ near the pretrained ${𝜋}_{\text{ref}}$. Two facts about $𝑟$ shape everything: whether it is differentiable, and that it is terminal (one scalar on the finished image, no per-step signal).

2. The families before the 2025 wave

Three older strategies set the stage; I recap them only enough to place the new work.

• Reward backprop. If $𝑟$ is differentiable, backpropagate its gradient straight through the sampler: ReFL (Xu et al., 2023), DRaFT (Clark et al., 2024), AlignProp (Prabhudesai et al., 2023). Sample-efficient, but needs a differentiable reward and over-optimizes fast.
• Policy gradient on the denoising MDP. Treat denoising as a trajectory and use PPO-style policy gradients with a KL penalty: DDPO (Black et al., 2023) and DPOK (Fan et al., 2023). Works with any black-box reward, but is sample-hungry.
• Preference optimization. Skip the reward model and the RL loop, optimizing directly on preference pairs: Diffusion-DPO (Wallace et al., 2024), and SPO (Liang et al., 2025) which adds per-step credit. Stable, but offline.

The 2025 wave is the arrival of GRPO (Shao et al., 2024) – the value-free, group-relative policy-gradient method behind the DeepSeek reasoning models – into image generation. I assume it here; if you want the mechanics (group sampling, the standardized advantage ${\widehat{𝐴}}_{𝑖}=\left({𝑟}_{𝑖}-\text{mean}\right)/\text{std}$, the clipped ratio and KL), see the earlier GRPO post. DanceGRPO (Xue et al., 2025) showed one GRPO recipe transfers across diffusion and flow backbones and across image and video. The interesting question is what breaks when you try it on a flow model – which is where the first paper comes in.

3. Flow-GRPO: making a deterministic sampler explorable

GRPO needs a stochastic policy: it forms an importance ratio ${𝜌}_{𝑖,𝑡}$ from the probability ${𝜋}_{𝜃}\left({𝑥}_{𝑡-1}|{𝑥}_{𝑡},𝑐\right)$ of each denoising step, and it needs randomness to explore. A flow model has neither. It generates by integrating a deterministic ODE, $d{𝑥}_{𝑡}={𝑣}_{𝜃}\left({𝑥}_{𝑡},𝑡\right)d𝑡$: fix the initial noise and the entire path – and the image – is determined. There is no per-step density and nothing to explore with.

Flow-GRPO (Liu et al., 2025) fixes this with two ideas.

Idea 1 – turn the ODE into an SDE. For the deterministic sampling ODE there is a whole family of stochastic differential equations that share its marginal distribution ${𝑝}_{𝑡}$ at every $𝑡$ but travel there noisily. Using the score-velocity identity for rectified flow, $\nabla log{𝑝}_{𝑡}\left(𝑥\right)=-𝑥/𝑡-\left(\left(1-𝑡\right)/𝑡\right){𝑣}_{𝜃}$, one Euler-Maruyama step becomes

$${𝑥}_{𝑡+\mathrm{\Delta }𝑡}={𝑥}_{𝑡}+\left[{𝑣}_{𝜃}\left({𝑥}_{𝑡},𝑡\right)+\frac{{𝜎}_{𝑡}^2}{2𝑡}\left({𝑥}_{𝑡}+\left(1-𝑡\right){𝑣}_{𝜃}\left({𝑥}_{𝑡},𝑡\right)\right)\right]\mathrm{\Delta }𝑡+{𝜎}_{𝑡}\sqrt{\mathrm{\Delta }𝑡}𝜀,𝜀\sim \mathrm{𝒩︀}\left(0,𝐼\right).$$

Now each step is a Gaussian: it has a closed-form density (so ${𝜌}_{𝑖,𝑡}$ and the $KL$ term exist in closed form) and the injected noise $𝜀$ finally gives the model something to explore. The noise scale ${𝜎}_{𝑡}=𝑎\sqrt{𝑡/\left(1-𝑡\right)}$ tunes exploration ($𝑎\approx 0.7$ works; too large collapses training). Because the marginals are preserved, inference is unchanged – the SDE is only for collecting rollouts.

Idea 2 – denoising reduction. Online RL is bottlenecked by rollout speed, and flow inference wants many steps ($𝑇=40$). Flow-GRPO collects training rollouts with far fewer ($𝑇=10$): the short, lower-quality trajectories still rank correctly, which is all the group-relative advantage needs, for a >4× speedup.

On SD3.5-Medium this lifts GenEval from 0.63 to 0.95 (past GPT-4o’s 0.84) and text-rendering OCR from 59% to 92%. Remove the $KL$ leash and quality and diversity collapse even as the proxy reward climbs – reward hacking in the flesh.

Flow-GRPO’s two costs each spawned follow-ups. Optimizing every one of the $𝑇$ steps is expensive, so MixGRPO (Li et al., 2025) runs the costly SDE-plus-GRPO update only inside a sliding window of timesteps (cheap ODE outside), roughly halving training time; and a single terminal reward is blunt, so TempFlow-GRPO (He et al., 2025) makes the reward timing-aware, giving denser per-step credit.

4. DiffusionNFT: do the RL on the forward process instead

Flow-GRPO works, but DiffusionNFT (Zheng et al., 2025) argues its whole setup fights the medium. Because GRPO optimizes the reverse (sampling) process, it is forced into a first-order SDE sampler (Idea 1 above), fragile log-probability estimation for ${𝜌}_{𝑖,𝑡}$, and an awkward two-model bolt-on for classifier-free guidance. All three stem from needing likelihoods of a sampling trajectory. So: drop the trajectory and move the RL to the forward (noising) process.

The reframing. Generate a batch of images with any solver, score them, and softly split them by reward into a positive and a negative set – formally an optimality probability $𝑟\in \left[0,1\right]$ per image (high reward → $𝑟$ near 1). Instead of a policy gradient, define an improvement direction (reinforcement guidance $\mathrm{\Delta }$) relating the improved velocity field to the current one:

$${𝑣}^{\ast }={𝑣}_{\text{old}}+\frac{1}{𝛽}\mathrm{\Delta },\mathrm{\Delta }\propto 𝛼\left({𝑣}^{+}-{𝑣}_{\text{old}}\right)=\left(1-𝛼\right)\left({𝑣}_{\text{old}}-{𝑣}^{-}\right),$$

where ${𝑣}^{+}$ and ${𝑣}^{-}$ are the velocity fields of the positive- and negative-reward data: move along the good, away from the bad.

The trick that makes it practical. Rather than train two networks for ${𝑣}^{+}$ and ${𝑣}^{-}$, parameterize both implicitly with a single network ${𝑣}_{𝜃}$,

$${𝑣}_{𝜃}^{+}=\left(1-𝛽\right){𝑣}_{\text{old}}+𝛽{𝑣}_{𝜃},{𝑣}_{𝜃}^{-}=\left(1+𝛽\right){𝑣}_{\text{old}}-𝛽{𝑣}_{𝜃},$$

and train ${𝑣}_{𝜃}$ with an ordinary reward-weighted flow-matching regression on the forward process (target velocity $𝑣={𝑥}_1-{𝑥}_0$, exactly as in pretraining):

$$\mathrm{ℒ︀}\left(𝜃\right)={𝔼}_{𝑡,{𝑥}_0,{𝑥}_1}\left[𝑟\cdot {\Vert {𝑣}_{𝜃}^{+}\left({𝑥}_{𝑡},𝑡\right)-𝑣\Vert }^2+\left(1-𝑟\right)\cdot {\Vert {𝑣}_{𝜃}^{-}\left({𝑥}_{𝑡},𝑡\right)-𝑣\Vert }^2\right].$$

Read plainly: fit the good images with the implicit positive policy and the bad ones with the implicit negative policy. One can show the minimizer satisfies ${𝑣}_{𝜃}^{\ast }={𝑣}_{\text{old}}+\left(2/𝛽\right)\mathrm{\Delta }$ – minimizing this supervised loss performs the RL improvement step. Because the loss lives on the forward process, it needs only clean images and rewards: no stored trajectories, no log-probs, any black-box solver, and no classifier-free-guidance machinery. It is, literally, supervised learning that happens to do RL.

On SD3.5-Medium it is 3×–25× more compute-efficient than Flow-GRPO, reaching GenEval 0.24 → 0.98 in about 1k steps where Flow-GRPO needs 5k-plus. The one hard dependency is the negative branch: drop the $\left(1-𝑟\right)$ term and the reward collapses almost immediately – the contrast between good and bad samples is what carries the signal.

5. RL beyond diffusion, and into products

The same logic reaches models that emit an image as a token sequence, where standard language-model GRPO applies almost directly. T2I-R1 (Jiang et al., 2025) inserts an explicit chain-of-thought reasoning stage before decoding; X-Omni (Geng et al., 2025) shows GRPO can revive discrete-token autoregressive generation, restoring the strong text rendering that had lagged diffusion; and UniRL (Mao et al., 2025) lets a unified understanding-and-generation model self-improve on its own samples.

And this is now shipping. The Qwen-Image-2.0-RL report (Xu et al., 2026) post-trains the production Qwen-Image (Wu et al., 2025) model with a GRPO-based framework plus RLHF and an on-policy distillation stage that folds the RL policy back into a fast student. Its rewards are composite vision-language scorers (alignment, aesthetics, portrait fidelity; instruction-following and identity preservation for editing), and much of the engineering is anti-reward-hacking: a hybrid classifier-free-guidance strategy, dropping prompt groups whose within-group reward spread is too small to yield useful advantages, and per-category reward-weight calibration. The gains are reported in production terms: Text-to-Image Arena Elo +78 and Image-Edit Arena Elo +93.

6. The recurring enemy, and the arc

Every method here is stalked by the same failure: $𝑟$ is only a proxy, and pushed hard enough the model games it – the score climbs while true quality and diversity collapse into one over-saturated style. This over-optimization has been dissected directly (Zhang et al., 2024), and the defenses recur throughout: the $𝛽KL$ leash (DPOK, Flow-GRPO), reward ensembling and re-weighting, prompt filtering (Qwen), and stronger reward models. No one has solved it; they spend a training budget staying ahead of it.

Method	Reward must be	RL acts on	Central object
Reward backprop (Clark et al., 2024)	differentiable	the full sampler	$𝜕𝑟/𝜕𝜃$
DDPO / DPOK (Black et al., 2023)	black-box	reverse-process MDP	policy grad $\nabla log𝜋\cdot 𝑟$
Diffusion-DPO (Wallace et al., 2024)	preference pairs	offline pairs	logistic likelihood ratio
Flow-GRPO (Liu et al., 2025)	black-box	reverse process (SDE)	group advantage ${\widehat{𝐴}}_{𝑖}$ + ratio $𝜌$
DiffusionNFT (Zheng et al., 2025)	black-box	the forward process	reward-weighted regression

The through-line is asking less and less of the reward while moving where the RL signal is applied: from a differentiable reward, to any black box on the reverse-process MDP, to a value-free group estimator, to DiffusionNFT’s supervised forward-process regression. Each step buys stability and speed. What has not moved is the bottleneck – the reward is still a proxy, and the next real gains will come less from a cleverer optimizer than from rewards that capture what we actually mean.

References

1. Prabhudesai, M., Goyal, A., Pathak, D., and Fragkiadaki, K. Aligning Text-to-Image Diffusion Models with Reward Backpropagation, 2023.
2. Clark, K., Vicol, P., Swersky, K., and Fleet, D. J. Directly Fine-Tuning Diffusion Models on Differentiable Rewards. International Conference on Learning Representations (ICLR), 2024.
3. Liu, J., Liu, G., Liang, J., Li, Y., Liu, J., Wang, X., Wan, P., Zhang, D., and Ouyang, W. Flow-GRPO: Training Flow Matching Models via Online RL, 2025.
4. Jiang, D., Guo, Z., Zhang, R., Zong, Z., Li, H., Zhuo, L., Yan, S., Heng, P.-A., and Li, H. T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT, 2025.
5. Zhang, Z., Zhang, S., Zhan, Y., Luo, Y., Wen, Y., and Tao, D. Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases. International Conference on Machine Learning (ICML), 2024.
6. He, X., Fu, S., Zhao, Y., Li, W., Yang, J., Yin, D., Rao, F., and Zhang, B. TempFlow-GRPO: When Timing Matters for GRPO in Flow Models, 2025.
7. Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., and Rombach, R. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. International Conference on Machine Learning (ICML), 2024.
8. Ghosh, D., Hajishirzi, H., and Schmidt, L. GenEval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems (Neurips), 2023.
9. Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. Advances in Neural Information Processing Systems (Neurips), 2023.
10. Xu, Y., Gao, K., Chen, Y., Chen, Y., Tang, Z., Liu, Z., Zhou, Z., Li, D., Meng, H., Cao, K., Li, J., Zhang, J., Peng, L., Jiang, L., Tang, N., Yin, S., Wu, T., Chen, X., Shu, Y., … Wu, C. Qwen-Image-2.0-RL Technical Report, 2026.
11. Liang, Z., Yuan, Y., Gu, S., Chen, B., Hang, T., Cheng, M., Li, J., and Zheng, L. Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
12. Li, J., Cui, Y., Huang, T., Kong, W., Cheng, Y., Zeng, C., Ma, Y., Fan, C., Yang, M., Zhong, Z., and Bo, L. MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE, 2025.
13. Wu, C., Li, J., Zhou, J., Lin, J., Gao, K., Yan, K., Yin, S.-ming, Bai, S., Xu, X., Chen, Y., Chen, Y., Tang, Z., Zhang, Z., Wang, Z., Yang, A., Yu, B., Cheng, C., Liu, D., Li, D., … Liu, Z. Qwen-Image Technical Report, 2025.
14. Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffusion Model Alignment Using Direct Preference Optimization. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
15. Xue, Z., Wu, J., Gao, Y., Kong, F., Zhu, L., Chen, M., Liu, Z., Liu, W., Guo, Q., Huang, W., and Luo, P. DanceGRPO: Unleashing GRPO on Visual Generation, 2025.
16. Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models. Advances in Neural Information Processing Systems (Neurips), 2023.
17. Zheng, K., Chen, H., Ye, H., Wang, H., Zhang, Q., Jiang, K., Su, H., Ermon, S., Zhu, J., and Liu, M.-Y. DiffusionNFT: Online Diffusion Reinforcement with Forward Process, 2025.
18. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024.
19. Geng, Z., Wang, Y., Ma, Y., Li, C., Rao, Y., Gu, S., Zhong, Z., Lu, Q., Hu, H., Zhang, X., Linus, Wang, D., and Jiang, J. X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again, 2025.
20. Mao, W., Yang, Z., and Shou, M. Z. UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning, 2025.
21. Black, K., Janner, M., Du, Y., Kostrikov, I., and Levine, S. Training Diffusion Models with Reinforcement Learning, 2023.
22. Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis, 2023.
23. Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. Advances in Neural Information Processing Systems (Neurips), 2023.