1. REPresentation Alignment (REPA)

The main challenge in training diffusion models stems from the need to learn a high-quality internal representation h. We demonstrate that the training process for generative diffusion models becomes significantly easier and more effective when supported by an external representation, ${𝑦}^{\ast }$. Specifically, we propose a simple regularization technique that leverages recent advances in self-supervised visual representations as ${𝑦}^{\ast }$, leading to substantial improvements in both training efficiency and the generation quality of diffusion transformers.

These insights inspire us to enhance generative models by incorporating external self-supervised representations. However, the problems are

1. Input mismatch: diffusion models work with noisy inputs while most self-supervised learning encoders are trained on clean images
2. These off-the-shelf vision encoders are not designed for tasks like reconstruction or generation

To overcome these technical hurdles, we guide the feature learning of diffusion models using a regularization technique called REPresentation Alignment (REPA) that distills pretrained self-supervised representations into diffusion representations, offering a flexible way to integrate high-quality representations.

Let $𝑓$ be a pretrained encoder, ${𝑥}_{\ast }$ be a clean image. REPA aligns ${ℎ}_{𝜑}\left({ℎ}_{𝑡}\right)$ with ${𝑦}_{\ast }$ where ${ℎ}_{𝜑}\left({ℎ}_{𝑡}\right)$ is a projection of an transformer encoder output ${ℎ}_{𝑡}={𝑓}_{𝜃}\left({𝑧}_{𝑡}\right)$ that through a trainable projection head ${ℎ}_{𝜑}$.

$${ℒ}_{\text{REPA}}\left(𝜃,𝜑\right)=-{𝐸}_{{𝑥}_{\ast },𝜀,𝑡}\left[\frac{1}{𝑁}\sum _{𝑛=1}^{𝑁}\text{ sim}\left({𝑦}_{\ast }^{\left[𝑛\right]},{ℎ}_{𝜑}\left({ℎ}_{𝑡}^{\left[𝑛\right]}\right)\right)\right]$$

2. Self-Representation Alignment (SRA)

SRA attains self-alignment by minimizing the patch-wise distance between the teacher's output (${𝑦}_{\ast }$) and the student's output variant (${𝑗}_{𝜓}\left(𝑦\right)$):

$${ℒ}_{\text{sa}}\left(𝜁,𝜓\right)={\mathrm{𝔼}}_{{𝑥}_{𝑡},𝑡,𝑐}\left[\frac{1}{𝑁}\sum _{𝑛=1}^{𝑁}\text{ dist}\left({𝑦}_{\ast }^{\left[𝑖\right]},{𝑗}_{𝜓}\left({𝑦}^{\left[𝑖\right]}\right)\right)\right]$$

where $\left[𝑛\right]$ is a patch index, $\text{dist}\left(·,·\right)$ is a pre-defined distance calculation function, and $𝜁$, $𝜑$ is the parameters of student diffusion transformer and the projection head. This objective is similar to that mentioned in REPA, except that we're aiming to align with DIT's own teacher instead of external transformer itself. The final loss is:

$$ℒ={ℒ}_{\text{ gen }}+𝜆{ℒ}_{\text{ sa}}$$

3. LayerSync

4. Velocity Refiner with Acceleration (VeRA)

Existing flow-based generative models, such as SiT, primarily utilize only the final layer's output for velocity prediction, under-utilizing rich intermediate representations. This under-utilization of internal features leads to slower model convergence and suboptimal generation performance in flow-based models.

4.1. Deep Supervision

DeepFlow employs deep supervision by inserting auxiliary velocity layers after selected intermediate transformer blocks. The corresponding deep supervision loss at these key transformer layers is defined as follows:

$${ℒ}_{\text{deep }}=𝐸\left[\sum _{𝑖=1}^{𝑘}{𝛽}^{𝑖}\left({\Vert {𝑣}_{𝜃}^{𝑖}\left({𝑥}_{𝑡}^{𝑖},𝑡,𝑐\right)-𝑉\Vert }^2\right)\right]$$

This approach encourages intermediate layers to develop meaningful velocity representations earlier in the network, improving gradient flow and feature alignment across layers.

4.2. Velocity Refiner with Acceleration (VeRA) Block

Positioned between adjacent branches, it refines velocity features from preceding layers and aligns them with subsequent processing stages. The block comprises three main components:

1. Acceleration Generation and Second-Order ODE Training

The VeRA block begins by generating an "acceleration feature" from the input velocity feature using a Multi-Layer Perceptron (ACC_MLP):

$${𝑎}_{{𝑡}_1}^{\ast }=\text{ACC\_MLP}\left({𝑣}_{{𝑡}_1}^{\ast }\right)$$

Then we can endow ${𝑎}_{𝑡}^{\ast }$ with acceleration property using a second-order ordinary differential equation ($2^{\text{nd}}$-ODE) as following equation:

$$\begin{array}{c}{ℒ}_{\text{acc }}=𝐸\left[\Vert 2^{\text{nd}}\text{-ODE}\left({𝑥}_{{𝑡}_1},{𝑣}_{{𝑡}_1},{𝑎}_{{𝑡}_1},{𝑑}_{{𝑡}_1\to 0}\right)-{𝑥}_0\Vert \right],\\ 2^{\text{ nd}}\text{-ODE }={𝑥}_{{𝑡}_1}+{𝑣}_{{𝑡}_1}\odot {𝑑}_{{𝑡}_1\to 0}+\frac{1}{2}{𝑎}_{{𝑡}_1}\odot {\left({𝑑}_{{𝑡}_1\to 0}\right)}^2\end{array}$$

2. Feature Concatenation and Time-gap Conditioning

After computing the acceleration features (${𝑎}_{𝑡}^{\ast }$ ), we concatenate these with the original velocity features (${𝑣}_{𝑡}^{\ast }$ ). To enable this concatenated feature to be aware of time-gap, we apply a time-gap-conditioned adaptive layer normalization with a following MLP as below:

$$\text{modulate}\left({𝑣}_{{𝑡}_1}^{\ast }\right)=\text{MLP}\left(\text{AdaLN-Zero}\left(\text{concat}\left({𝑣}_{{𝑡}_1}^{\ast },{𝑎}_{{𝑡}_1}^{\ast }\right),𝑇\left({𝑑}_{{𝑡}_1\to {𝑡}_2}\right)\right)\right)$$

3. Spatial Information Integration via Cross-Attention

Beyond feature alignment with temporal property using different time-steps, the VeRA block also integrates spatial context by employing a cross-attention (CA) mechanism. This mechanism facilitates interaction between two spaces: modulated velocity feature space from previous step and spatial feature space from an original patchified image as noted in following equation.

$${𝑣}_{{𝑡}_1\to {𝑡}_2}^{\ast }=\text{CA}\left(\text{modulate}\left({𝑣}_{{𝑡}_1}^{\ast }\right),{𝑥}_{{𝑡}_1}\right)$$

DeepFlow incorporates deep supervision and VeRA block

$${ℒ}_{\text{total }}={ℒ}_{\text{ deep }}+𝜆{ℒ}_{\text{ acc}}$$

5. Decoupled Diffusion Transformer (DDT)

The authors of "Decoupled Diffusion Transformer" (DDT) identify a fundamental "optimization dilemma" in existing diffusion transformers. In conventional architectures, the same neural network modules must simultaneously handle two conflicting tasks: encoding low-frequency semantic information from noisy inputs and decoding high-frequency details for reconstruction. This dual responsibility creates optimization tension because effective semantic encoding requires suppressing high-frequency noise, while detail reconstruction demands preserving and generating high-frequency components.

DDT resolves this optimization conflict through a principled decoupled encoder-decoder architecture that explicitly separates semantic extraction from detail reconstruction:

5.1. Condition Encoder

The condition encoder specializes in extracting low-frequency semantic components from three inputs: the noisy latent image ${𝑥}_{𝑡}$, timestep $𝑡$, and class label $𝑦$. It employs interleaved attention and feed-forward blocks similar to DiT/SiT, with timestep and class conditioning via AdaLN-Zero injection.

Crucially, the encoder receives direct supervision through Representation Alignment (REPAlign), which enforces consistency between extracted features and pre-trained vision foundation model representations (DINOv2).

5.2. Velocity Decoder

The velocity decoder focuses on processing the noisy latent ${𝑥}_{𝑡}$ alongside the self-condition feature ${𝑧}_{𝑡}$ to predict the velocity field ${𝑣}_{𝑡}$. Operating within a linear flow diffusion framework, it minimizes the flow matching loss.

5.3. Inference Acceleration Through Encoder Sharing

A key innovation of DDT is leveraging the consistency of extracted semantic features. ${𝑧}_{𝑡}$ across adjacent timesteps for inference acceleration. The authors propose two strategies:

• Uniform Encoder Sharing: Recomputing ${𝑧}_{𝑡}$ at fixed intervals (every K steps) rather than every denoising step.

• Statistical Dynamic Programming: A more sophisticated approach that frames optimal sharing as a minimal-cost path problem. Using a pre-computed similarity matrix $𝑆$ of cosine distances between ${𝑧}_{𝑡}$ features across timesteps, dynamic programming finds the optimal recalculation schedule that maximizes sharing while minimizing performance degradation.

6. Representation Entanglement for Generation (REG)

Introduce the class token ${\text{cls }}_0={𝑓}_0\left[0\right]$ from the vision foundation model to entangle with image latents for providing the discriminative guidance. ${\text{cls }}_{𝑡}={𝛼}_{𝑡}{\text{ cls }}_0+{𝜎}_{𝑡}{𝜀}_{\text{ cls}}$.

The prediction loss is formulated as:

$${ℒ}_{\text{pred }}=\int 𝐸\left[{\Vert 𝑣\left({𝑧}_{𝑡},𝑡\right)-{\dot{𝛼}}_{𝑡}{𝑧}_0-{\dot{𝜎}}_{𝑡}{𝜀}_{𝑧}\Vert }^2+𝛽{\Vert 𝑣\left({\text{cls}}_{𝑡},𝑡\right)-{\dot{𝛼}}_{𝑡}{\text{ cls }}_0-{\dot{𝜎}}_{𝑡}{𝜀}_{\text{ cls}}\Vert }^2\right]𝑑𝑡$$

The total loss is formulated as:

$${ℒ}_{\text{total }}={ℒ}_{\text{ pred }}+𝜆{ℒ}_{\text{ REPA}}$$

7. ReDi

Rather than aligning diffusion features with external representations via distillation, we propose to jointly model both images (specifically their VAE latents) and their high-level semantic features extracted from a pretrained vision encoder (e.g., DINOv2) within the same diffusion process.

7.1. Representation Guidance

To ensure the generated images remain strongly influenced by the visual representations during inference, we introduce Representation Guidance. This technique during inference modifies the posterior distribution to: ${\widehat{𝑝}}_{𝜃}\left({𝑥}_{𝑡},{𝑧}_{𝑡}\right)\propto {𝑝}_{𝜃}\left({𝑥}_{𝑡}\right){𝑝}_{𝜃}{\left({𝑧}_{𝑡}|{𝑥}_{𝑡}\right)}^{{𝑤}_{𝑟}}$, where ${𝑤}_{𝑟}$ controls how strongly samples are pushed toward higher likelihoods of the conditional distribution ${𝑝}_{𝜃}\left({𝑧}_{𝑡}|{𝑥}_{𝑡}\right)$. This yields the guided score function:

$${\nabla }_{{𝑥}_{𝑡}}\log {\widehat{𝑝}}_{𝜃}\left({𝑥}_{𝑡},{𝑧}_{𝑡}\right)={\nabla }_{{𝑥}_{𝑡}}\log {𝑝}_{𝜃}\left({𝑥}_{𝑡}\right)+{𝑤}_{𝑟}\left({\nabla }_{{𝑥}_{𝑡}}\log {𝑝}_{𝜃}\left({𝑧}_{𝑡}|{𝑥}_{𝑡}\right)\right)$$$$={\nabla }_{{𝑥}_{𝑡}}\log {𝑝}_{𝜃}\left({𝑥}_{𝑡}\right)+{𝑤}_{𝑟}\left({\nabla }_{{𝑥}_{𝑡}}\log {𝑝}_{𝜃}\left({𝑧}_{𝑡}\right)-{\nabla }_{{𝑥}_{𝑡}}\log {𝑝}_{𝜃}\left({𝑥}_{𝑡}\right)\right)$$

By recalling the equivalence of denoisers and scores, we implement this representation-guided prediction ${𝜀}_{𝜃}\left({𝑥}_{𝑡},{𝑧}_{𝑡},𝑡\right)$ at each denoising step as follows:

$${\widehat{𝜀}}_{𝜃}\left({𝑥}_{𝑡},{𝑧}_{𝑡},𝑡\right)={𝜀}_{𝜃}\left({𝑥}_{𝑡},𝑡\right)+{𝑤}_{𝑟}\left({𝜀}_{𝜃}\left({𝑥}_{𝑡},{𝑧}_{𝑡},𝑡\right)-{𝜀}_{𝜃}\left({𝑥}_{𝑡},𝑡\right)\right)$$

8. REPA-E

Higher representation-alignment score correlates with improved generation performance. This offers an alternate path for improving final generation performance using representation-alignment score as a proxy.

The maximum achievable alignment score with vanilla-REPA is bottlenecked by the VAE latent space features.

8.1. End-to-End Training with REPA

Batch-Norm Layer for VAE Latent Normalization. To enable end-to-end training, we first introduce a batchnorm layer between the VAE and latent diffusion model. Typical LDM training involves normalizing the VAE features using precomputed latent statistics (e.g., std $=\frac{1}{0.1825}$ for SD-VAE). This helps normalize the VAE latent outputs to zero mean and unit variance for more efficient training for the diffusion model. However, with end-to-end training the statistics need to be recomputed whenever the VAE model is updated - which is expensive. To address this, we propose the use of a batchnorm layer which uses the exponential moving average (EMA) mean and variance as a surrogate for dataset-level statistics. The batch-norm layer thus acts as a differentiable normalization operator without the need for recomputing dataset level statistics after each optimization step.

End-to-End Representation-Alignment Loss. We next enable end-to-end training, by using the REPA loss for updating the parameters for both VAE and LDM during training.

Diffusion Loss with Stop-Gradient. Backpropagating the diffusion loss to the VAE causes a degradation of latent-space structure.

VAE Regularization Losses. Finally, we introduce regularization losses ${ℒ}_{\text{reg}}$ for VAE ${𝒱}_{𝜑}$, to ensure that the end-to-end training process does not impact the reconstruction performance (rFID) of the original VAE.

Overall Training. The overall training is then performed in an end-to-end manner using the following loss,

$$ℒ\left(𝜃,𝜑,𝜔\right)={ℒ}_{\text{ DIFF }}\left(𝜃\right)+𝜆{ℒ}_{\text{ REPA}}\left(𝜃,𝜑,𝜔\right)+𝜂{ℒ}_{\text{ REG}}\left(𝜑\right)$$

9. U-REPA

Adapting REPA to U-Net architectures.

10. SoftREPA

While modern text-to-image (T2I) generative models have achieved remarkable success, a persistent challenge is the occasional misalignment between the generated image and the input text prompt. This can manifest as incorrect objects, attributes, or compositions. Existing approaches to fix this often require extensive fine-tuning or specialized preference datasets, which can be computationally expensive.

10.1. The SoftREPA Method

SoftREPA enhances text-image alignment by leveraging contrastive learning with a small set of trainable parameters, known as "soft tokens," while keeping the large pre-trained model frozen.

10.1.1. Learnable Soft Tokens

Soft tokens are learnable vectors that do not correspond to any specific words in a vocabulary. During the generation process, these tokens are prepended to the original text embeddings at various layers of the model's denoiser.

Because the main model remains frozen, only these soft tokens—fewer than 1 million parameters in total—are optimized during training. They act as adaptable guides, steering the model's internal representations toward better semantic alignment with the text prompt without requiring a full model retrain.

10.1.2. Contrastive T2I Alignment Loss

SoftREPA is trained using a contrastive framework that teaches the model to distinguish between correctly paired images and text (positive pairs) and mismatched pairs (negative pairs).

The "similarity" between an image $𝑥$ and a text $𝑦$ is defined based on the model's denoising performance. Intuitively, if the pair is a good match, the model ${𝑣}_{𝜃}$ should accurately predict the noise vector $𝜀$ that was added to the image. The similarity logit $\widetilde{𝑙}$ is formulated as an exponential of the negative denoising error:

$$\widetilde{𝑙}\left(𝑥,𝑦,𝑠\right)=\exp \left(-\frac{{\Vert {𝑣}_{𝜃}\left({𝑥}_{𝑡},𝑡,𝑦,𝑠\right)-\left(𝜀-{𝑥}_0\right)\Vert }^2}{𝜏\left(𝑡\right)}\right)$$

Here, a smaller denoising error results in a higher similarity score (closer to 1). The final SoftREPA loss is a contrastive objective that maximizes the similarity of positive pairs while minimizing it for negative pairs. Given a positive pair $\left(𝑥,𝑦\right)$ and a set of negative texts ${𝑦}^{𝑗}$, the loss for the learnable tokens $𝑠$ is:

$${ℒ}_{\text{SoftREPA }}\left(𝑠\right)=-𝐸\left[\log \left(\frac{\exp \left(\widetilde{𝑙}\left(𝑥,𝑦,𝑠\right)\right)}{\sum _{𝑗}\exp \left(\widetilde{𝑙}\left(𝑥,{𝑦}^{𝑗},𝑠\right)\right)}\right)\right]$$

This objective function effectively pushes the model's predicted noise for a positive pair closer to the ground truth while pushing the predictions for negative pairs further away.

11. Representation Autoencoders (RAE)

DiT have emerged as powerful models for generative tasks, regularly using a variational autoencoder (VAE) to compress input images into low-dimensional latent spaces for the diffusion process. However, most existing DiTs still depend on traditional VAE encoders that:

• Use outdated architectures
• Restrict information capacity with low-dimensional latents
• Rely on loss functions optimized only for reconstruction, limiting generative quality

11.1. High Fidelity Reconstruction From Frozen Encoders

DiT does not work out of the box. To our surprise, the standard diffusion recipe fails with RAE. Training directly on RAE latents causes a small backbone such as DiT-S to completely fail, while a larger backbone like DiT-XL significantly underperforms it's counterpart with the SD-VAE latents. To investigate this observation, we raise several hypotheses detailed below, which we will discuss in the following sections:

• Suboptimal design for diffusion transformers. When modeling high-dimensional RAE tokens, the optimal design choices for diffusion transformers can diverge from those of the standard DiT, which was originally tailored for low-dimensional VAE tokens

$\to$ Suboptimal design for diffusion transformers. We now fix the width of DiT to be at least as large as the RAE token dimension. For RAE with the DINOv2-B encoder, we pair it with DiT-XL in our following experiments.

• Suboptimal noise scheduling. Prior noise scheduling and loss re-weighting tricks are derived for image-based or VAE-based input, and it remains unclear if they transfer well to high-dimension semantic tokens

$\to$ Suboptimal noise scheduling. We now default the noise schedule to be dependent on the effective data dimension for all our following experiments.

• Diffusion generates noisy latents. VAE decoders are trained to reconstruct images from noisy latents, making them more tolerant to small noises in diffusion outputs. In contrast, RAE decoders are trained on only clean latents and may therefore struggle to generalize.

$\to$ Diffusion generates noisy latents. We now adopt the noise-augmented decoding for all our following experiments.

12. Self-supervised representations for Visual Generation (SVG)

The authors introduce SVG—a latent diffusion generator without a VAE. Instead, SVG uses self-supervised DINO features, which retain strong semantic discriminability. The framework consists of a frozen DINO backbone for feature extraction and a residual branch for fine visual details. The diffusion process takes place directly in this semantic space.

13. End-to-end Pixel-space Generative model (EPG)

Most high-resolution image generative models (e.g., diffusion, consistency models) rely on VAE-based latent representations to enable efficient training and high quality. Pure pixel-space generative modeling (operating directly on raw pixels, without a latent bottleneck) is much more challenging due to higher variance and complexity, and thus usually underperforms latent-space approaches.

This paper proposes a two-stage framework to significantly improve pixel-space generative models:

• Self-supervised Pre-training:

  • An encoder is trained via contrastive and representation consistency losses, ensuring that features are robust to noise and sampling trajectory. This step aligns the semantic representations between noisy inputs and their clean counterparts in pixel space.

• End-to-End Fine-tuning:

  • The projection head is removed, and the pre-trained encoder is paired with a randomly initialized decoder to jointly optimize for image generation via diffusion or consistency objectives.
  • Proposed adaptive temperature scheduling stabilizes contrastive training and prevents early-stage training collapse.

14. Discriminative Generative Image Transformer (DiGIT)

This paper introduces a new perspective on image generative modeling, highlighting that an optimal latent space for autoregressive generation should not be solely optimized for pixel reconstruction, but should also emphasize stability—meaning robustness to perturbations and resistance to error accumulation. The authors observe that autoregressive models perform worse than diffusion or iterative models (despite sharing the same latent spaces induced by standard autoencoders like VQGAN) because their generation process amplifies instability in the latent space.

To address this, the paper proposes a simple but effective method:

• The encoder and decoder are trained separately. The encoder is realized as a discriminative self-supervised model (such as DINOv2), which extracts robust, semantically meaningful features from image patches without training for reconstruction.
• These patch-level features are then discretized into tokens via K-Means clustering, forming a discrete and more stable latent codebook (tokenizer).
• An autoregressive Transformer is trained to generate these discrete tokens, following the standard causal next-token prediction setup (like GPT). The pixel decoder is trained separately to reconstruct images from the sequence of tokens.

Key findings:

• The discriminatively-induced latent space is much more stable under input noise and less sensitive to token prediction errors, making it more suitable for autoregressive image generation.
• The resulting model, called DiGIT, achieves state-of-the-art results on image understanding and image generation tasks. It even surpasses diffusion models and scales well with increased model size (mirroring the scaling success of GPT in text).
• Ablation studies show that a larger token vocabulary improves performance; the stability of the latent space is empirically and theoretically analyzed, drawing parallels between self-supervised discriminative encoders (LDA) and reconstructive autoencoders (PCA).
• This approach challenges the conventional view that reconstruction-optimized latent spaces are ideal for generative modeling, advocating for stability-targeted designs instead.

15. UniFlow

,caption: [Comparison of different training paradigms for unified tokenizers.])

UniFlow proposes a new unified vision tokenizer designed for both visual understanding and generation tasks, overcoming trade-offs faced by existing approaches.

Method Design: UniFlow flexibly adapts pretrained vision encoders and introduces a lightweight pixel flow decoder for high-fidelity pixel reconstruction. Its core innovation is a hierarchical adaptive self-distillation mechanism, letting the unified encoder inherit strong semantic capability while preserving fine visual details and improving training efficiency.

References

1. Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think
2. No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves
3. Deeply Supervised Flow-Based Generative Models
4. Decoupled Diffusion Transformer
5. Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think
6. Boosting Generative Image Modeling via Joint Image-Feature Synthesis
7. REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers
8. U-REPA: Aligning Diffusion U-Nets to ViTs
9. Aligning Text to Image in Diffusion Models is Easier Than You Think
10. LayerSync: Self-aligning Intermediate Layers
11. Diffusion Transformers with Representation Autoencoders
12. Advancing End-to-End Pixel Space Generative Modeling via Self-supervised Pre-training
13. Latent Diffusion Model without Variational Autoencoder
14. Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective
15. UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation