1. One problem, many fixes

Joint-embedding SSL starts with a simple wish: two related views $𝑥$ and ${𝑥}^{\prime }$ should have similar representations. If

$$𝑧={𝑓}_{𝜙}\left(𝑥\right),{𝑧}^{\prime }={𝑓}_{𝜙}\left({𝑥}^{\prime }\right),$$

then we want $𝑧\approx {𝑧}^{\prime }$. By itself this collapses. The encoder can set ${𝑓}_{𝜙}\left(𝑥\right)=𝑐$ for every input, and every positive pair will be perfectly aligned.

The field has many ways to prevent this:

• SimCLR (Chen et al., 2020), MoCo, and CPC (Oord et al., 2018) use negative samples.
• BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021) use a predictor plus stop-gradient.
• DINO (Caron et al., 2021) uses an EMA teacher with centering and sharpening.
• Barlow Twins (Zbontar et al., 2021) and VICReg (Bardes et al., 2022) regularize variance and covariance.
• LeJEPA’s SIGReg (Balestriero & LeCun, 2025) and VISReg (Wu et al., 2026) test the embedding distribution with random one-dimensional projections.

The usual summary is alignment and uniformity (Wang & Isola, 2020): pull positives together, then spread the representation out. Latent distribution matching (LDM) (Mikulasch & Zenke, 2026) makes that picture more precise. SSL is choosing a desired latent geometry, then learning an encoder whose induced latent distribution matches it.

2. The objective

Start with ordinary maximum likelihood:

$${𝔼}_{{𝑃}_{\text{data}}\left(𝑥\right)}\left[log{𝑃}_{𝜃}\left(𝑥\right)\right].$$

This says: draw real data $𝑥$ from ${𝑃}_{\text{data}}$, ask how much probability the model ${𝑃}_{𝜃}$ assigns to it, and average.

Now introduce an encoder:

$$𝑧={𝑓}_{𝜙}\left(𝑥\right).$$

This creates a new distribution in latent space. If $𝑥\sim {𝑃}_{\text{data}}$, then $𝑧={𝑓}_{𝜙}\left(𝑥\right)$ has an induced distribution ${𝑄}_{𝜙}\left(𝑧\right)$. The three objects are:

1. ${𝑃}_{\text{data}}\left(𝑥\right)$: the real data distribution. We have samples, not a formula.
2. ${𝑄}_{𝜙}\left(𝑧\right)$: the embedding distribution produced by the current encoder.
3. ${𝑃}_{𝜃}\left(𝑧\right)$: the latent model we want the embeddings to match.

For an invertible change of variables, likelihood in data space can be rewritten as latent likelihood plus latent entropy:

$${𝔼}_{{𝑃}_{\text{data}}\left(𝑥\right)}\left[log{𝑃}_{𝜃}\left(𝑥\right)\right]\propto \underset{\text{latent likelihood}}{\underbrace{{𝔼}_{{𝑄}_{𝜙}}\left[log{𝑃}_{𝜃}\left(𝑧\right)\right]}}+\underset{\text{latent entropy}}{\underbrace{{𝐻}_{{𝑄}_{𝜙}}\left[𝑧\right]}}=-{𝐷}_{\text{KL}}\left({𝑄}_{𝜙}\left(𝑧\right)\parallel {𝑃}_{𝜃}\left(𝑧\right)\right).$$

The likelihood term says encoded samples should land where the latent model has high probability. The entropy term says the encoder should not fold many data points onto the same latent point.

This is why ICA is the right analogy (Hyvärinen & Oja, 2000). Linear ICA observes mixtures $𝑥=𝐴𝑠$ and learns

$$𝑧=𝑊𝑥$$

so that $𝑧$ looks like independent sources:

$${𝑃}_{𝜃}\left(𝑧\right)=\prod _{𝑖}{𝑃}_{𝜃}\left({𝑧}_{𝑖}\right).$$

ICA chooses a source model and learns a transformation whose induced distribution matches it. SSL does the same thing with a deep encoder and paired views.

For SSL, the objects are pairs:

$$\left(𝑥,{𝑥}^{\prime }\right)\to \left(𝑧,{𝑧}^{\prime }\right)=\left({𝑓}_{𝜙}\left(𝑥\right),{𝑓}_{𝜙}\left({𝑥}^{\prime }\right)\right).$$

These pairs induce ${𝑄}_{𝜙}\left(𝑧,{𝑧}^{\prime }\right)$. We choose a latent pair model

$${𝑃}_{𝜃}\left(𝑧,{𝑧}^{\prime }\right)={𝑃}_{𝜃}\left(𝑧\right){𝑃}_{𝜃}\left({𝑧}^{\prime }|𝑧\right),$$

and train the encoder so that

$${𝑄}_{𝜙}\left(𝑧,{𝑧}^{\prime }\right)\approx {𝑃}_{𝜃}\left(𝑧,{𝑧}^{\prime }\right).$$

The LDM objective is

$${\mathrm{ℱ︀}}_{\text{LDM}}=-{𝐷}_{\text{KL}}\left({𝑄}_{𝜙}\left(𝑧,{𝑧}^{\prime }\right)\parallel {𝑃}_{𝜃}\left(𝑧,{𝑧}^{\prime }\right)\right)=\underset{\text{alignment}}{\underbrace{{𝔼}_{{𝑄}_{𝜙}}\left[log{𝑃}_{𝜃}\left(𝑧,{𝑧}^{\prime }\right)\right]}}+\underset{\text{uniformity}}{\underbrace{{𝐻}_{{𝑄}_{𝜙}}\left[𝑧,{𝑧}^{\prime }\right]}}.$$

The split is natural. Since

$$log{𝑃}_{𝜃}\left(𝑧,{𝑧}^{\prime }\right)=log{𝑃}_{𝜃}\left({𝑧}^{\prime }|𝑧\right)+log{𝑃}_{𝜃}\left(𝑧\right),$$

the conditional ${𝑃}_{𝜃}\left({𝑧}^{\prime }|𝑧\right)$ defines how related views should align, while the marginal ${𝑃}_{𝜃}\left(𝑧\right)$ and entropy term define the global shape of the embedding cloud. This is the main idea: choose a latent geometry, then learn ${𝑓}_{𝜙}$ to realize it.

2.1. Mutual information is not the main point

Many SSL papers describe the goal as maximizing mutual information between views (Oord et al., 2018) (Shwartz-Ziv et al., 2023). LDM says that is not the central force. If we add MI to the objective, we get

$${\mathrm{ℱ︀}}_{\text{MI}}={\mathrm{ℱ︀}}_{\text{LDM}}+{𝐼}_{{𝑄}_{𝜙}}\left[𝑧,{𝑧}^{\prime }\right]={𝔼}_{{𝑄}_{𝜙}}\left[log{𝑃}_{𝜃}\left(𝑧,{𝑧}^{\prime }\right)\right]+2{𝐻}_{{𝑄}_{𝜙}}\left[𝑧\right],$$

assuming both views have the same marginal entropy.

MI cannot choose the representation geometry. If $\widetilde{𝑧}=𝑔\left(𝑧\right)$ is an invertible reparameterization, then

$$𝐼\left(𝑧,{𝑧}^{\prime }\right)=𝐼\left(\widetilde{𝑧},\widetilde{{𝑧}^{\prime }}\right).$$

So MI can say information is preserved, but not whether classes, positions, or dynamics are linearly exposed. In (Mikulasch & Zenke, 2026), turning MI on or off barely changes linear-probe accuracy, while changing the entropy estimator moves accuracy by much more.

3. Axis 1: the latent model

The latent model is the inductive bias. Different choices of ${𝑃}_{𝜃}\left({𝑧}^{\prime }|𝑧\right)$ give different alignment losses.

• Gaussian conditional. Choose

  $${𝑃}_{𝜃}\left({𝑧}^{\prime }|𝑧\right)=\mathrm{𝒩︀}\left({𝑧}^{\prime };𝑧,{𝜎}^2𝐼\right).$$

  Then

  $$log{𝑃}_{𝜃}\left({𝑧}^{\prime }|𝑧\right)=-\frac{1}{2{𝜎}^2}{\Vert {𝑧}^{\prime }-𝑧\Vert }^2+𝐶,$$

  which gives the MSE invariance loss of VICReg.

• von Mises-Fisher conditional. Put latents on the unit sphere and choose

  $${𝑃}_{𝜃}\left({𝑧}^{\prime }|𝑧\right)\propto exp\left(𝛽{𝑧}^{\top }{𝑧}^{\prime }\right).$$

  The log-likelihood is cosine similarity, which gives SimCLR’s positive-pair term.

• Dirichlet conditional. Put latents on the simplex and choose

  $${𝑃}_{𝜃}\left({𝑧}^{\prime }|𝑧\right)=\text{Dir}\left({𝑧}^{\prime };𝜏𝑧+1\right).$$

  The log-likelihood contains $𝜏{\sum }_{𝑘}{𝑧}_{𝑘}log{𝑧}_{𝑘}^{\prime }$, which gives the teacher-student cross-entropy used by DINO-like methods.

• Predictive conditional. For time or masked prediction, choose

  $${𝑃}_{𝜃}\left({𝑧}_{𝑡}|{𝑧}_{:𝑡}\right)=\mathrm{𝒩︀}\left({𝑧}_{𝑡};{𝑝}_{𝜃}\left({𝑧}_{:𝑡}\right),\mathrm{\Sigma }\right).$$

  This gives the JEPA family (Assran et al., 2023) (Bardes et al., 2024): predict the next or missing latent, not the next or missing pixels.

So SSL is not mainly compression. It is a way to choose a coordinate system on the data manifold. The latent model says which geometry we want.

4. Axis 2: the uniformity mechanism

Alignment alone collapses, so each method also needs a way to make ${𝑄}_{𝜙}\left(𝑧\right)$ spread out. In LDM this is the entropy or marginal-matching side.

Uniformity mechanism	What it does	Methods
KDE / contrastive density estimate	$-\frac{1}{𝑛}{\sum }_{𝑖}log{\sum }_{𝑗}exp\left(𝛽{𝑧}_{𝑖}^{\top }{𝑧}_{𝑗}\right)$ repels crowded samples	SimCLR, CPC
Gaussian entropy via $log|{\mathrm{\Sigma }}_{𝑧}|$	increase variance and reduce covariance	VICReg, Barlow Twins
Conditional entropy plug-in	use predictor plus stop-gradient	BYOL, SimSiam, JEPA
Sliced distribution test	match random projections to $\mathrm{𝒩︀}\left(0,𝐼\right)$	SIGReg, VISReg

SimCLR’s negatives can be read as a kernel density entropy estimator. If embeddings crowd together, the local density estimate is high and entropy is low, so the loss pushes points apart.

VICReg approximates Gaussian entropy:

$${𝐻}_{{𝑄}_{𝜙}}\left[𝑧\right]\propto log\left|{\mathrm{\Sigma }}_{𝑧}\right|.$$

Near a diagonal covariance,

$$log\left|{\mathrm{\Sigma }}_{𝑧}\right|\approx \sum _{𝑖}log{\mathrm{\Sigma }}_{𝑖𝑖}-\frac{1}{2}\sum _{𝑖\ne 𝑗}\frac{{\mathrm{\Sigma }}_{𝑖𝑗}{\mathrm{\Sigma }}_{𝑗𝑖}}{{\mathrm{\Sigma }}_{𝑖𝑖}{\mathrm{\Sigma }}_{𝑗𝑗}}.$$

This is why VICReg has a variance term and a covariance term (Shwartz-Ziv et al., 2023). The weakness is also visible: covariance only sees second-order statistics.

BYOL, SimSiam, and JEPA use predictor plus stop-gradient. In predictive LDM, the target contains

$$𝔼\left[log{𝑃}_{𝜃}\left({𝑧}_{𝑡}|{𝑧}_{:𝑡}\right)\right]+{𝐻}_{{𝑄}_{𝜙}}\left[{𝑧}_{𝑡}|{𝑧}_{:𝑡}\right].$$

The stop-gradient objective approximates the conditional entropy term:

$$𝔼\left[log{𝑃}_{𝜃}\left({𝑧}_{𝑡}|{𝑧}_{:𝑡}\right)\right]-𝔼\left[log{𝑃}_{\text{SG}\left[𝜃\right]}\left({𝑧}_{𝑡}|\text{SG}\left[{𝑧}_{:𝑡}\right]\right)\right].$$

This makes the usual predictor, target network, and stop-gradient less mysterious. They are not just asymmetry tricks. They make a conditional entropy estimate usable.

DINO fits the alignment side cleanly through teacher-student cross-entropy on the simplex. Its anti-collapse side, centering plus sharpening, is less clearly an entropy estimator. This is why the LDM paper treats it as more heuristic.

5. From implicit entropy to explicit sketching

If uniformity means matching the marginal ${𝑄}_{𝜙}\left(𝑧\right)$, then we can skip entropy estimation and directly test the distribution. LeJEPA (Balestriero & LeCun, 2025) chooses

$$𝑃\left(𝑧\right)=\mathrm{𝒩︀}\left(0,𝐼\right).$$

SIGReg checks this with random one-dimensional projections and an Epps-Pulley normality test (Epps & Pulley, 1983). By the Cramer-Wold theorem (Cramér & Wold, 1936), matching all one-dimensional projections matches the full distribution.

VISReg (Wu et al., 2026) keeps the Gaussian target but separates scale, center, and shape. With centered embeddings $\widehat{𝑍}\in {ℝ}^{𝑁\times 𝐷}$:

$${\mathrm{ℒ︀}}_{\text{scale}}=\frac{1}{𝐷}\sum _{𝑗=1}^{𝐷}{\left(1-{𝜎}_{𝑗}\left(\widehat{𝑍}\right)\right)}^2,{\mathrm{ℒ︀}}_{\text{center}}={\Vert 𝜇\Vert }_2^2.$$

Then it removes scale before testing shape:

$$\widetilde{𝑍}=\widehat{𝑍}/\left(\text{sg}\left(𝜎\right)+𝜀\right),$$

and matches random projections to standard-normal quantiles:

$${\mathrm{ℒ︀}}_{\text{shape}}=\frac{1}{𝐾}\sum _{𝑘=1}^{𝐾}{\Vert \text{sort}\left(\widetilde{𝑍}{𝑤}_{𝑘}\right)-{𝑞}_{\mathrm{𝒩︀}}\Vert }_2^2.$$

The full loss is

$$\mathrm{ℒ︀}=\left(1-𝜆\right){\mathrm{ℒ︀}}_{\text{pred}}+𝜆\left({𝜆}_{\text{scale}}{\mathrm{ℒ︀}}_{\text{scale}}+{𝜆}_{\text{shape}}{\mathrm{ℒ︀}}_{\text{shape}}+{𝜆}_{\text{center}}{\mathrm{ℒ︀}}_{\text{center}}\right).$$

Compared with SIGReg, VISReg keeps a stronger gradient near collapse and lets scale and shape be reweighted separately. Compared with VICReg, it tests more than covariance.

6. What this buys

The first payoff is identifiability. With a predictive Gaussian or vMF conditional, LDM can recover the true latent variables up to an affine transformation under the paper’s assumptions (Mikulasch & Zenke, 2026). The source of identifiability is the conditional noise model, not the architecture.

The second payoff is robustness. VISReg reports that explicit regularization methods remain stable on long-tailed ImageNet-LT and low-rank Galaxy10, while DINO can collapse on the long-tailed setting (Wu et al., 2026). VISReg also benefits from its scale-shape split: hard data can be handled by increasing the shape weight.

The tradeoff is that global marginal matching does not directly constrain spatial token structure. That likely explains why dense prediction is still a gap compared with methods such as MoCoV3 and iBOT.

7. A design recipe

The LDM view turns SSL design into a short checklist:

1. Choose ${𝑃}_{𝜃}\left({𝑧}^{\prime }|𝑧\right)$, which fixes the alignment loss and the inductive bias.
2. Choose a uniformity mechanism, and ask whether it estimates only covariance, whether its gradient survives collapse, and what it costs.
3. Separate scale from shape when possible.
4. Do not expect MI to choose the geometry. Entropy and explicit marginal matching are doing that work.

Seen this way, SSL is close in spirit to ICA. Pick a latent model, keep the encoder informative on the data manifold, and match distributions. The old anti-collapse toolbox, negatives, EMA teachers, stop-gradients, covariance penalties, is a collection of entropy estimators of uneven quality. VISReg replaces part of that implicit machinery with an explicit test of the embedding distribution.

References

1. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E. A Simple Framework for Contrastive Learning of Visual Representations. International Conference on Machine Learning (ICML), 2020.
2. Hyvärinen, A., and Oja, E. Independent component analysis: algorithms and applications. Neural Networks, 2000.
3. Bardes, A., Ponce, J., and LeCun, Y. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. International Conference on Learning Representations (ICLR), 2022.
4. Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. International Conference on Machine Learning (ICML), 2021.
5. Hyvärinen, A., Sasaki, H., and Turner, R. E. Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning. International Conference on Artificial Intelligence and Statistics (AISTATS), 2019.
6. Chen, X., and He, K. Exploring Simple Siamese Representation Learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
7. Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., and Brendel, W. Contrastive Learning Inverts the Data Generating Process. International Conference on Machine Learning (ICML), 2021.
8. Mikulasch, F. A., and Zenke, F. Understanding Self-Supervised Learning via Latent Distribution Matching, 2026.
9. Shwartz-Ziv, R., Balestriero, R., Kawaguchi, K., Rudner, T. G. J., and LeCun, Y. An Information Theory Perspective on Variance-Invariance-Covariance Regularization. Advances in Neural Information Processing Systems (Neurips), 2023.
10. Balestriero, R., and LeCun, Y. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics, 2025.
11. Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. Á., Guo, Z., Azar, M. G., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. Advances in Neural Information Processing Systems (Neurips), 2020.
12. Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
13. Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging Properties in Self-Supervised Vision Transformers. IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
14. Wang, T., and Isola, P. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. International Conference on Machine Learning (ICML), 2020.
15. Wu, H., Balestriero, R., and Levine, M. VISReg: Variance-Invariance-Sketching Regularization for JEPA training, 2026.
16. Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., and Lucic, M. On Mutual Information Maximization for Representation Learning. International Conference on Learning Representations (ICLR), 2020.
17. Cramér, H., and Wold, H. Some Theorems on Distribution Functions. Journal of the London Mathematical Society, 1936.
18. Bardes, A., Garrido, Q., Ponce, J., Chen, X., Rabbat, M., LeCun, Y., Assran, M., and Ballas, N. Revisiting Feature Prediction for Learning Visual Representations from Video. Transactions on Machine Learning Research (TMLR), 2024.
19. Oord, A. van den, Li, Y., and Vinyals, O. Representation Learning with Contrastive Predictive Coding, 2018.
20. Epps, T. W., and Pulley, L. B. A test for normality based on the empirical characteristic function. Biometrika, 1983.