1. Use implicit generative models

We define a prior distribution ${𝑝}_{𝑧}\left(𝑧\right)$ and a generative model $𝐺\left(𝑧\right)$ to generate samples. Thus we have ${𝑝}_{𝑔}\left(𝑥\right)=\int {𝑝}_{𝑧}\left(𝑧\right)𝛿\left(𝑥-𝐺\left(𝑧\right)\right)𝑑𝑧$. If we view $𝑞\left(𝑧\right)={𝑝}_{𝑧}\left(𝑧\right)$ and $𝑞\left(𝑥|𝑧\right)=𝛿\left(𝑥-𝐺\left(𝑧\right)\right)$, then we are doing variational inference.

1.1. GAN

We want to have some metric to measure the similarity between ${𝑝}_{𝑔}\left(𝑥\right)$ and ${𝑝}_{\text{data }}\left(𝑥\right)$. GAN uses a discriminator to measure the similarity.

• Generator $𝐺$: Maps random noise $𝑧$ to data-like samples $𝐺\left(𝑧\right)$
• Discriminator $𝐷$: Classifies samples as real (from data) or fake (from generator)

1.1.1. The Discriminator's Goal

The discriminator wants to:

• Output high probability for real data: maximize $\log 𝐷\left(𝑥\right)$ for $𝑥\sim {𝑝}_{\text{ data }}\left(𝑥\right)$
• Output low probability for fake data: maximize $\log \left(1-𝐷\left(𝐺\left(𝑧\right)\right)\right)$ for $𝑧\sim {𝑝}_{𝑧}\left(𝑧\right)$

Combined objective for discriminator:

$$\underset{𝐷}{\max }{𝐸}_{𝑥\sim {𝑝}_{\text{ data }}\left(𝑥\right)}\left[\log 𝐷\left(𝑥\right)\right]+{𝐸}_{𝑧\sim {𝑝}_{𝑧}\left(𝑧\right)}\left[\log \left(1-𝐷\left(𝐺\left(𝑧\right)\right)\right)\right]$$

1.1.2. The Generator's Goal

The generator wants to fool the discriminator:

• Make $𝐷\left(𝐺\left(𝑧\right)\right)$ as close to 1 as possible
• Minimize $\log \left(1-𝐷\left(𝐺\left(𝑧\right)\right)\right)$

Generator objective:

$$\underset{𝐺}{\min }{𝐸}_{𝑧\sim {𝑝}_{𝑧}\left(𝑧\right)}\left[\log \left(1-𝐷\left(𝐺\left(𝑧\right)\right)\right)\right]$$

Combining both objectives, we get the famous minimax game:

$$\underset{𝐺}{\min }\underset{𝐷}{\max }𝑉\left(𝐺,𝐷\right)={𝐸}_{𝑥\sim {𝑝}_{\text{ data }}\left(𝑥\right)}\left[\log 𝐷\left(𝑥\right)\right]+{𝐸}_{𝑧\sim {𝑝}_{𝑧}\left(𝑧\right)}\left[\log \left(1-𝐷\left(𝐺\left(𝑧\right)\right)\right)\right]$$

We can't solve the minimax problem directly, so we alternate:

1. Fix G, train D: Given current generator, train discriminator to distinguish real from fake
2. Fix D, train G: Given current discriminator, train generator to fool it
3. Repeat: This creates an arms race that drives both models to improve

Question: What happens at the optimal solution?

At equilibrium, the generator should recover the data distribution: ${𝑝}_{𝑔}={𝑝}_{\text{ data}}$.

Proof sketch:

1. For fixed $𝐺$, the optimal discriminator is:

   $${𝐷}_{𝐺}^{\ast }\left(𝑥\right)=\frac{{𝑝}_{\text{data }}\left(𝑥\right)}{{𝑝}_{\text{data }}\left(𝑥\right)+{𝑝}_{𝑔}\left(𝑥\right)}$$

2. Substituting this back into the objective:

   $$𝐶\left(𝐺\right)=\underset{𝐷}{\max }𝑉\left(𝐺,𝐷\right)={𝐸}_{𝑥\sim {𝑝}_{\text{ data}}}\left[\log \frac{{𝑝}_{\text{data }}\left(𝑥\right)}{{𝑝}_{\text{data }}\left(𝑥\right)+{𝑝}_{𝑔}\left(𝑥\right)}\right]+{𝐸}_{𝑥\sim {𝑝}_{𝑔}}\left[\log \frac{{𝑝}_{𝑔}\left(𝑥\right)}{{𝑝}_{\text{data }}\left(𝑥\right)+{𝑝}_{𝑔}\left(𝑥\right)}\right]$$

3. This can be rewritten as:

   $$𝐶\left(𝐺\right)=-\log \left(4\right)+2·\text{JSD}\left({𝑝}_{\text{data }}\Vert {𝑝}_{𝑔}\right)$$

   where JSD is the Jensen-Shannon divergence.

4. Since $\text{JSD }\ge 0$, the global minimum is achieved when ${𝑝}_{𝑔}={𝑝}_{\text{ data}}$.

1.1.3. Evaluation of GAN

1. Inception Score (IS)

$𝑝\left(𝑦|𝑥\right)=\text{ Inception-v3 }\left(𝑥\right),𝑝\left(𝑦\right)={𝐸}_{𝑥}\left[𝑝\left(𝑦|𝑥\right)\right],\text{ IS }=\exp \left({𝐸}_{𝑥}\left[\text{KL}\left(𝑝\left(𝑦|𝑥\right)\Vert 𝑝\left(𝑦\right)\right)\right]\right)$, the higher the better.

IS only measures the quality of the generated samples, but we also want the distribution to be similar to ${𝑝}_{\text{data}}$. If the model just memorize the training data, it will also have high IS.

2. Fréchet Inception Distance (FID)

FID measures the distance between real and generated image distributions in Inception feature space.

For both real images ${𝑋}_{𝑟}=\left\{{𝑥}_1^{𝑟},{𝑥}_2^{𝑟},\dots ,{𝑥}_{𝑁}^{𝑟}\right\}$ and generated images ${𝑋}_{𝑔}=\left\{{𝑥}_1^{𝑔},{𝑥}_2^{𝑔},\dots ,{𝑥}_{𝑀}^{𝑔}\right\}$:

$${𝐹}_{𝑟}=\text{Inception}\left({𝑋}_{𝑟}\right)\in {\mathrm{ℝ}}^{𝑁\times 2048},{𝐹}_{𝑔}=\text{Inception}\left({𝑋}_{𝑔}\right)\in {\mathrm{ℝ}}^{𝑀\times 2048}$$

Assume features follow multivariate Gaussian distributions:

$${𝐹}_{𝑟}\sim 𝒩\left({𝜇}_{𝑟},{\mathrm{\Sigma }}_{𝑟}\right)\text{ and }{𝐹}_{𝑔}\sim 𝒩\left({𝜇}_{𝑔},{\mathrm{\Sigma }}_{𝑔}\right)$$

Compute sample statistics:

$${𝜇}_{𝑟}=\frac{1}{𝑁}\sum _{𝑖=1}^{𝑁}{𝑓}_{𝑖}^{𝑟},{\mathrm{\Sigma }}_{𝑟}=\text{Cov}\left({𝐹}_{𝑟}\right),{𝜇}_{𝑔}=\frac{1}{𝑀}\sum _{𝑗=1}^{𝑀}{𝑓}_{𝑗}^{𝑔},{\mathrm{\Sigma }}_{𝑔}=\text{Cov}\left({𝐹}_{𝑔}\right)$$

The distance between two multivariate Gaussians:

$$\text{FID }=\Vert {𝜇}_{𝑟}-{𝜇}_{𝑔}{\Vert }^2+\text{Tr}\left({\mathrm{\Sigma }}_{𝑟}+{\mathrm{\Sigma }}_{𝑔}-2{\left({\mathrm{\Sigma }}_{𝑟}{\mathrm{\Sigma }}_{𝑔}\right)}^{\frac{1}{2}}\right)$$

Component	Meaning
$\Vert {𝜇}_{𝑟}-{𝜇}_{𝑔}{\Vert }^2$	Difference in central tendency (Are generated images "on average" similar to real ones?)
$\text{Tr}\left({\mathrm{\Sigma }}_{𝑟}+{\mathrm{\Sigma }}_{𝑔}-2{\left({\mathrm{\Sigma }}_{𝑟}{\mathrm{\Sigma }}_{𝑔}\right)}^{\frac{1}{2}}\right)$	Difference in variation patterns (Do generated images have similar diversity and correlations?)

Lower is better: FID = 0 when ${𝜇}_{𝑟}={𝜇}_{𝑔}$ and ${\mathrm{\Sigma }}_{𝑟}={\mathrm{\Sigma }}_{𝑔}$. FID values are dataset-dependent. Compare only within the same dataset and experimental setup.

1.1.4. Deep Convolutional GANs (DCGAN)

• Use fully convolutional layers
• Use batch normalization to stabilize training
• Avoid ReLU activation in discriminator to avoid gradient vanishing
• Use small learning rate and momentum

1.1.5. Improved Training Techniques for GAN

• Feature Matching
• Minibatch discrimination
• Historical averaging
• One-sided label smoothing
• Virtual batch normalization

1.1.6. Wasserstein GANs (WGANs)

When using JS divergence, gradient vanishes when ${𝑝}_{\text{data}}$ and ${𝑝}_{𝑔}$ are fully disjoint and this will cause mode collapse. So we use Wasserstein distance instead.

$$\underset{𝐺}{\min }\underset{𝐷,{\Vert 𝐷\Vert }_{𝐿}\le 1}{\max }{𝐸}_{𝑥\sim {𝑝}_{\text{ data}}}\left[𝐷\left(𝑥\right)\right]-{𝐸}_{𝑥\sim {𝑝}_{𝐺}}\left[𝐷\left(𝑥\right)\right]$$

But how to add Lipschitz constraint?

1. Weight Clipping (original WGAN)

If each layer of the network has bounded weights, and we use activation functions that are Lipschitz (like ReLU or tanh), then the composition of these functions maintains the Lipschitz property.

2. Gradient Penalty (WGAN-GP)

$$ℒ={𝐸}_{𝑥\sim {𝑝}_{\text{ data}}}\left[𝐷\left(𝑥\right)\right]-{𝐸}_{𝑥\sim {𝑝}_{𝐺}}\left[𝐷\left(𝑥\right)\right]+𝜆\cdot {𝐸}_{𝑥\sim 𝑟\left(𝑥\right)}\left[{\left(\Vert {\nabla }_{𝑥}𝐷\left(𝑥\right)\Vert -1\right)}^2\right]$$

where $𝑟\left(𝑥\right)=\left(1-𝜀\right){𝑥}_{\text{data }}+𝜀{𝑥}_{𝐺}$ with $𝜀\sim 𝑈\left[0,1\right]$.

1.1.7. BigGAN, GigaGAN, R3GAN, BiGAN, BigBiGAN

• BigGAN: scaling up for quality, class-conditional batch normalization
• GigaGAN: text-to-image at scale
• R3GAN: Reliable, Realistic, and Robust
• BiGAN: Bidirectional GAN
• BigBiGAN: Scaling Bidirectional Learning