Introduction of generative adversarial networks (GANs)

Last updated on:7 days ago

GANs are widely used in generating new data such as images and sequences by using the training dataset. It can help us enlarge the dataset to balance data for deep learning jobs. The application seems like two stages of deep learning architecture.

Concepts

Understand the difference between generative and discriminative models.

Generative

Generative models can generate new data instances and capture the joint probability $p(X, Y)$, or just $p(X)$ if there are no labels.

Discriminative

Discriminative models discriminate between different data instances and capture the conditional probability $p(Y|X)$.

GAN Training

Because a GAN contains two separately trained networks, its training algorithm must address two complications

GANs must juggle two different kinds of training (generator and discriminator).

GAN convergence is hard to identify.

Alternating Training

  1. The discriminator trains for one or more epochs.
  2. The generator trains for one or more epochs.
  3. Repeat steps 1 and 2 to continue to train the generator and discriminator networks.

We keep the generator constant during the discriminator training phase. As discriminator training tries to figure out how to distinguish real data from fake, it has to learn how to recognize the generator’s flaws.

Similarly, we keep the discriminator constant during the generator training phase. Otherwise, the generator would be trying to hit a moving target and might never converge.

Convergence

As the generator improves with training, the discriminator performance worsens because the discriminator can’t quickly tell the difference between real and fake. If the generator succeeds perfectly, then the discriminator has a $50%$ accuracy.

Mode Collapse

If a generator produces an especially plausible output, the generator may learn to produce only that output. The generator is always trying to find the one output that seems most reasonable to the discriminator.

Attempts to Remedy

Try to force the generator to broaden its scope by preventing it from optimizing.

During generator training, gradients propagate through the discriminator network to the generator network (although the discriminator does not update its weights). So the weights in the discriminator network influence the updates to the generator network.

While a GAN can use the same loss for both generator and discriminator training (or the same loss differing only in sign), it’s not required. In fact, it’s more common to use different losses for the discriminator and the generator.

Loss function

The generator and discriminator losses look different in the end, even though they derive from a single formula.

Loss function:

$$\mathcal{L} = E_x [\log (D(x))] + E_z [\log (1 - D(G(z)))]$$

Conditional GANs loss function:

Q&A

1.Identify problems that GANs can solve.

Example uses

Create training data

Increase image resolution

Morph audio

2.Understand the roles of the generator and discriminator in a GAN system.

The discriminator in a GAN is simply a classifier. It tries to distinguish real data from the data created by the generator. It could use any network architecture appropriate to the type of data it’s classifying.

The generator part of a GAN learns to create fake data by incorporating feedback from the discriminator. It learns to make the discriminator classify its output as real.

  1. Understand the advantages and disadvantages of standard GAN loss functions.

GANs try to replicate a probability distribution. Therefore, they should use loss functions that reflect the distance between the distribution of the data generated by the GAN and the real data distribution.

Reference

[1] Generative Adversarial Networks

[2] Karras, T., Aila, T., Laine, S. and Lehtinen, J., 2017. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196.

[3] Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J. and Aila, T., 2021, May. Alias-free generative adversarial networks. In Thirty-Fifth Conference on Neural Information Processing Systems.

[4] Gui, J., Sun, Z., Wen, Y., Tao, D. and Ye, J., 2020. A review on generative adversarial networks: Algorithms, theory, and applications. arXiv preprint arXiv:2001.06937.