Seeing What a GAN Cannot Generate

State-of-the art GANs can create increasingly realistic images, yet they are not perfect.

What is a GAN unable to generate? This repository contains the code for the ICCV 2019 paper Seeing What a GAN Cannot Generate, which introduces a framework that can be used to answer this question.

GAN reconstruction	Real photo

Our goal is not to benchmark how far the generated distribution is from the target. Instead, we want to visualize and understand what is different between real and fake images.

Mode-dropping and the problem of visualizing omissions

We visualize the omissions of an image generator in two ways.

1. We identify omissions within the distribution of images.
2. We identify omissions within individual images.

Seeing omissions in the distribution

To understand omissions in a GAN's output distribution, we compare segmentation statistics between the GAN output and the training distribution.

A Progressive GAN trained to generate LSUN outdoor church images is analyzed below.

The model does not generate enough pixels of people, cars, fences, palm trees, or signboards compared to the training distribution. The script run_fsd.sh and the notebook seeing_distributions.ipynb show how we collect and visualize these segmentation statistics.

Seeing omissions in individual images

To understand omission in specific GAN-generated output, we must pair the output with a real photo that shows what the GAN should have drawn but did not. So we compare real training photos to a reconstructed image derived from the model of the GAN.

These visualizations are created by run_invert.sh.

People

As seen in the distribution statistics, thie GAN does not draw enough people. By visualizing reconstructions, we can see how: the GAN seems to avoid drawing large person figures entirely, instead synthesizing plausible scenes without people.

GAN reconstruction	Real photo

Vehicles

A similar effect is seen for vehicles.

GAN reconstruction	Real photo

Signs

GAN reconstruction	Real photo

Monuments

GAN reconstruction	Real photo

Palm trees

GAN reconstruction	Real photo