Evaluating generative models with FID versus IS

WHAT THIS TESTS: Whether you can quantitatively assess a generative model and reason about what each metric does and does not capture.

A GOOD ANSWER COVERS: Generative models lack a single ground-truth target, so evaluation compares distributions. Frechet Inception Distance embeds both real and generated images using activations from a pretrained Inception network, typically a pooling layer. It models each set of activations as a multivariate Gaussian, estimating a mean vector and covariance matrix, then computes the Frechet, or 2-Wasserstein, distance between the two Gaussians. The formula combines the squared difference of means with a term involving both covariances. Lower FID means the generated feature distribution is closer to the real one. Inception Score instead measures only generated images: it rewards confident class predictions per image and a diverse marginal over classes, with no reference to real data. FID's advantages are that it uses real images as the reference, jointly reflects fidelity and diversity, penalizes mode dropping, and correlates better with human judgment; it is also sensitive to common corruptions like blur or noise.

COMMON WRONG ANSWERS: Saying higher FID is better; claiming IS compares against the real dataset; treating FID as bias-free when it is biased by sample count and depends on the chosen feature layer and image preprocessing; ignoring that both rely on an ImageNet-trained backbone that may not suit non-natural images.

LIKELY FOLLOW-UPS: Why is FID biased with small sample sizes? What can FID miss that precision and recall metrics catch? Why might FID be misleading for medical or satellite imagery? How do you make FID comparisons fair across papers?

ONE CONCRETE EXAMPLE: To compare two image generators, sample fifty thousand images from each and use the same number of real images. Extract Inception features, compute the two Gaussians, and evaluate FID. A model scoring around five is much closer to the real distribution than one scoring fifty. If a generator drops a class entirely, its real-data covariance mismatch raises FID, whereas Inception Score might still look acceptable because it never consulted real images.