Figure 1.9: The asymmetry of the Kullback-Leibler divergence. These contour plots show the minimum KL divergence solution between two zero mean Gaussians. The first Gaussian, N1 has a covariance matrix with a very high correlation. Its eigenvalues are 1 and 0.01 and the eigenvectors are at forty-five degrees to the axis. In each plot three contours from the distribution are shown as solid lines. The second Gaussian, N2, is taken to be spherical and governed by a single variance parameter, σ2. The variance parameter has been optimised in both diagrams so as to minimise the KL divergence. Three contours from the resulting distributions are shown by dotted lines. In diagram (a) R N1 log N1N2 was used. Note the approximation has high probability where the true distribution has high probability. In (b) R N2 log N2N1 was used, note the approximation has low probability where the true distribution has low value.