# Measuring Difference in Probability Distributions

## Kullback-Leibler (KL) divergence&#x20;

$$
\begin{equation\*}
D\_{KL}(P||Q)=\sum\_{s\in S}p\_p(s)\log\left(\frac{p\_p(s)}{p\_q(s)}\right) = E\_{s\sim P}\left\[\log\frac{p(x)}{q(x)}\right]
\end{equation\*}
$$

Key properties&#x20;

* non-symmetric&#x20;
* $$D\_{KL}(P||Q)\geq 0$$, and only is 0 when two distributions are the same&#x20;
