Information theory
Consider a random variable representing cointhrows (Bernoulli Variable with Σ = {0,1} ). Let the true
probability distribution be p(0) = r, p(1) = 1-r.
Someone guesses a different distribution q(0) = s, q(1) =1-s.
(a) Find expressions for the Kullback–Leibler distances D(p||q) andD(q||p) between the
two distributions in terms of r and s.
(b) Show that in general, D(p||q) ≠D(q||p) and that equalityoccurs iff r = s.
(c) Compute D(p||q) and D(q||p) for the case r = 1/2 and s =1/4.
