[TOC]

0.1. 多変量正規分布のKLダイバージェンス

2つの$d$次元多変量正規分布$p\left(x\right)=N(x;{\mu }_1,{\Sigma }_1),q\left(x\right)=N(x;{\mu }_2,{\Sigma }_2)$のKLダイバージェンス$KL\left[P\right(X\left)||Q\right(X\left)\right]$は以下である.

$KL\left[P\right(X\left)||Q\right(X\left)\right]=\frac{1}{2}(\log \frac{|{\Sigma }_2|}{|{\Sigma }_1|}+Tr[{\Sigma }_2^{-1}{\Sigma }_1]+({\mu }_1-{\mu }_2{)}^T{\Sigma }_2^{-1}({\mu }_1-{\mu }_2)-d)$

参考資料

平均$\mu$,共分散行列$\Sigma$の$d$次元多変量正規分布は以下で与えられる. $$N(x;\mu ,\Sigma )\equiv \frac{|\Sigma {|}^{-1/2}}{(2\pi {)}^{d/2}}\exp (-\frac{1}{2}(x-\mu {)}^T\Sigma (x-\mu ))$$ ここで,$p\left(x\right)=N(x;{\mu }_1,{\Sigma }_1),q\left(x\right)=N(x;{\mu }_2,{\Sigma }_2)$の間のKL情報量 $$KL\left[P\right(X\left)||Q\right(X\left)\right]\equiv \int p\left(x\right)\log \frac{p\left(x\right)}{q\left(x\right)}dx$$ を計算する.まず,$\log \frac{p\left(x\right)}{q\left(x\right)}$を整理する. $$\begin{array}{cc}\log \frac{p\left(x\right)}{q\left(x\right)}& =\log p\left(x\right)-\log q\left(x\right)\\ & =\frac{1}{2}(\log \frac{|{\Sigma }_2|}{|{\Sigma }_1|}+(x-{\mu }_2{)}^T{\Sigma }_2^{-1}(x-{\mu }_2)-(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1\left)\right)\end{array}$$ よって, $$KL\left[P\right(X\left)||Q\right(X\left)\right]=\frac{1}{2}(\log \frac{|{\Sigma }_2|}{|{\Sigma }_1|}+E_p\left[\right(x-{\mu }_2{)}^T{\Sigma }_2^{-1}(x-{\mu }_2)]-E_p\left[\right(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)])$$ となる.

また,$K$を体($R$),$x\in K^n,A\in K^{n\times n}$とすると簡単な計算からトレースについて$x^{T}Ax=Tr\left[x^{T}Ax\right]=Tr\left[Ax{x}^T\right]$が一般に成立することに注意すると, $$\begin{array}{cc}{E}_p\left[\right(x-{\mu }_2{)}^T{\Sigma }_2^{-1}(x-{\mu }_2)]& =E_p\left[Tr\right[{\Sigma }_2^{-1}(x-{\mu }_2)(x-{\mu }_2{)}^T]]\\ & =Tr\left[E_p\left[{\Sigma }_2^{-1}\right(x-{\mu }_2\left)\right(x-{\mu }_2{)}^T]\right]\\ & =Tr\left[{\Sigma }_2^{-1}\left(E_p\right[x{x}^T]-E_p[x{\mu }_2^T]-E_p[{\mu }_2{x}^T]+E_p[{\mu }_2{\mu }_2^T\left]\right)\right]\end{array}$$ また,平均の定義より$E_p\left[x\right]={\mu }_1$であるので,$E_p\left[x{\mu }_2^T\right]={\mu }_1{\mu }_2^T,E_p\left[{\mu }_2{x}^T\right]={\mu }_2{\mu }_1^T$である.

また,分散の定義より $$\begin{array}{cc}{\Sigma }_1& \equiv E_p\left[\right(x-{\mu }_1\left)\right(x-{\mu }_1{)}^T]\\ & =E_p\left[x{x}^T\right]-E_p\left[x{\mu }_1^T\right]-E_p\left[{\mu }_1{x}^T\right]+E_p\left[{\mu }_1{\mu }_1^T\right]\\ & =E_p\left[x{x}^T\right]-{\mu }_1{\mu }_1^T\end{array}$$ であるので, $$\begin{array}{cc}{E}_p\left[\right(x-{\mu }_2{)}^T{\Sigma }_2^{-1}(x-{\mu }_2)]& =Tr\left[{\Sigma }_2^{-1}\left(E_p\right[x{x}^T]-E_p[x{\mu }_2^T]-E_p[{\mu }_2{x}^T]+E_p[{\mu }_2{\mu }_2^T\left]\right)\right]\\ & =Tr\left[{\Sigma }_2^{-1}({\Sigma }_1+{\mu }_1{\mu }_1^T-{\mu }_1{\mu }_2^T-{\mu }_2{\mu }_1^T+{\mu }_2{\mu }_2^T)\right]\\ & =Tr\left[{\Sigma }_2^{-1}({\Sigma }_1+({\mu }_1-{\mu }_2\left)\right({\mu }_1-{\mu }_2{)}^T)\right]\\ & =Tr\left[{\Sigma }_2^{-1}{\Sigma }_1\right]+({\mu }_1-{\mu }_2{)}^T{\Sigma }_2^{-1}({\mu }_1-{\mu }_2)\end{array}$$ また第3項についても同様に計算を行うと $$\begin{array}{cc}{E}_p\left[\right(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)]& =E_p\left[Tr\right[{\Sigma }_1^{-1}(x-{\mu }_1)(x-{\mu }_1{)}^T]]\\ & =Tr\left[{\Sigma }_1^{-1}{E}_p\left[\right(x-{\mu }_1\left)\right(x-{\mu }_1{)}^T]\right]\\ & =Tr\left[{\Sigma }_1^{-1}{\Sigma }_1\right]\\ & =d\end{array}$$ ゆえに, $$KL\left[P\right(X\left)||Q\right(X\left)\right]=\frac{1}{2}(\log \frac{|{\Sigma }_2|}{|{\Sigma }_1|}+Tr[{\Sigma }_2^{-1}{\Sigma }_1]+({\mu }_1-{\mu }_2{)}^T{\Sigma }_2^{-1}({\mu }_1-{\mu }_2)-d)$$

Let $S$ is a $n$-by-$n$ symmetric matrix and $U$ is a $n$-by-$n$ orthogonal matrix.

Then $det\left(S\right)=det\left(US{U}^T\right)$

(proof)

Since $det\left(S\right)$ is the products of $n$ eigenvalues of $S$, it is necessary to prove that all eigenvalues of $S$ and $US{U}^T$ are same.

As $S$ is a symmetric matrix, $US{U}^T$ is also symmetric matrix. Hence, there exists a orthogonal matrix $Q$ such that $QUS{U}^T{Q}^T=QUS(QU{)}^T$ is the diagonal matrix whose diagonal entries are the eigenvalues of $US{U}^T$ and $S$.