Introduction to the Linear Exponential Family

In summary, for a distribution in the linear exponential family, the mean can be expressed as E[X]=−g′(θ)η′(θ)\mathbb{E}[X] = -\frac{g'(\theta)}{\eta'(\theta)}, and the variance can be expressed as Var(X)=g′(θ)η′(θ)Var(X) = \frac{g'(\theta)}{\eta'(\theta)}. These expressions relate the mean and variance of the distribution to the first and second derivatives of the log-likelihood function of the distribution.

Some author3 min read

A probability distribution satisfy the following probability density function/probability function as

ln⁡fX(x;θ)=h(x)+g(θ)+η(θ)⋅T(x)\ln f_X(x; \theta) = h(x) + g(\theta)+ \eta(\theta) \cdot T(x)

are in the exponential family\textbf{exponential family}.

In this section, we will focus on a specific family of distribution called linear exponential family, which satisfy the above formula with T(x)=xT(x) = x.

Example 1.\textbf{Example 1.} any exponential distribution is in the linear exponential family.

Recall that the pdf of exponential distribution is the following f(x)=1θexp⁡(−1θx)f(x) = \frac{1}{\theta} \exp(-\frac{1}{\theta}x), if we take ln⁡\ln both sides we will have

ln⁡f(x)=ln⁡(1θ)+(−1θ)x\ln f(x) = \ln\left(\frac{1}{\theta}\right) + \left(-\frac{1}{\theta}\right) x

,

which is in the linear exponential family.

Example 2.\textbf{Example 2.} any normal distribution is in the linear exponential family, with constant variance.

Recall that the pdf of normal distribution is the following:

f(x)=1σ2πexp⁡(−12(x−μσ)2)f(x) =\frac{1}{\sigma \sqrt{2\pi}} \exp\left(-\frac{1}{2}\left(\frac{x -\mu}{\sigma}\right)^2\right)

.

Taking ln⁡\ln both sides giving us the following:

f(x)=ln⁡(1σ2π)−12(x−μσ)2=ln⁡(1σ2π)−12(x2+μ2−2μxσ2)f(x) = \ln\left(\frac{1}{\sigma \sqrt{2\pi}}\right) - \frac{1}{2}\left(\frac{x -\mu}{\sigma}\right)^2 = \ln\left(\frac{1}{\sigma \sqrt{2\pi}}\right) - \frac{1}{2}\left(\frac{x^2 +\mu^2 -2\mu x}{\sigma^2}\right) =ln⁡(1σ2π)−x22σ2−μ22σ2+μxσ2= \ln\left(\frac{1}{\sigma \sqrt{2\pi}}\right) - \frac{x^2}{2\sigma^2} -\frac{\mu^2}{2\sigma^2} +\frac{\mu x}{\sigma^2} =(ln⁡(1σ2π)−x22σ2)−μ22σ2+μxσ2= \left(\ln\left(\frac{1}{\sigma \sqrt{2\pi}}\right) - \frac{x^2}{2\sigma^2}\right) -\frac{\mu^2}{2\sigma^2} +\frac{\mu x}{\sigma^2}

Since σ\sigma remains constant, we have h(x)=ln⁡(1σ2π)−x22σ2h(x) = \ln\left(\frac{1}{\sigma \sqrt{2\pi}}\right) - \frac{x^2}{2\sigma^2} and g(x)=−x22σ2g(x) = -\frac{x^2}{2\sigma^2} and η(x)=xσ2\eta(x) =\frac{x }{\sigma^2}

Example 3.\textbf{Example 3.} any gamma distribution is in the linear exponential family with kk being constant.

Recall that the pdf of gamma distribution is the following:

f(x)=1Γ(k)θkxk−1exp⁡(−x/θ)f(x) = \frac{1}{\Gamma(k)\theta^k} x^{k-1}\exp(-x/\theta)

By taking ln⁡\ln both sides, we will have

ln⁡(f(x))=(k−1)ln⁡x−ln⁡(Γ(k)θk)−xθ\ln(f(x)) = (k-1)\ln x -\ln \left({\Gamma(k)\theta^k}\right) - \frac{x}{\theta}

So gamma distribution is in the linear family with kk being constant.

Two important properties of a distribution are the mean and variance. We will show that the mean and variance of a distribution in the linear exponential family can be expressed in terms of the first and second derivative.

Recall

ln⁡fX(x;θ)=h(x)+g(θ)+η(θ)⋅x\ln f_X(x; \theta) = h(x) + g(\theta)+ \eta(\theta) \cdot x

We can take derivative with respect to θ\theta both sides and get the following:

∂∂θfX(x;θ)=(g′(θ)+η′(θ)⋅x)fX(x;θ)\frac{\partial}{\partial \theta} f_X(x;\theta)= (g'(\theta) + \eta'(\theta) \cdot x)f_X(x;\theta)

Timing the infinitestimum dxdx and integrate, we will have the following:

∫∂∂θfX(x;θ)dx=∫(g′(θ)+η′(θ)⋅x)fX(x;θ)dx\int\frac{\partial}{\partial \theta} f_X(x;\theta) dx= \int (g'(\theta) + \eta'(\theta) \cdot x)f_X(x;\theta)dx

Note due to Leibniz integral rule, we will have

∂∂θ1=g′(θ)+η′(θ)⋅∫xfX(x;θ)dx\frac{\partial}{\partial \theta} 1 = g'(\theta) + \eta'(\theta) \cdot \int xf_X(x;\theta)dx

and so we will have the mean of our function to be

−g′(θ)η′(θ)-\frac{g'(\theta)}{\eta'(\theta)}

So we have E[X]=−g′(θ)η′(θ)\mathbb{E}[X] = -\frac{g'(\theta)}{\eta'(\theta)}, which means −E[X]η′(θ)=g′(θ)-\mathbb{E}[X] \eta'(\theta) = g'(\theta)

We will reuse our formula above ∂∂θfX(x;θ)=(g′(θ)+η′(θ)⋅x)fX(x;θ)\frac{\partial}{\partial \theta} f_X(x;\theta)= (g'(\theta) + \eta'(\theta) \cdot x)f_X(x;\theta) and first note that we can substitute g′(θ)g'(\theta) with the above result and get

∂∂θfX(x;θ)=η′(θ)(x−E[X])fX(x;θ)\frac{\partial}{\partial \theta} f_X(x;\theta)= \eta'(\theta)( x-\mathbb{E}[X])f_X(x;\theta)

If we take a partial derivative with respect to θ\theta, we will get the following:

∂2∂θ2fX(x;θ)=η′′(θ)(x−E[X])fX(x;θ)−η′(θ)E′[X]fX(x;θ)+η′(θ)2(x−E[X])2fX(x;θ)\frac{\partial^2}{\partial \theta^2} f_X(x;\theta) = \eta''(\theta)( x-\mathbb{E}[X])f_X(x;\theta)\\ -\eta'(\theta) \mathbb{E}'[X]f_X(x;\theta)\\ + \eta'(\theta)^2( x-\mathbb{E}[X])^2f_X(x;\theta)

If we integrate both sides with respect to xx, we will have

0=−η′(θ)E′[X]+η′(θ)2Var(X)0 = -\eta'(\theta) \mathbb{E}'[X]\\ + \eta'(\theta)^2 Var(X)

So the variance is

Var(X)=g′(θ)η′(θ)Var(X) = \frac{g'(\theta)}{\eta'(\theta)}