× HomeMath BoardArchiveType

A probability distribution satisfy the following probability density function/probability function as $$\ln f_X(x; \theta) = h(x) + g(\theta)+ \eta(\theta) \cdot T(x)$$

are in the $\textbf{exponential family}$.

In this section, we will focus on a specific family of distribution called linear exponential family, which satisfy the above formula with $T(x) = x$.

$\textbf{Example 1.}$ any exponential distribution is in the linear exponential family.

Recall that the pdf of exponential distribution is the following $f(x) = \frac{1}{\theta} \exp(-\frac{1}{\theta}x)$, if we take $\ln$ both sides we will have

$$\ln f(x) = \ln\left(\frac{1}{\theta}\right) + \left(-\frac{1}{\theta}\right) x$$,

which is in the linear exponential family.

$\textbf{Example 2.}$ any normal distribution is in the linear exponential family, with constant variance.

Recall that the pdf of normal distribution is the following:

$$f(x) =\frac{1}{\sigma \sqrt{2\pi}} \exp\left(-\frac{1}{2}\left(\frac{x -\mu}{\sigma}\right)^2\right)$$.

Taking $\ln$ both sides giving us the following:

$$f(x) = \ln\left(\frac{1}{\sigma \sqrt{2\pi}}\right) - \frac{1}{2}\left(\frac{x -\mu}{\sigma}\right)^2 = \ln\left(\frac{1}{\sigma \sqrt{2\pi}}\right) - \frac{1}{2}\left(\frac{x^2 +\mu^2 -2\mu x}{\sigma^2}\right)$$

$$= \ln\left(\frac{1}{\sigma \sqrt{2\pi}}\right) - \frac{x^2}{2\sigma^2} -\frac{\mu^2}{2\sigma^2} +\frac{\mu x}{\sigma^2}$$

$$= \left(\ln\left(\frac{1}{\sigma \sqrt{2\pi}}\right) - \frac{x^2}{2\sigma^2}\right) -\frac{\mu^2}{2\sigma^2} +\frac{\mu x}{\sigma^2}$$

Since $\sigma$ remains constant, we have $h(x) = \ln\left(\frac{1}{\sigma \sqrt{2\pi}}\right) - \frac{x^2}{2\sigma^2}$ and $g(x) = -\frac{x^2}{2\sigma^2}$ and $\eta(x) =\frac{x }{\sigma^2} $

$\textbf{Example 3.}$ any gamma distribution is in the linear exponential family with $k$ being constant.

Recall that the pdf of gamma distribution is the following:

$$f(x) = \frac{1}{\Gamma(k)\theta^k} x^{k-1}\exp(-x/\theta)$$

By taking $\ln$ both sides, we will have

$$\ln(f(x)) = (k-1)\ln x -\ln \left({\Gamma(k)\theta^k}\right) - \frac{x}{\theta}$$

So gamma distribution is in the linear family with $k$ being constant.

Two important properties of a distribution are the mean and variance. We will show that the mean and variance of a distribution in the linear exponential family can be expressed in terms of the first and second derivative.

Recall

$$\ln f_X(x; \theta) = h(x) + g(\theta)+ \eta(\theta) \cdot x$$

We can take derivative with respect to $\theta$ both sides and get the following:

$$\frac{\partial}{\partial \theta} f_X(x;\theta)= (g'(\theta) + \eta'(\theta) \cdot x)f_X(x;\theta)$$

Timing the infinitestimum $dx$ and integrate, we will have the following:

$$\int\frac{\partial}{\partial \theta} f_X(x;\theta) dx= \int (g'(\theta) + \eta'(\theta) \cdot x)f_X(x;\theta)dx$$

Note due to Leibniz integral rule, we will have

$$\frac{\partial}{\partial \theta} 1 = g'(\theta) + \eta'(\theta) \cdot \int xf_X(x;\theta)dx$$

and so we will have the mean of our function to be $$-\frac{g'(\theta)}{\eta'(\theta)}$$

So we have $\mathbb{E}[X] = -\frac{g'(\theta)}{\eta'(\theta)}$, which means $-\mathbb{E}[X] \eta'(\theta) = g'(\theta)$

We will reuse our formula above $\frac{\partial}{\partial \theta} f_X(x;\theta)= (g'(\theta) + \eta'(\theta) \cdot x)f_X(x;\theta)$ and first note that we can substitute $g'(\theta)$ with the above result and get

$$\frac{\partial}{\partial \theta} f_X(x;\theta)= \eta'(\theta)( x-\mathbb{E}[X])f_X(x;\theta)$$

If we take a partial derivative with respect to $\theta$, we will get the following:

$$\frac{\partial^2}{\partial \theta^2} f_X(x;\theta) = \eta''(\theta)( x-\mathbb{E}[X])f_X(x;\theta)\\ -\eta'(\theta) \mathbb{E}'[X]f_X(x;\theta)\\ + \eta'(\theta)^2( x-\mathbb{E}[X])^2f_X(x;\theta)$$

If we integrate both sides with respect to $x$, we will have

$$0 = -\eta'(\theta) \mathbb{E}'[X]\\ + \eta'(\theta)^2 Var(X)$$

So the variance is $$Var(X) = \frac{g'(\theta)}{\eta'(\theta)}$$