In summary, for a distribution in the linear exponential family, the mean can be expressed as E[X]=−η′(θ)g′(θ), and the variance can be expressed as Var(X)=η′(θ)g′(θ). These expressions relate the mean and variance of the distribution to the first and second derivatives of the log-likelihood function of the distribution.
Some author··3 min read
A probability distribution satisfy the following probability density function/probability function as
lnfX(x;θ)=h(x)+g(θ)+η(θ)⋅T(x)
are in the exponential family.
In this section, we will focus on a specific family of distribution called linear exponential family, which satisfy the above formula with T(x)=x.
Example 1. any exponential distribution is in the linear exponential family.
Recall that the pdf of exponential distribution is the following f(x)=θ1exp(−θ1x), if we take ln both sides we will have
lnf(x)=ln(θ1)+(−θ1)x
,
which is in the linear exponential family.
Example 2. any normal distribution is in the linear exponential family, with constant variance.
Recall that the pdf of normal distribution is the following:
Since σ remains constant, we have h(x)=ln(σ2π1)−2σ2x2 and g(x)=−2σ2x2 and η(x)=σ2x
Example 3. any gamma distribution is in the linear exponential family with k being constant.
Recall that the pdf of gamma distribution is the following:
f(x)=Γ(k)θk1xk−1exp(−x/θ)
By taking ln both sides, we will have
ln(f(x))=(k−1)lnx−ln(Γ(k)θk)−θx
So gamma distribution is in the linear family with k being constant.
Two important properties of a distribution are the mean and variance. We will show that the mean and variance of a distribution in the linear exponential family can be expressed in terms of the first and second derivative.
Recall
lnfX(x;θ)=h(x)+g(θ)+η(θ)⋅x
We can take derivative with respect to θ both sides and get the following:
∂θ∂fX(x;θ)=(g′(θ)+η′(θ)⋅x)fX(x;θ)
Timing the infinitestimum dx and integrate, we will have the following:
∫∂θ∂fX(x;θ)dx=∫(g′(θ)+η′(θ)⋅x)fX(x;θ)dx
Note due to Leibniz integral rule, we will have
∂θ∂1=g′(θ)+η′(θ)⋅∫xfX(x;θ)dx
and so we will have the mean of our function to be
−η′(θ)g′(θ)
So we have E[X]=−η′(θ)g′(θ), which means −E[X]η′(θ)=g′(θ)
We will reuse our formula above ∂θ∂fX(x;θ)=(g′(θ)+η′(θ)⋅x)fX(x;θ) and first note that we can substitute g′(θ) with the above result and get
∂θ∂fX(x;θ)=η′(θ)(x−E[X])fX(x;θ)
If we take a partial derivative with respect to θ, we will get the following: