× HomeMath BoardArchiveType

Couple exercises from Sampling Distributions by Xiaojing Ye (Department of Mathematics & Statistics, Georgia State University) (p1-15)

Let's consider a number of i.i.d random variables $X_1, X_2,...,X_n$, then we want to estimate some property underlying with the distribution. This estimate is called statistics. For example the mean and the sample variance are two statistics for random variables.

We can define them as

$$\bar{X} = \frac{1}{n} \sum_i X_i$$

$$\sigma(X) = \frac{1}{n-1} \sum_{i}X_i$$

Sampling distribution of the mean

Theorem. If $X_1, \ldots, X_n$ is a random sample of a distribution with mean $\mu$ and variance $\sigma^2$, then the sample mean $\bar{X}$ satisfies

$$ \mathbb{E}[\bar{X}]=\mu, \quad \operatorname{var}[\bar{X}]=\frac{\sigma^2}{n} $$

Proof.

$$\mathbb{E}[\bar{X}]= \mathbb{E}\left[\frac{1}{n} \sum_i X_i\right] = \frac{1}{n} \mathbb{E}\left[ \sum_i X_i\right]=\frac{1}{n} \sum_i\mathbb{E}\left[ X_i\right]$$

Since $X_1, X_2, ... , X_n$ are i.i.d, we will have

$$\mathbb{E}[X_i] = \mu$$ and then $\mathbb{E}[\bar{X}] = \mu.$

$$\operatorname{var}\left[\bar{X}\right] = \operatorname{var}\left[\frac{1}{n}\sum_i X_i\right]=\frac{1}{n^2} \operatorname{var}\left[\sum_i X_i\right]= \frac{1}{n^2} \sum_i\operatorname{var}\left[ X_i\right]$$

Since $\operatorname{var}[X_i] = \sigma$, then we will have $\operatorname{var}\left[\bar{X}\right] =\frac{\sigma}{n}. $

We can denote $\mathbb{E}[\bar{X}] = \mu_{\bar{X}}$ and $\operatorname{var}[X] = \sigma_{\bar{X}}^2$ and $\sigma_{\bar{x}}$ is called the sample error.

Theorem (Chebyshev's inequality). Let $X$ be a RV with mean $\mu$ and variance $\sigma^2$, then for any $c>0$, there is

$$\mathrm{P}(|X-\mu| \geq c) \leq \frac{\sigma^2}{c^2}$$

Proof.

$$\mathrm{P}(|X-\mu| \geq c) = \int_{|x-\mu| \ge c} f(x) \, dx \le \int_{|x-\mu| \ge c} \frac{|x-\mu|^2}{c^2} f(x)\, dx = \frac{1}{c^2}\int_{|x-\mu| \ge c} |x-\mu|^2 f(x)\, dx \le \frac{1}{c^2}\int |x-\mu|^2 f(x) \, dx = \frac{\sigma^2}{c^2}.$$

Example. By Chebyshev's inequality, we have for any fixed $c>0$ that

$$\mathrm{P}(|\bar{X}-\mu| \ge c \sigma) \le \frac{1}{n c^2} .$$

$$\mathrm{P}(|\bar{X}-\mu| \le c \sigma) \ge 1- \frac{1}{n c^2}$$

as $n$ goes to infinity, we will have $\mathrm{P}(|\bar{X}-\mu| \le c \sigma) = 1$, which is weak law of large number.

Central Limit Theorem. Let $X_1, \ldots, X_n$ be a random sample from a distribution with mean $\mu$ and variance $\sigma^2$. Denote $\bar{X}_n$ their sample mean. Define

$$Z_n=\frac{\bar{X}_n-\mu}{\sigma / \sqrt{n}}$$

Then the limiting distribution of $Z_n$ as $n \rightarrow \infty$ is the standard normal distribution.

We know the following properties of moment generating function:

Let $M_X(t)$ be moment generating function and $a,b$ are constants.

$$M_{X+a}(t) = \mathbb{E}[\exp((X+a)t)] = \exp(at) \mathbb{E}[\exp(Xt)] = \exp(at) M_{X}(t)$$

$$M_{bX}(t) = \mathbb{E}[\exp(bXt)] = \mathbb{E}[\exp(Xbt)]=M_X(bt)$$

$$M_{\frac{X+a}{b}}(t) = \exp\left(\frac{a}{b}\right) M_{\frac{X}{b}}(t) = \exp\left(\frac{a}{b}\right) M_{X}\left(\frac{t}{b}\right)$$

Proof of CLT.

Let's consider $M_{Z_n}(t) = M_{\frac{\bar{X_n}-\mu}{\sigma/\sqrt{n}}}(t) = \exp\left(-\frac{\mu\sqrt{n}}{\sigma}t\right) M_{\bar{X_n}}\left(\frac{t \sqrt{n}}{\sigma }\right)$

$$M_{Z_n}(t) = \exp\left(-\frac{\mu\sqrt{n}}{\sigma}t\right) M_{n\bar{X_n}}\left(\frac{t }{\sigma \sqrt{n}}\right)$$

Since $n\bar{X_n} = X_1 + X_2+...+X_n$, we will have

$$M_{Z_n}(t) = \exp\left(-\frac{\mu\sqrt{n}}{\sigma}t\right) \prod_{i} M_{X_i}\left(\frac{t }{\sigma \sqrt{n}}\right) = \exp\left(-\frac{\mu\sqrt{n}}{\sigma}t\right) \left(M_{X}\left(\frac{t}{\sigma \sqrt{n}}\right)\right)^n$$

Since we know that $M_X(t) = 1+ m_X(1) t + \frac{m_X(2)}{2} t^2 + .... $, where $m_X(n)$ is the n-th moment of $X$. then we will have

$$M_X\left(\frac{t}{\sigma \sqrt{n}}\right) = 1 +m_X(1) \frac{t}{\sigma \sqrt{n}} + \frac{m_X(2)}{2}\left(\frac{t}{\sigma \sqrt{n}}\right)^2 +...$$

We will denote $$h(t) = m_X(1) \frac{t}{\sigma \sqrt{n}} + \frac{m_X(2)}{2}\left(\frac{t}{\sigma \sqrt{n}}\right)^2 +...$$

Then if we take $\ln$ on $M_{Z_n}(t)$, we will have

$$\ln M_{Z_n}(t) = -\frac{\mu\sqrt{n}}{\sigma}t + n \ln M_X\left(\frac{t}{\sigma \sqrt{n}}\right) = -\frac{\mu\sqrt{n}}{\sigma}t + n \ln (1+h(t)) = -\frac{\mu\sqrt{n}}{\sigma}t + n \times (h(t)-(h(t))^2/2+(h(t))^3 / 6 ....)$$

$$=\frac{t^2}{2} + O\left(\frac{1}{n^{1/2}}\right)$$

Therefore, we have $\ln M_{Z_n}(t) = \frac{t^2}{2}$ as $n$ goes to infinity, which coincides with the guassian distribution's moment generating function.

Sample distribution with finite population and without replacement

Suppose that we have a population of $N$, we want to draw $n$ sample out of our $N$ population and also, the sample is ordered. Then the probability of drawing $n$ sample with a fixed configuration is equal to $\frac{1}{P_{N}^n} = \frac{1}{\frac{N!}{(N-n)!}} = \frac{(N-n)!}{N!}$

If we want to know the probability of having one specific sample in our drawing where its position is fixed, then we will have $$\frac{P_{N-1}^{n-1}}{P_{N}^{n}} = \frac{\frac{(N-1)!}{(N-n)!}}{\frac{N!}{(N-n)!}} = \frac{1}{N}$$

Let's denote the position as $r$ and our selected element as $x$, then we will have the following pmf

$$f_r(x) =\frac{1}{N}$$

where $x$ can take any value from the population and $r$ can take any value from $1$ to $n$.

$$\mu_r = \sum_{i}x_i f_r(x_i) = \frac{1}{N} \sum_i x_i := \mu$$

$$\sigma^2_r = \mathbb{E}[(X_r-\mu_r)^2] = \sum_i (x_i -\mu_r)^2f_r(x_i) = \frac{1}{N} \sum_i(x_i - \mu_r)^2 := \sigma^2$$

For joint pmf of $(x_r, x_s)$ for position $r,s$, we will have

$$g_{rs}(x_r, x_s) = \frac{P_{N-2}^{n-2}}{P_{N}^n} = \frac{\frac{(N-2)!}{(N-n)!}}{\frac{N!}{(N-n)!}} = \frac{1}{N(N-1)}$$

Then the covariance of $(x_r,x_s)$ will be

$$\operatorname{Cov}(x_r, x_s) = \mathbb{E}[(X_r -\mu)(X_s -\mu)]$$

$$= \sum_{r\not =s} (x_r - \mu) (x_s -\mu) g_{rs}(x_r, x_s)$$

$$= \frac{1}{N(N-1)}\sum_{r\not =s} (x_r - \mu) (x_s -\mu) $$

$$= \frac{1}{N(N-1)}\sum_{r=1}^N (x_r - \mu) \sum_{s=1, s\not = r}^N (x_s -\mu)$$

Since we have $\sum_{s=1}^N x_s = N\mu = \sum_{s=1}^N \mu$, then we will have

$$\sum_{s=1, s\not = r}^N (x_s -\mu) = -(x_r - \mu)$$

So, we will have

$$\operatorname{Cov}(x_r, x_s) = -\frac{1}{N(N-1)}\sum_{r=1}^N (x_r - \mu)^2 = -\frac{\sigma^2}{N-1}$$

Now let's find the mean and variance of $\bar{X_n} =\frac{1}{n} \sum_{i=1}^n X_i $,

$$\mathbb{E}[\bar{X_n}] = \frac{1}{n} \sum_{i=1}^n \mathbb{E}[X_i] = \mu$$.

$$\operatorname{Var}[\bar{X_n}] = \frac{1}{n^2}\operatorname{Var}[\sum_{i=1}^n X_i]$$

$$= \frac{1}{n^2}\left(\sum \operatorname{Var}[X_i]+\sum_{i \not=j} \operatorname{Cov}(X_i, X_j)\right)$$

$$= \frac{1}{n^2}\left(n \sigma^2-(n(n-1)) \frac{\sigma^2}{N-1}\right)$$

$$= \frac{\sigma^2}{n} (1-\frac{n-1}{N-1})$$

$$= \frac{\sigma^2}{n} \frac{N-n}{N-1}$$