Nad Am

Linear Regression Error Decomposition

A short derivation of the error decomposition of OLS

Some author1 min read
yi−yˉ=yi−yi^+yi^−yˉy_i - \bar{y} = y_i - \hat{y_i} + \hat{y_i} -\bar{y} ∑(yi−yˉ)2=∑(yi−yi^)2+∑(yi^−yˉ)2+2∑(yi−yi^)(yi^−yˉ)\sum (y_i-\bar{y})^2 = \sum (y_i - \hat{y_i})^2 + \sum (\hat{y_i} -\bar{y})^2 + 2\sum( y_i - \hat{y_i})(\hat{y_i} -\bar{y}) (yi−yi^)(yi^−yˉ)=(yi−b0−b1xi)(b0+b1xi−yˉ)=(yi−yˉ+xˉb1−b1xi)(yˉ−xˉb1+b1xi−yˉ)( y_i - \hat{y_i})(\hat{y_i} -\bar{y}) = (y_i - b_0 -b_1 x_i) (b_0 +b_1 x_i -\bar{y} ) =(y_i - \bar{y} + \bar{x} b_1 -b_1 x_i) (\bar{y} - \bar{x} b_1 +b_1 x_i -\bar{y} ) =(yi−yˉ+(xˉ−xi)b1)(b1xi−xˉb1)= (y_i - \bar{y} + (\bar{x} - x_i)b_1) ( b_1 x_i- \bar{x} b_1 ) =(yi−yˉ−(xi−xˉ)b1)b1(xi−xˉ)= (y_i - \bar{y} -(x_i -\bar{x}) b_1)b_1 (x_i -\bar{x}) =b1(yi−yˉ)(xi−xˉ)−b12(xi−xˉ)2= b_1(y_i - \bar{y})(x_i -\bar{x}) - b_1^2 (x_i -\bar{x})^2

We know that b1=Cov(X,Y)Var(X)b_1 = \frac{Cov(X,Y)}{Var(X)}

so we will have

∑(yi−yi^)(yi^−yˉ)=Cov(X,Y)Var(X)Cov(X,Y)−(Cov(X,Y)Var(X))2Var(X)=0\sum ( y_i - \hat{y_i})(\hat{y_i} -\bar{y}) = \frac{Cov(X,Y)}{Var(X)} Cov(X,Y) - \left(\frac{Cov(X,Y)}{Var(X)}\right)^2 Var(X) = 0

We call ∑(yi−yˉ)2\sum(y_i - \bar{y})^2 total variation without the knowledge of xx and ∑(yi−yi^)2\sum(y_i - \hat{y_i})^2 total variation remaining with the knowledge of xx and ∑(yi^−yˉ)2\sum (\hat{y_i} -\bar{y})^2 total variation explained with the knowledge of xx