- [[linear regression closed-form matrix solution]] ## Idea The variance of $\hat{\beta}$ is given by the following (see also [[unbiased estimator]]): $ \widehat{\operatorname{Var}}(\hat{\beta})=\hat{\sigma}^{2}\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} $ Which is equivalent to the following: $\frac{\text{residual error}}{\text{covariance of x}}$ Or the following: $ \begin{aligned} &\hat{\sigma}^{2}=\frac{1}{n-k} \sum\left(y_{i}-\hat{y}_{i}\right)^{2} \\ &\operatorname{Var}\left(\hat{\beta}\right)=\frac{\sigma^{2}}{\sum\left(x_{i}-\bar{x}\right)^{2}} \end{aligned} $ Note that the closed form solution for variance of $\hat{\beta}$ does not depend on the value of $\beta$. ## Implications ### Reduce SE of estimates (make p-values smaller) Given the formula above, there are two ways to reduce the variance/SE (i.e., increased statistical significance) associated with an estimate: - reduce residual error (numerator), $\sigma^2$ - i.e., reduce the variance in the outcome $y$ - include covariates that explain variance in the outcome $y$ to reduce unexplained variance - increase variance of the covariate $x$ (denominator) - e.g., increase the variance in the treatment It can be derived from (note also the [[calculate coefficient standard errors of linear models|sandwich formula]]) $ \operatorname{Var}(\hat{\beta})=\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \sigma^{2} \mathbf{I} \mathbf{X}\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} $ because $ \operatorname{Var}(A X)=A \times \operatorname{Var}(X) \times A^{\prime} $ $ \begin{gathered} V[\hat{\beta}]=V\left[\left(X^{T} X\right)^{-1} X^{T} Y\right] \\ =\left(X^{T} X\right)^{-1} X^{T} V[Y] X\left(X^{T} X\right)^{-1} \\ =\sigma^{2}\left(X^{T} X\right)^{-1} \end{gathered} $ # References - [The Mean and Variance of Estimated Regression Parameters in a Full Rank Gauss-Markov Linear Model - YouTube](https://www.youtube.com/watch?v=jyBtfhQsf44) - https://online.stat.psu.edu/stat462/node/132/ - [r - How are the standard errors of coefficients calculated in a regression? - Cross Validated](https://stats.stackexchange.com/questions/44838/how-are-the-standard-errors-of-coefficients-calculated-in-a-regression/44841#44841) - [How calculate variance-covariance matrix of coefficients for multivariate (multiple) linear regression? - Cross Validated](https://stats.stackexchange.com/questions/467306/how-calculate-variance-covariance-matrix-of-coefficients-for-multivariate-multi)