Linear regression

Definition

Let $n \ge p \ge 1$. Let $X$ be a known $n \times p$ real matrix, the design matrix, whose $i$-th row $x_i^T$ holds the explanatory variables of unit $i$. The linear regression model for a response vector $y = (y_1, \ldots, y_n)^T$ is $$y = X \beta + \varepsilon, \qquad \text{that is,} \qquad y_i = x_i^T \beta + \varepsilon_i,$$ where:

1. $\beta \in \R^p$ is an unknown vector of coefficients;
2. $\varepsilon = (\varepsilon_1, \ldots, \varepsilon_n)^T$ is a random error vector with mean $\mathbb{E}\varepsilon = 0$;
3. the errors are uncorrelated with a common unknown variance $\sigma^2 > 0$, that is, $\operatorname{Cov}(\varepsilon) = \sigma^2 I_n$.

Usually $X$ is assumed to have full column rank $p$, and often the errors are assumed normal.