Lasso

Remarks

The objective is convex and tends to infinity as $\norm{b}_1 \to \infty$, so a minimiser exists. It need not be unique when the columns of $X$ are linearly dependent. There is no closed formula in general; the estimate is computed by coordinate descent or by the LARS algorithm. Authors scale the objective in different ways, for example with $\frac{1}{2n}$ in front of the squared norm, which only rescales $\lambda$.

Example: assume $X^T X = I_p$ and put $z = X^T y$, the ordinary least squares estimate. Then $\norm{y - Xb}_2^2 = \norm{z - b}_2^2 + \text{const}$, and the problem splits into the $p$ problems of minimising $\frac{1}{2}(z_j - b_j)^2 + \lambda \abs{b_j}$. The solution is the soft thresholding $\hat\beta_j = \sgn(z_j) \max(\abs{z_j} - \lambda, 0)$. So every $j$ with $\abs{z_j} \le \lambda$ gets $\hat\beta_j = 0$.

Non-example: with the squared norm $\norm{b}_2^2$ in place of $\norm{b}_1$, the same orthonormal case gives $\hat\beta_j = z_j / (1 + 2\lambda)$, which is never $0$ when $z_j \ne 0$; that is ridge regression, not the lasso.

Equivalent form: for each $\lambda > 0$ and each lasso estimate $\hat\beta$, the vector $\hat\beta$ minimises $\norm{y - Xb}_2^2$ under the constraint $\norm{b}_1 \le \norm{\hat\beta}_1$.