Logistic regression

Remarks

It is the generalized linear model with Bernoulli responses and the logit link $g(\mu) = \log(\mu / (1 - \mu))$, which is the canonical link. The log-likelihood is $\ell(\beta) = \sum_i \big( y_i x_i^T \beta - \log(1 + e^{x_i^T \beta}) \big)$, a concave function. The maximum likelihood estimate solves $\sum_i (y_i - p_i) x_i = 0$ and has no closed formula in general. For a regressor $x_{ij}$, the number $e^{\beta_j}$ is the odds ratio for a unit increase of $x_{ij}$ with the other regressors fixed.

Example: with $x_i = (1, t_i)$ and $\beta = (-2, 0.5)$, at $t = 4$ the log-odds is $0$, so $p = \sigma(0) = 1/2$; at $t = 6$ it is $1$, so $p = e/(1 + e) \approx 0.73$.

Non-example: suppose the data are perfectly separated, that is, some $\gamma \in \R^p$ has $x_i^T \gamma > 0$ whenever $y_i = 1$ and $x_i^T \gamma < 0$ whenever $y_i = 0$. Then $\ell(c\gamma)$ increases to $0$ as $c \to \infty$, while $\ell < 0$ everywhere, so the maximum likelihood estimate does not exist. The model is still defined, but this fitting method fails.