LASSO and Regularised Regression

LASSO (Least Absolute Shrinkage and Selection Operator) is a regression method that adds a penalty to the fitting criterion, forcing coefficients to shrink toward zero. Its defining property is that it sets many coefficients to exactly zero, effectively selecting a sparse subset of predictors. This makes it particularly useful when you have many predictors relative to the number of observations, or when you want automatic variable selection alongside prediction.

LASSO is already used in this course for binary classification in Logistic Regression: Going Further and the microbiome classification example. This page covers the underlying concepts that apply regardless of outcome type.

Why Regularisation?

Standard regression minimises the residual sum of squares (RSS). When predictors are many or correlated, the solution overfits: coefficients become large and unstable, fitting noise in the training data rather than the underlying signal.

Regularisation adds a penalty term to the objective function that discourages large coefficients:

$\text{Minimise} \quad RSS + \lambda \cdot \text{Penalty}$

The hyperparameter $\lambda$ controls the strength of the penalty. When $\lambda = 0$ the solution is identical to standard regression. As $\lambda$ increases, coefficients shrink and the model becomes simpler.

The L1 and L2 Penalties

The penalty term distinguishes the three main regularised regression methods:

LASSO (L1 penalty): penalises the sum of absolute values of coefficients.

$\text{Penalty}_{L1} = \sum_{j=1}^{p} |\beta_j|$

Ridge (L2 penalty): penalises the sum of squared coefficients.

$\text{Penalty}_{L2} = \sum_{j=1}^{p} \beta_j^2$

Elastic net: a weighted combination of L1 and L2.

$\text{Penalty}_{EN} = \alpha \sum_{j=1}^{p} |\beta_j| + (1 - \alpha) \sum_{j=1}^{p} \beta_j^2$

where $\alpha \in [0, 1]$ is a second hyperparameter: $\alpha = 1$ gives LASSO, $\alpha = 0$ gives ridge.

Why LASSO produces sparsity

The geometry of the L1 penalty explains why LASSO sets coefficients to exactly zero while ridge does not. The L1 constraint region is a diamond (in two dimensions) with corners on the axes. The RSS contours tend to touch the constraint region at a corner, where one or more coefficients are exactly zero. The L2 constraint region is a sphere with no corners, so the RSS contours touch it at a point where all coefficients are small but rarely zero.

In practice: use LASSO when you believe only a few predictors matter and want variable selection. Use ridge when you believe many predictors each contribute a small effect and you want to keep them all but shrink them. Use elastic net when predictors are correlated: LASSO tends to arbitrarily select one from a group of correlated variables, while elastic net spreads the weight across the group.

The Regularisation Path

As $\lambda$ increases from zero, coefficients shrink and eventually reach zero. Plotting all coefficients against $\lambda$ produces a regularisation path: a visual summary of which variables are selected at each level of regularisation.

library(glmnet)

# Example: predict continuous outcome (linear regression)
data(mtcars)
X <- as.matrix(mtcars[, -1])   # predictors
y <- mtcars$mpg                 # continuous outcome

# Fit LASSO path (alpha = 1)
lasso_path <- glmnet(X, y, alpha = 1)

# Plot regularisation path
plot(lasso_path, xvar = "lambda", label = TRUE)
# Each line is one predictor
# Left: small lambda, all predictors included
# Right: large lambda, only most important predictors survive

Reading the plot: on the left, $\lambda$ is small and all predictors have non-zero coefficients. Moving right, $\lambda$ increases, coefficients shrink, and lines hit zero one by one. The numbers at the top show how many predictors remain non-zero. The last predictor to reach zero is the most robustly associated with the outcome.

Choosing Lambda

The regularisation path shows all possible solutions. Cross-validation selects the best $\lambda$ for prediction:

set.seed(123)
cv_lasso <- cv.glmnet(X, y, alpha = 1, nfolds = 10)

plot(cv_lasso)
# x-axis: log(lambda)
# y-axis: cross-validated mean squared error (with standard error bands)
# Left dashed line: lambda.min
# Right dashed line: lambda.1se

Two standard choices:

lambda.min: the value of $\lambda$ with the lowest cross-validated error. Gives the most accurate predictions but includes more predictors.

lambda.1se: the largest $\lambda$ whose error is within one standard error of the minimum. Gives a sparser, simpler model that sacrifices little predictive accuracy. Preferred when interpretability matters.

# Coefficients at each choice
coef(cv_lasso, s = "lambda.min")
coef(cv_lasso, s = "lambda.1se")

# Predictions
predict(cv_lasso, newx = X, s = "lambda.min")

LASSO for Linear vs Logistic Regression

The glmnet package handles both via the family argument:

# Continuous outcome (linear regression)
cv_lasso_lin <- cv.glmnet(X, y,
                           family = "gaussian",
                           alpha  = 1)

# Binary outcome (logistic regression)
cv_lasso_log <- cv.glmnet(X, y_binary,
                           family = "binomial",
                           alpha  = 1)

# Multi-class outcome
cv_lasso_multi <- cv.glmnet(X, y_factor,
                             family = "multinomial",
                             alpha  = 1)

The penalty and lambda selection work identically regardless of outcome type. Only the loss function changes.

Tuning Alpha in Elastic Net

When predictors are correlated, LASSO selects one from each correlated group somewhat arbitrarily. Elastic net distributes weight across correlated predictors more evenly. The alpha hyperparameter controls the mix and can be tuned with cross-validation:

library(caret)

set.seed(123)
cv_enet <- train(
  mpg ~ .,
  data      = mtcars,
  method    = "glmnet",
  trControl = trainControl(method = "cv", number = 10),
  tuneGrid  = expand.grid(
    alpha  = c(0, 0.25, 0.5, 0.75, 1),   # 0 = ridge, 1 = LASSO
    lambda = 10^seq(-3, 1, length = 20)))

print(cv_enet$bestTune)
# Best combination of alpha and lambda

plot(cv_enet)

Common Pitfalls

Not scaling predictors. LASSO penalises all coefficients equally on the scale they are measured. Variables with large units will be penalised more than variables with small units unless the data are scaled first. glmnet standardises predictors internally by default (standardize = TRUE), so the coefficients it returns are on the original scale. This is usually what you want, but be aware it is happening.

Reporting lambda.min as if it were validated. The cross-validated error at lambda.min is itself slightly optimistic because the same data were used to choose $\lambda$ . For a fully honest estimate, use nested cross-validation or a held-out test set.

Interpreting LASSO coefficients as causal. Like all regression coefficients, LASSO coefficients reflect association not causation. A predictor set to zero by LASSO is not necessarily unimportant biologically; it may be correlated with a selected predictor and its signal was absorbed by that variable instead.