Logistic Regression: Going Further

This page covers three extensions of binary logistic regression for situations where the outcome has more than two categories or where regularisation is needed for high-dimensional data.

Multinomial Logistic Regression

When the outcome has three or more unordered categories, multinomial logistic regression fits a separate binary logistic model for each category relative to a reference category.

library(nnet)

# Example: predict plant community type from environmental variables
# Community types: grassland, woodland, wetland (unordered)

multi_model <- multinom(community ~ moisture + pH + nitrogen,
                        data = vegetation_data)

summary(multi_model)

# Odds ratios for each predictor and each category vs. reference
exp(coef(multi_model))

# Predictions
predict(multi_model, newdata = new_sites, type = "class")
predict(multi_model, newdata = new_sites, type = "probs")

The reference category is the first level of the outcome factor. All coefficients are interpreted relative to that category.

Ordinal Logistic Regression

When categories have a natural order (mild, moderate, severe; low, medium, high), ordinal logistic regression respects that ordering. The proportional odds model assumes that the effect of each predictor is the same across all thresholds between adjacent categories.

library(MASS)

# Predict disease severity: mild < moderate < severe
severity <- factor(patient_data$severity,
                   levels  = c("mild", "moderate", "severe"),
                   ordered = TRUE)

ord_model <- polr(severity ~ age + crp + bmi,
                  data   = patient_data,
                  Hess   = TRUE)

summary(ord_model)

# Odds ratios
exp(coef(ord_model))

# Test proportional odds assumption
library(brant)
brant(ord_model)
# Non-significant: assumption plausibly met

If the proportional odds assumption is violated, fit separate binary models for each threshold or use a partial proportional odds model.

Regularised Logistic Regression

When the number of predictors approaches or exceeds the number of observations, standard logistic regression overfits. Regularisation adds a penalty to the log-likelihood that shrinks coefficients toward zero.

Three common penalties:

LASSO (L1): shrinks many coefficients to exactly zero, producing a sparse model. Useful for variable selection.
Ridge (L2): shrinks all coefficients but rarely to zero. Better when many predictors contribute small effects.
Elastic net: a weighted combination of L1 and L2 penalties. More stable than LASSO when predictors are correlated.

library(glmnet)

X <- as.matrix(your_data[, predictor_cols])
y <- your_data$outcome

# LASSO (alpha = 1)
cv_lasso <- cv.glmnet(X, y, family = "binomial", alpha = 1, nfolds = 10)

# Ridge (alpha = 0)
cv_ridge <- cv.glmnet(X, y, family = "binomial", alpha = 0, nfolds = 10)

# Elastic net (alpha = 0.5)
cv_enet  <- cv.glmnet(X, y, family = "binomial", alpha = 0.5, nfolds = 10)

# Plot cross-validation error vs. lambda
plot(cv_lasso)

# Coefficients at optimal lambda
coef(cv_lasso, s = "lambda.min")

# Predictions
predict(cv_lasso, newx = X_test, s = "lambda.min", type = "response")

lambda.min gives the lambda with the lowest cross-validation error. lambda.1se gives the most regularised model within one standard error of the minimum, producing a sparser model that is often preferred when interpretability matters.

Comparing penalty types

# Extract non-zero predictors for LASSO
coef_lasso <- coef(cv_lasso, s = "lambda.min")
selected <- rownames(coef_lasso)[coef_lasso[, 1] != 0]
selected <- selected[selected != "(Intercept)"]
cat("LASSO selected", length(selected), "predictors\n")

# Ridge keeps all predictors but shrinks them
coef_ridge <- coef(cv_ridge, s = "lambda.min")
# All non-zero, but smaller than unpenalised estimates

Which Extension to Use?

Situation	Method
Three or more unordered outcome categories	Multinomial logistic regression
Ordered outcome categories	Ordinal logistic regression (proportional odds)
More predictors than samples	LASSO or elastic net
Many predictors with small effects	Ridge regression
Correlated predictors and variable selection	Elastic net