Logistic Regression

Logistic regression models the probability that an observation belongs to one of two categories. It is the standard method for binary classification when interpretability matters: the coefficients have a direct biological meaning as odds ratios, and significance can be tested for each predictor individually.

Unlike LDA, logistic regression does not assume multivariate normality or equal covariances between groups. It is more robust to assumption violations and is often the better default for biological data where normality cannot be guaranteed.

The Logit Link

Linear regression predicts a continuous outcome directly. Logistic regression instead models the log-odds (logit) of the outcome as a linear function of the predictors:

$\log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots$

The logit transformation maps probabilities (bounded between 0 and 1) to the real line, where a linear model can be applied. The inverse of the logit is the logistic function, which maps the linear predictor back to a probability bounded between 0 and 1.

Fitting in R

# Simulate data: disease diagnosis based on age and BMI
set.seed(123)
n   <- 300
age <- rnorm(n, mean = 50, sd = 15)
bmi <- rnorm(n, mean = 25, sd = 5)
lp  <- -8 + 0.05 * age + 0.1 * bmi
disease <- rbinom(n, 1, prob = 1 / (1 + exp(-lp)))

df <- data.frame(age = age, bmi = bmi, disease = disease)

# Fit logistic regression
model <- glm(disease ~ age + bmi, data = df, family = binomial)
summary(model)

Interpreting Coefficients

Coefficients from glm() are on the log-odds scale. Exponentiate them to get odds ratios, which are easier to interpret:

round(exp(coef(model)), 5)
# (Intercept)         age         bmi 
#     0.00023     1.08166     1.04246

An odds ratio of 1.05 for age means that for each additional year of age, the odds of disease increase by 5%. An odds ratio below 1 indicates a protective effect.

# Odds ratios with 95% confidence intervals
exp(cbind(OR = coef(model), confint(model)))

Confidence intervals that do not cross 1 correspond to statistically significant predictors.

Odds ratios are not probabilities

A coefficient of 0.5 does not mean a 50% increase in the probability of disease. It means the odds multiply by exp(0.5) = 1.65. The two are only approximately equal when the outcome is rare.

Predictions

# Predicted probabilities
new_data <- data.frame(age = c(40, 55, 70), bmi = c(22, 27, 32))
round(predict(model, newdata = new_data, type = "response"), 2)
#    1    2    3 
# 0.01 0.05 0.18

# Class predictions using 0.5 threshold
predicted_class <- ifelse(
  predict(model, type = "response") > 0.5, 1, 0)

The 0.5 threshold is not always optimal. A lower threshold increases sensitivity at the cost of specificity; a higher threshold does the reverse. The ROC (Receiver Operating Characteristic - signal detection theory) curve makes this trade-off visible by plotting the true positive rate (sensitivity) against the false positive rate (1 − specificity) across every possible threshold from 0 to 1. The full curve is discussed in Model Evaluation below.

Model Evaluation

Confusion matrix

library(caret)

confusionMatrix(factor(predicted_class),
                factor(df$disease),
                positive = "1")
#> Accuracy   : 0.883
#> Sensitivity: 0.850   (true positive rate)
#> Specificity: 0.900   (true negative rate)

ROC curve and AUC

The ROC curve is useful precisely because it decouples model quality from threshold choice. You first evaluate how well the model discriminates at all, then separately decide where to set the threshold based on the costs in your specific context. In a screening test for a serious disease, missing a true case (low sensitivity) may be far more costly than a false alarm, so you would shift the threshold down even if it increases false positives.

library(pROC)

predicted_probs <- predict(model, type = "response")
roc_obj <- roc(df$disease, predicted_probs)

plot(roc_obj, print.auc = TRUE, col = "steelblue", lwd = 2,
     main = "ROC Curve")
auc(roc_obj)

How to read the plot:

Each point on the curve corresponds to one threshold. Moving left along the curve means being stricter: you call fewer cases positive, catching fewer true positives but also making fewer false alarms. Moving right means being more permissive: you flag more cases as positive, gaining sensitivity but accepting more false positives. A model with no discrimination power produces a diagonal line from (0, 0) to (1, 1), equivalent to random guessing. A perfect model hugs the top-left corner, achieving high sensitivity with no false positives. Real models fall somewhere in between.

AUC (Area Under the Curve) summarises the entire curve as a single number. It is the probability that a randomly chosen positive case receives a higher predicted probability than a randomly chosen negative case.

AUC	Interpretation
0.90 to 1.00	Excellent
0.80 to 0.90	Good
0.70 to 0.80	Acceptable
0.60 to 0.70	Poor
0.50 to 0.60	No better than chance

Cross-Validation

Never report only training accuracy. Use cross-validation or a held-out test set.

# 10-fold CV with caret
set.seed(123)
cv_model <- train(
  factor(disease) ~ age + bmi,
  data    = df,
  method  = "glm",
  family  = binomial,
  trControl = trainControl(
    method          = "cv",
    number          = 10,
    classProbs      = TRUE,
    summaryFunction = twoClassSummary),
  metric = "ROC")

print(cv_model)

Assumptions

Binary outcome: the response must be 0/1 (or a two-level factor).

Independence: observations must be independent. Repeated measures or clustered data require mixed-effects logistic regression.

Linearity of the logit: the relationship between continuous predictors and the log-odds should be approximately linear. Check with a Box-Tidwell test or by plotting the logit against each predictor.

No severe multicollinearity: check with VIF as you would for linear regression.

Sufficient events per variable: a rough rule is at least 10 events (cases with outcome = 1) per predictor. Too few events leads to unstable estimates and possible complete separation.

library(car)
vif(model)           # Should be below 5

table(df$disease)    # Count events
# At least 10 events per predictor in the model

Logistic Regression vs LDA

	Logistic Regression	LDA
Normality assumed	No	Yes
Equal covariances assumed	No	Yes
Output	Probabilities and odds ratios	Discriminant scores
Interpretation	Direct: odds ratios	Indirect: loadings
Number of groups	Typically two (see extensions)	Two or more
High-dimensional data	Use regularisation	Use PCA first

As a default for binary outcomes in biology, logistic regression is usually the safer choice. LDA may perform better when its assumptions are genuinely met and sample sizes are moderate.

Exercise

Using mtcars, predict transmission type (am: 0 = automatic, 1 = manual) with mpg, hp, and wt as predictors.

Fit three models: mpg alone, mpg + hp, and mpg + hp + wt
Compare them with AIC and a likelihood ratio test
Report odds ratios with 95% confidence intervals for the best model
Plot the ROC curve and report AUC

Solution

library(pROC)

# 1. Fit models
m1 <- glm(am ~ mpg,           data = mtcars, family = binomial)
m2 <- glm(am ~ mpg + hp,      data = mtcars, family = binomial)
m3 <- glm(am ~ mpg + hp + wt, data = mtcars, family = binomial)

# 2. Compare
AIC(m1, m2, m3)
anova(m1, m2, m3, test = "Chisq")
# Lower AIC and significant LRT improvement favour m3

# 3. Odds ratios for best model
exp(cbind(OR = coef(m3), confint(m3)))

# 4. ROC curve
probs <- predict(m3, type = "response")
roc_obj <- roc(mtcars$am, probs)
plot(roc_obj, print.auc = TRUE, col = "steelblue", lwd = 2)