Correlation & Covariance

Correlation and covariance are the mathematical language of relationships between variables. Nearly every multivariate method covered in this course builds on them: PCA decomposes the covariance matrix, clustering relies on distance metrics derived from correlations, LDA uses within- and between-group covariance, and ordination preserves correlation structure in reduced dimensions.

---

Covariance

Covariance measures how two variables change together:

$$\text{cov}(X, Y) = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{n-1}$$

• Positive: X and Y tend to increase together
• Negative: X increases as Y decreases
• Zero: no linear relationship

height <- c(160, 165, 170, 175, 180)
weight <- c(55, 60, 65, 70, 75)

cov(height, weight)
#> [1] 62.5

For multiple variables, covariances are organised in a covariance matrix. The diagonal contains variances; the off-diagonal entries are pairwise covariances.

data <- data.frame(height, weight, age = c(20, 22, 25, 28, 30))
cov(data)
#>        height weight  age
#> height   62.5   62.5 25.0
#> weight   62.5   62.5 25.0
#> age      25.0   25.0 16.5

The Problem with Covariance

Covariance depends on the units of measurement. Expressing height in metres instead of centimetres changes the covariance 100-fold, even though the relationship is identical. This makes covariances difficult to compare across variables.

---

Correlation

Correlation standardises covariance to always fall between -1 and +1:

$$\text{cor}(X, Y) = \frac{\text{cov}(X, Y)}{\sigma_X \cdot \sigma_Y}$$

It is unitless and directly interpretable:

Value	Interpretation
+1.0	Perfect positive linear relationship
+0.7 to +0.9	Strong positive
+0.5 to +0.7	Moderate positive
+0.3 to +0.5	Weak positive
0	No linear relationship
Negative values	Inverse relationships

data(iris)
cor_matrix <- cor(iris[, 1:4])
round(cor_matrix, 2)
#>              Sepal.Length Sepal.Width Petal.Length Petal.Width
#> Sepal.Length         1.00       -0.12         0.87        0.82
#> Sepal.Width         -0.12        1.00        -0.43       -0.37
#> Petal.Length         0.87       -0.43         1.00        0.96
#> Petal.Width          0.82       -0.37         0.96        1.00

---

Which Method to Use

Three correlation coefficients are commonly used. The choice depends on your data:

Method	Measures	Use when
Pearson	Linear relationships	Continuous, approximately normal data
Spearman	Monotonic relationships	Ordinal data, outliers, non-normality
Kendall	Rank concordance	Small samples, many tied ranks

x <- c(1, 2, 3, 4, 5, 6, 7, 8, 9, 100)  # note the outlier
y <- c(2, 4, 6, 8, 10, 12, 14, 16, 18, 20)

cor(x, y, method = "pearson")   #> 0.62 — pulled by the outlier
cor(x, y, method = "spearman")  #> 1.00 — robust to outlier
cor(x, y, method = "kendall")   #> 1.00 — also robust

Correlation measures linear (or monotonic) relationships only

A zero correlation signifies the absence of a linear relationship rather than the absence of any relationship. Therefore, always plot your data.

x <- seq(-5, 5, 0.1)
y <- x^2           # perfect nonlinear relationship
cor(x, y)          #> 0

---

Visualising Correlation Structure

A correlation heatmap is usually the most informative first step:

library(corrplot)

corrplot(cor(iris[, 1:4]),
         method = "circle",
         type = "upper",
         order = "hclust",
         addCoef.col = "black",
         tl.col = "black",
         tl.srt = 45)

For a fuller overview including scatterplots and distributions:

library(GGally)

ggpairs(iris,
        columns = 1:4,
        aes(color = Species, alpha = 0.5))

---

Covariance vs. Correlation: When to Use Which

Correlation is best for interpretation, visualization, and comparing variables on different scales. Covariance is preferable when scale carries meaning, particularly when running PCA on variables with the same unit to maintain their natural variance.

PCA and scaling

PCA on the correlation matrix weights all variables equally. PCA on the covariance matrix lets variables with larger variance dominate. When in doubt, scale your data (scale. = TRUE in prcomp()).

---

Exercise

Using the mtcars dataset:

1. Compute and visualise the correlation matrix
2. Identify the three strongest pairwise correlations
3. Test whether Pearson and Spearman give meaningfully different results

Solution

library(corrplot)

cor_p <- cor(mtcars, method = "pearson")
cor_s <- cor(mtcars, method = "spearman")

corrplot(cor_p, method = "circle", type = "upper",
         order = "hclust", tl.col = "black")

source("https://www.gdc-docs.ethz.ch/UniBS/MultivariateStatistics/examples/top_cor_pairs.R")
top_cor_pairs(cor_s, n = 5, use_abs = FALSE)
# Strongest correlations (Pearson)
# cyl  & disp:  0.92
# cyl  & hp  :  0.90
# disp & wt  :  0.90

# Compare Pearson vs Spearman
round(cor_p - cor_s, 2)
# Large differences indicate outliers or non-linearity