Correlation

Correlation denoted as $\rho$, is a standardized measure of the degree of linear relationship between two variables. Unlike covariance, correlation values are normalized between -1 and 1. The correlation is defined as:

where ${\sigma }_X$ and ${\sigma }_Y$ are the standard deviations of $X$ and $Y$, respectively. A correlation of +1 indicates a perfect positive linear relationship, -1 indicates a perfect negative linear relationship, and 0 indicates no linear relationship.

One caution carries forward into everything built on them: all three are estimated from a finite sample of history, and correlation in particular is notoriously unstable — it is measured in calm markets and relied upon in violent ones, where it moves toward $+1$ exactly when the diversification it promised is needed (section “MPT Under Deep Uncertainty”). The mathematics below is exact. The inputs are not.

There is a second, quieter problem, and it is arithmetic rather than behavioral. A portfolio of $N$ assets has $N(N-1)∕2$ distinct pairwise correlations to estimate, and that count grows with the square of $N$ while your data grows only with the length of the sample. Fifty holdings means 1,225 correlations; three years of monthly returns gives you 36 observations per series to estimate them from. The estimates cannot all be good, and they are not. Most of what a sample correlation matrix appears to tell you — this pair hedges that one, this cluster moves together — is sampling noise that would come out differently on a different three years. Only a handful of the relationships in a large matrix are real and persistent; the rest is structure the estimator invented.

This matters because of what sits downstream. An optimizer handed such a matrix does not treat the noise as noise. It hunts for the most attractive-looking combinations, which are disproportionately the pairs whose correlations happen to be mis-estimated in the flattering direction, and it concentrates the portfolio into exactly those. The apparent diversification is then largest precisely where the evidence for it is weakest, and it fails out of sample. This is the concrete reason a naive mean-variance optimization produces the unstable, wildly-concentrated weights described in section “MPT Under Deep Uncertainty”, and the reason the practical fixes — shrinking the estimates toward a simple structure, or replacing the point estimates with a small number of explicit views as Black-Litterman does — work by throwing information away rather than adding it. Treat any correlation you did not measure across multiple regimes as a guess with a decimal point attached.