Portfolio Optimization

Markowitz's Mean-Variance Optimization

In 1952, Harry Markowitz published "Portfolio Selection" — the paper that launched modern portfolio theory and eventually earned him the Nobel Prize. His key insight: investors care about both expected return and risk (variance), and the composition of the portfolio determines both.

The efficient frontier is the set of portfolios that maximize expected return for a given level of risk, or equivalently, minimize risk for a given expected return. Any portfolio below the frontier is "dominated" — you could get more return for the same risk by moving to the frontier.

The Math

Given n assets with expected returns μ = [μ₁, μ₂, ..., μₙ] and covariance matrix Σ (an n×n matrix of return covariances), the portfolio with weights w = [w₁, ..., wₙ] has:

Expected return: E[R_p] = wᵀ · μ = Σᵢ wᵢ × μᵢ
Variance: σ²_p = wᵀ · Σ · w = Σᵢ Σⱼ wᵢ wⱼ σᵢⱼ

The optimization problem: given a target return μ*, find weights w that minimize variance subject to wᵀμ = μ* and Σwᵢ = 1 (fully invested). With a short-selling constraint, add wᵢ ≥ 0.

The Maximum Sharpe Portfolio

Among all efficient portfolios, the one with the highest Sharpe ratio — maximum excess return per unit of risk — is called the tangency portfolio. It lies at the tangent point between the efficient frontier and a line from the risk-free rate. In the CAPM framework, the tangency portfolio is the market portfolio.

In practice, finding the tangency portfolio requires numerical optimization. The most common approach: parameterize by risk aversion λ and solve the unconstrained problem:

max: wᵀμ − λ/2 × wᵀΣw

Sweeping λ from high (risk averse → low-vol portfolio) to low (risk tolerant → high-return portfolio) traces out the efficient frontier.

The Covariance Matrix: The Hard Part

The covariance matrix Σ has n(n+1)/2 parameters. For a 50-stock portfolio, that's 1,275 parameters to estimate — but you typically have only 252 daily return observations per year. With more parameters than observations, the estimated covariance matrix becomes unstable and "over-fit" to historical noise.

Professionals use regularization techniques to stabilize Σ:

• Shrinkage (Ledoit-Wolf): Shrink the sample covariance matrix toward a structured target (like identity or a single-factor model covariance). This dramatically improves out-of-sample performance.
• Factor models: Model covariance through factor exposures: Σ = B·F·Bᵀ + D, where B is a factor loading matrix, F is factor covariance, and D is diagonal idiosyncratic variance. This uses far fewer parameters.
• Equal-weight heuristic: The famous "1/N" portfolio (equal weights) often beats optimized portfolios out-of-sample because it avoids estimation error entirely.

Implementation in Python

For a small portfolio, you can implement basic mean-variance optimization using only NumPy. The key steps:

1. Estimate expected returns (or use equal returns as a naive assumption)
2. Compute the sample covariance matrix from return histories
3. Set up the optimization problem with constraints
4. Solve numerically (scipy.optimize.minimize for general cases)
5. Plot the efficient frontier by solving at multiple target return levels

In the exercise, we'll build the core portfolio math from scratch using only Python standard library tools.