Law Of Large Numbers

Definition and Overview

Basic Concept

Law of Large Numbers (LLN): a fundamental theorem in probability stating that the sample average of independent, identically distributed (i.i.d.) random variables converges to the expected value as sample size increases.

Key Idea

Convergence of averages to population mean ensures stability of long-term frequencies and validates statistical inference.

Scope

Applies to sequences of random variables with finite expected values under certain conditions.

Historical Development

Early Insights

Jacob Bernoulli (1713) first proved the theorem for Bernoulli trials, called Bernoulli's theorem.

Subsequent Extensions

Poisson, Chebyshev, Markov, and Kolmogorov extended LLN to more general random variables and weaker conditions.

Modern Framework

Kolmogorov formalized Strong LLN using measure theory in 1930s; current theory integrates ergodic theory and stochastic processes.

Types of Law of Large Numbers

Weak Law of Large Numbers (WLLN)

Sample average converges in probability to the expected value as sample size tends to infinity.

Strong Law of Large Numbers (SLLN)

Sample average converges almost surely (with probability 1) to the expected value.

Distinction

SLLN implies WLLN; SLLN provides stronger mode of convergence; WLLN sufficient for many practical applications.

Formal Statements and Conditions

Weak Law of Large Numbers

Let X₁, X₂, ..., Xₙ be i.i.d. random variables with finite mean μ. For any ε>0:

P(|(X̄ₙ − μ)| > ε) → 0 as n → ∞ 

Strong Law of Large Numbers

Under same conditions:

P(limₙ→∞ X̄ₙ = μ) = 1 

Conditions

Finite first moment for WLLN; finite first moment plus independence and identically distributed variables for SLLN; extensions exist for dependent or non-identical variables with additional constraints.

Proof Sketches

WLLN via Chebyshev's Inequality

Variance finite: apply Chebyshev's inequality to sample mean, showing probability of large deviations tends to zero.

SLLN via Kolmogorov's Three-Series Theorem

Decomposes partial sums; uses Borel-Cantelli lemma to establish almost sure convergence.

Intuition

Large samples average out random fluctuations, stabilizing around expected value.

Applications

Statistics

Justifies sample mean as consistent estimator of population mean.

Finance

Risk modeling and portfolio theory rely on convergence properties of averages.

Quality Control

Predictability of long-run averages supports product consistency and process monitoring.

Examples and Illustrations

Coin Tosses

Proportion of heads approaches 0.5 as number of tosses increases.

Dice Rolls

Average value converges to expected 3.5 with large number of rolls.

Real-World Data

Empirical averages of measurements stabilize with increasing observations.

Relation to Other Limit Theorems

Central Limit Theorem (CLT)

LLN ensures convergence of sample mean; CLT describes distribution of normalized sums around mean.

Ergodic Theorem

Generalizes LLN to dependent processes and dynamical systems under ergodicity assumptions.

Glivenko-Cantelli Theorem

Similar convergence for empirical distribution functions rather than means.

Statistical Consistency and LLN

Consistency Definition

Estimator is consistent if it converges in probability to true parameter as sample size grows.

Role of LLN

LLN guarantees consistency of sample mean and other estimators under suitable conditions.

Implications

Foundation for inferential statistics, hypothesis testing, and confidence intervals.

Limitations and Assumptions

Independence Assumption

Classical LLN requires independence; dependent variable versions require additional conditions.

Identical Distribution

Many forms require identical distribution; relaxations exist but with weaker conclusions.

Finite Expectation

Expected value must be finite; infinite variance or mean may invalidate LLN.

Simulation and Empirical Demonstrations

Monte Carlo Methods

LLN underpins accuracy of Monte Carlo simulations by ensuring sample averages approximate expected values.

Empirical Convergence

Visualizing sample mean against sample size demonstrates convergence behavior.

Algorithm Example

Initialize sum = 0For i = 1 to N: Generate random variable X_i sum += X_i sample_mean = sum / i Record sample_meanPlot sample_mean vs. i to observe convergence 

Key Formulas and Inequalities

Chebyshev's Inequality

P(|X̄ₙ − μ| ≥ ε) ≤ Var(X̄ₙ) / ε² = σ² / (n ε²) 

Bernoulli's Theorem (Special Case)

For Bernoulli random variables X_i with parameter p:

P(|X̄ₙ − p| ≥ ε) → 0 as n → ∞ 

Summary Table

References

• Feller, W. "An Introduction to Probability Theory and Its Applications," Vol. 1, Wiley, 1968, pp. 219-227.
• Durrett, R. "Probability: Theory and Examples," 4th ed., Cambridge University Press, 2010, pp. 217-250.
• Billingsley, P. "Probability and Measure," 3rd ed., Wiley, 1995, pp. 356-370.
• Loève, M. "Probability Theory," 4th ed., Springer, 1977, pp. 252-280.
• Chung, K. L. "A Course in Probability Theory," 3rd ed., Academic Press, 2001, pp. 101-130.