Multiple Regression

Definition and Overview

Concept

Multiple regression: statistical method predicting dependent variable (Y) from two or more independent variables (X1, X2,..., Xk). Extension of simple linear regression. Purpose: model linear relationships, quantify influence of predictors.

Dependent Variable

Continuous scale variable. Examples: income, blood pressure, test scores. Outcome measured, explained by predictors.

Predictor Variables

Independent variables, continuous or categorical (after dummy coding). May be correlated. Used to explain variance in dependent variable.

Historical Context

Developed early 20th century. Applications expanded in economics, social sciences, biology, engineering. Fundamental tool in statistical modeling.

Key Features

Simultaneous estimation of multiple effects, control for confounding, prediction, hypothesis testing.

"Multiple regression analysis is the backbone of statistical modeling in empirical research." -- Draper & Smith (1998)

Model Specification

General Form

Mathematical model: Y = β0 + β1X1 + β2X2 + ... + βkXk + ε. β0: intercept. βi: regression coefficients. ε: random error term.

Regression Equation

Y = β0 + β1X1 + β2X2 + ... + βkXk + ε

Interpretation of Terms

β0: expected value when all Xs = 0. βi: expected change in Y per unit increase in Xi, holding others constant. ε: residual variation unexplained by model.

Inclusion of Interaction Terms

Model can include product terms (X1*X2) to assess interaction effects. Expands linear model to capture conditional relationships.

Dummy Variables

Categorical predictors encoded as binary indicators. Allow inclusion of qualitative factors. Reference group baseline interpretation.

Assumptions

Linearity

Relationship between dependent variable and each predictor is linear. Violations require transformation or nonlinear modeling.

Independence

Observations are independent. No autocorrelation in residuals. Important in time series or clustered data.

Homoscedasticity

Constant variance of residuals across all levels of predictors. Heteroscedasticity biases standard errors, inference.

Normality of Residuals

Residuals approximately normally distributed. Critical for hypothesis tests and confidence intervals.

No Perfect Multicollinearity

Predictors not perfectly correlated. Allows unique estimation of coefficients.

Estimation Methods

Ordinary Least Squares (OLS)

Minimizes sum of squared residuals. Best linear unbiased estimator under Gauss-Markov assumptions.

Maximum Likelihood Estimation

Estimates parameters maximizing likelihood of observed data. Equivalent to OLS under normal errors.

Gradient Descent

Iterative numerical optimization. Used for large datasets or when OLS matrix inversion infeasible.

Regularization Techniques

Ridge and Lasso regression add penalty terms to control overfitting and multicollinearity.

Matrix Formulation

β = (X'X)^(-1) X'Y

X: design matrix including intercept column. β: vector of coefficients.

Interpretation of Coefficients

Magnitude

Indicates strength of association per unit change in predictor, holding others constant.

Sign

Positive β: direct relationship. Negative β: inverse relationship.

Statistical Significance

Determined by t-tests. Significance implies predictor contributes to explaining variance.

Confidence Intervals

Range of plausible values for β. Narrow intervals indicate precise estimates.

Standardized Coefficients

Coefficients computed on standardized variables. Useful for comparing relative importance.

Goodness-of-Fit Metrics

R-squared (R²)

Proportion of variance in Y explained by predictors. Range: 0 to 1. Higher is better fit.

Adjusted R-squared

Adjusts R² for number of predictors. Penalizes model complexity.

Root Mean Squared Error (RMSE)

Standard deviation of residuals. Lower values indicate better predictive accuracy.

F-statistic

Tests overall significance of model. Null: all βi = 0.

Information Criteria

AIC, BIC used for model comparison balancing fit and complexity.

Multicollinearity

Definition

High correlation among predictor variables. Causes instability in coefficient estimates.

Consequences

Inflated standard errors, unreliable t-tests, coefficient sign reversals.

Detection Methods

Variance Inflation Factor (VIF), condition indices, correlation matrix inspection.

Remedies

Remove or combine variables, apply dimension reduction (PCA), use regularization.

Thresholds

VIF > 5 or 10 indicates problematic multicollinearity.

Hypothesis Testing

Testing Individual Coefficients

Null hypothesis: βi = 0 (no effect). Use t-statistic: t = βi / SE(βi).

Testing Overall Model

Null hypothesis: all βi = 0. Use F-test comparing model to intercept-only model.

Confidence Intervals

Constructed as βi ± t* SE(βi). Interpret parameter uncertainty.

p-values

Probability of observing t or F-statistic under null. p < 0.05 commonly considered significant.

Type I and Type II Errors

Type I: false positive. Type II: false negative. Balance via significance level and power.

Residual Analysis

Purpose

Evaluate model assumptions and identify model inadequacies.

Residual Plots

Plot residuals vs fitted values for homoscedasticity and linearity check.

Normal Q-Q Plot

Assess normality of residuals. Deviations indicate non-normality.

Influence Diagnostics

Leverage, Cook’s distance identify influential observations affecting estimates.

Autocorrelation

Durbin-Watson test detects residual correlation in sequential data.

Model Selection and Validation

Variable Selection Techniques

Forward selection, backward elimination, stepwise selection based on criteria like AIC.

Cross-Validation

Partition data into training and test sets to evaluate model generalizability.

Information Criteria

AIC and BIC penalize complexity to avoid overfitting.

Regularization Methods

Lasso (L1) and Ridge (L2) regression reduce complexity and multicollinearity impact.

Model Diagnostics

Check residuals, multicollinearity, and predictive performance before finalizing model.

Applications

Economics

Forecasting GDP, inflation, unemployment using multiple indicators.

Social Sciences

Modeling effects of education, income, demographics on behavior or attitudes.

Healthcare

Predicting patient outcomes from clinical measurements and demographics.

Environmental Science

Assessing impact of pollution, weather, land use on ecological variables.

Engineering

Modeling system performance based on multiple design parameters.

Limitations

Linearity Constraint

Cannot capture nonlinear relationships without transformations or alternative models.

Sensitivity to Outliers

Outliers can distort coefficient estimates and predictions.

Multicollinearity Impact

High correlation among predictors complicates interpretation and reduces precision.

Overfitting Risk

Including too many predictors can lead to poor generalization on new data.

Assumption Violations

Violations reduce validity of inference and prediction accuracy.

References

• Draper, N. R., & Smith, H. (1998). Applied Regression Analysis. Wiley, Vol. 326, 1998, pp. 45-120.
• Montgomery, D. C., Peck, E. A., & Vining, G. G. (2012). Introduction to Linear Regression Analysis. Wiley, 5th ed., 2012, pp. 75-110.
• Kutner, M. H., Nachtsheim, C. J., Neter, J., & Li, W. (2005). Applied Linear Statistical Models. McGraw-Hill, 5th ed., 2005, pp. 200-250.
• James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An Introduction to Statistical Learning. Springer, 2013, pp. 90-140.
• Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate Data Analysis. Pearson, 7th ed., 2010, pp. 110-160.