Regression Analysis
Regression is a statistical technique widely applied in finance, economics, investing, and other fields to study how variables are related. The method focuses on examining the strength, direction, and nature of the relationship between a dependent variable (commonly represented as Y) and one or more independent variables (predictors). It assumes a straight-line relationship between variables, often referred to as the line of best fit. This line illustrates how changes in one variable are associated with changes in another.
• The slope of the line explains the rate of change in the dependent variable with respect to the independent variable.
• The intercept (value of Y when X=0) indicates the baseline level of the dependent variable when the predictor is absent.
Types of Linear Regression
→ Involves only one independent variable.
→ Useful when the relationship is between a single predictor and the response variable.
→ Example: Predicting sales (Y) based on advertising expenditure (X).
2. Multiple Linear Regression:
→ Uses two or more predictors to explain or forecast the dependent variable.
→ It measures the individual effect of each predictor while statistically controlling for the others.
→ Example: Estimating house prices (Y) based on square footage, number of rooms, and location.
Assumptions
1. Linearity : The conditional expectation of y given X is linear in parameters. If the true relationship is non-linear, linear models will be biased unless transformed features are used
2. Independence of errors : Observations (and their errors) are independent. Violation (e.g., time-series autocorrelation) invalidates standard errors and tests.
3. Homoscedasticity: Homoscedasticity refers to constant variance of errors. Heteroscedasticity makes OLS still unbiased but standard errors inconsistent.
4. Normality of errors (for inference):εi are normally distributed (mainly needed for small-sample exact inference).
5. No perfect multicollinearity: Predictors are not exact linear combinations of each other; otherwise multicollinearity exist.
6. Exogeneity / Zero conditional mean:If predictors correlate with the error (omitted variables, measurement error), estimates are biased.
7. Correct model specification: No important predictors omitted, and functional form is appropriate.
8. No influential outliers:Extreme points can unduly affect coefficient estimates.
Step-by-step processing
1. Define the goal: Decide whether the goal is explanation (estimating effects, causal claims) or prediction (minimise out-of-sample error).
2. Collect and clean data: Handle missing values, check ranges, correct obvious errors.
3. Exploratory Data Analysis (EDA): Visualise relationships (scatterplots for continuous predictors), summary statistics, correlations, and identify outliers.
→Transform skewed variables (log, Box–Cox).
→Create interaction or polynomial terms if non-linear effects are suspected.
→Standardise/scale predictors when helpful (e.g., for regularized methods).
5. Split data (if predictive): Reserve a test set or use cross-validation to estimate out-of-sample performance.
6. Fit the model:Use OLS (or another fitting method) to obtain β^.
7.Check diagnostics (assumptions):
→ Residuals vs fitted values plot (linearity, homoscedasticity).
→ Q–Q plot of residuals (normality).
→ Variance Inflation Factor (VIF) for multicollinearity.
→ Durbin–Watson or autocorrelation function (time series).
→ Breusch–Pagan or White test (heteroscedasticity).
→ Cook’s distance / leverage (influential points).
8. Remediate issues if present:
→ Nonlinearity: add polynomial terms or transform variables; or use non-linear methods.
→ Heteroscedasticity: use weighted least squares or robust (heteroscedasticity-consistent) standard errors.
→ Multicollinearity: remove or combine predictors, or use ridge/LASSO.
→ Autocorrelation: use time-series models (ARIMA, GLS) or include lagged variables.
→ Outliers: investigate, transform, or use robust regression.
9. Model selection and regularisation: Model selection and regularisation. Use adjusted R square, AIC/BIC, cross-validation, or penalised methods (ridge, LASSO) to avoid overfitting.
10. Evaluate performance:
→ For fit: R square, adjusted R square.
→ For prediction: RMSE, MAE, MAPE, cross-validated error.
→ For inference: coefficient estimates, standard errors, t-tests, p-values, confidence intervals, and an overall F-test.
11. Interpret results: Report effect sizes in meaningful units, statistical significance, and practical significance. Distinguish correlation from causation.
12. Report limitations and diagnostics: . Be transparent about assumption checks, model choices, possible biases, and the domain of applicability.
Diagnostics & common remedial actions
Residual plot shows fan shape (heteroscedasticity) :Try variance stabilising transform (e.g., log y), weighted least squares, or robust SEs.
Residuals not centered around zero / curvature : Add polynomial or interaction terms; check for omitted variables.
High VIF (>5 or >10) : Multicollinearity; consider removing correlated predictors, principal components, or regularization.
Autocorrelation (time series): Include lagged variables, use GLS, or time-series specific models.
Non-normal residuals but large sample : Central limit theorem often mitigates; otherwise use bootstrap for inference.
Influential outliers (large Cook’s distance) :Investigate data point, consider robust regression or model with/without point and report sensitivity.
Advantages of Regression
Simplicity and interpretability:Coefficients have straightforward unit-based interpretations.
Computationally efficient: Closed-form OLS for modest sized problems.
Strong inferential framework: Well-developed tools for hypothesis tests and confidence intervals.
Baseline model:Good starting point before trying more complex models.
Extendable:Basis for generalized linear models (GLMs) and regularised variants (ridge, LASSO).
Works well when assumptions approx. hold:Produces unbiased, efficient estimates under classical assumptions.
Disadvantages of Regression
Requires linearity (in parameters):Poor at capturing complex nonlinear relationships without feature engineering.
Sensitive to outliers and influential points.
Multicollinearity: Multicollinearity can inflate variances of estimates and make interpretation unstable.
Inference depends on assumptions:Violations (heteroscedasticity, autocorrelation, endogeneity) lead to invalid standard errors or biased estimates.
Omitted variable bias:If a relevant predictor is left out and correlated with included predictors, estimates are biased.
Not ideal for very high-dimensional problems.
Causal claims need extra work:Regression alone doesn’t prove causality and requires design (experiments), instrumental variables, or strong assumptions.