Ridge Regression Explained: How to Run, Read and Report It
Ridge regression is a form of regularised linear regression designed for situations where predictors are strongly correlated or numerous relative to the sample size. By adding a penalty on the size of the coefficients, it trades a small amount of bias for a large reduction in variance, which often improves prediction. This guide explains the method and a fully worked collinear example.
What is ridge regression?
Ordinary least squares minimises the residual sum of squares. Ridge regression minimises the same quantity plus an L2 penalty on the coefficients:
β̂ridge = argmin [ Σ(yi − xi′β)² + λ Σβj² ] = (X′X + λI)−1X′y
Predictors are standardised (and y is centred) so that the penalty treats all coefficients equally; the intercept is not penalised. The tuning parameter λ ≥ 0 controls shrinkage: λ = 0 gives OLS, and larger λ pulls all coefficients toward zero, but never exactly to zero (Hoerl & Kennard, 1970). The effective degrees of freedom, trace[X′X(X′X + λI)−1], falls from p toward 0 as λ grows.
When should you use it?
Use it when predictors are highly correlated (large variance inflation factors), when there are many predictors relative to observations, or when prediction accuracy matters more than interpreting individual coefficients. It keeps all predictors in the model; if you want variable selection, see lasso regression.
Choosing λ
λ is chosen by cross-validation: fit the model on part of the data across a grid of λ values, predict the held-out part, and pick the λ with the lowest prediction error (Hastie et al., 2009). Report how λ was chosen, the folds and the software scaling, since different programs define λ differently.
Assumptions and caveats
Ridge relies on the same linear-model assumptions as OLS (linearity, independent errors, constant variance). Predictors must be on a comparable scale, so standardise first. Ridge coefficients are biased, so classical confidence intervals and p-values are not valid in the usual way; bootstrap or Bayesian intervals are alternatives.
Original example (illustrative)
Question. Four correlated measures (x₁ to x₄) of study behaviour are used to predict an exam score (n = 24). The correlations among predictors are high (largest |r| = 0.95), and VIFs are 12.0, 17.2, 11.5, 11.7, far above the commonly used warning level of 5 to 10 (though O'Brien (2007) cautions against rigid cut-offs).
Show the full illustrative dataset (CSV, copy into the tool)
x1,x2,x3,x4,score 65.7,68.9,68.9,67.2,77.3 31.1,30.2,28.2,31.7,29.1 58.1,53.2,53.9,55.5,68.7 44.8,42.3,43.3,45.8,55.9 45.8,47.6,49.9,45.3,67.2 50.8,49.7,48.3,49.0,59.0 31.8,31.2,34.1,32.3,39.6 47.4,45.7,44.2,50.2,57.3 45.4,42.9,47.5,40.6,62.1 77.7,72.8,76.5,78.3,100.5 52.0,54.0,53.5,51.6,71.7 48.9,48.2,50.0,48.5,51.6 40.7,42.8,48.7,48.2,48.7 47.3,44.2,43.5,51.2,50.9 39.3,37.8,40.3,45.1,55.0 42.7,48.6,45.2,50.7,58.3 54.5,48.6,56.8,48.6,60.5 49.9,45.3,44.8,47.1,54.5 56.5,60.6,55.7,56.4,73.5 45.7,51.5,47.8,49.7,53.3 50.3,44.1,50.7,44.1,50.8 62.1,62.5,62.9,66.1,80.5 57.8,55.8,51.1,57.4,84.6 47.9,43.7,42.2,42.3,52.6
Standardised OLS coefficients are 5.81, 6.20, −0.27, 2.46, some with unstable signs. Ridge coefficients over a range of λ (standardised predictors, centred outcome) are:
| λ | x₁ | x₂ | x₃ | x₄ | Effective df |
|---|---|---|---|---|---|
| 0 (OLS) | 5.81 | 6.20 | −0.27 | 2.46 | 4.00 |
| 1 | 4.77 | 4.79 | 1.35 | 3.16 | 2.73 |
| 5 | 3.81 | 3.80 | 2.55 | 3.31 | 1.62 |
| 8.2 (CV choice) | 3.55 | 3.54 | 2.71 | 3.22 | 1.37 |
| 20 | 3.03 | 3.04 | 2.65 | 2.88 | 1.02 |
Five-fold cross-validation over a grid of λ from 0.01 to 100 selects λ = 8.2. Cross-validated RMSE is 6.86 points for OLS and 6.48 for ridge. Effective df at the chosen λ is 1.36, compared with 4 for OLS.
Interpretation. As λ increases, the erratic OLS coefficients shrink and stabilise toward a more plausible shared pattern, which reflects that the predictors carry overlapping information. Ridge predicts held-out data better than OLS here (lower cross-validated error). Individual ridge coefficients should be read as shrunken, partly pooled effects and not as causal contributions. The example is invented and small; real analyses need larger samples and repeated cross-validation.
How to run ridge regression with StatFAS
Open the Ridge Regression Calculator. Arrange one outcome column and several predictor columns.
Understanding the output
- Coefficients at the chosen λ: shrunken estimates, usually reported on the standardised scale.
- λ and its selection method: essential for reproducibility.
- Coefficient path plot: how each coefficient shrinks as λ grows.
- Cross-validation error: the basis for choosing λ and comparing with OLS.
- R² or RMSE on held-out data: use out-of-sample performance, not training fit.
APA 7 reporting example
Because the four predictors were highly collinear (VIF = 11.5–17.2), ridge regression was used. Predictors were standardised and λ = 8.2 was selected by five-fold cross-validation. Cross-validated RMSE was 6.48 for ridge and 6.86 for OLS. Standardised ridge coefficients were 3.55, 3.54, 2.71, 3.22 for x₁ to x₄.
Common mistakes and troubleshooting
- Forgetting to standardise predictors, so that variables on large scales are penalised unfairly.
- Tuning λ on the full data without cross-validation, which overfits.
- Reporting p-values from an OLS-style output as if valid after shrinkage.
- Expecting variable selection. Ridge shrinks but keeps every predictor.
- Comparing λ values across software without checking the scaling convention.
Limitations and alternatives
Ridge is best when many predictors each have small effects. If you believe only a few predictors matter, lasso or elastic net may fit better. If you simply need adjusted coefficients for a handful of moderately correlated predictors with enough data, ordinary multiple linear regression with a collinearity check may suffice. For a decision guide, see the regression overview.
Conclusion
Ridge regression stabilises regression when predictors overlap. Standardise, choose λ by cross-validation, judge success by out-of-sample error and interpret the coefficients as shrunken, not causal, effects.
1.
statfas-ridge-regression-interface.png, alt: "StatFAS Ridge Regression Calculator interface"2.
statfas-ridge-regression-example-data.png, alt: "Example data entered in the StatFAS Ridge Regression Calculator"3.
statfas-ridge-regression-settings.png, alt: "Ridge Regression Calculator settings in StatFAS"4.
statfas-ridge-regression-result.png, alt: "Ridge Regression Calculator results generated in StatFAS"Frequently asked questions
What is the difference between ridge and lasso regression?
Ridge uses an L2 penalty that shrinks coefficients but keeps all of them in the model. Lasso uses an L1 penalty that can shrink some coefficients exactly to zero, performing variable selection.
Do I need to standardise predictors for ridge regression?
Yes. The penalty depends on the scale of each coefficient, so predictors should be standardised to make the shrinkage comparable across variables.
How do I choose the ridge penalty λ?
Use cross-validation over a grid of λ values and choose the value with the lowest cross-validated error, or one standard error above the minimum for a simpler model. Report the method and scaling.
References
- Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67.
- Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning (2nd ed.). Springer.
- O'Brien, R. M. (2007). A caution regarding rules of thumb for variance inflation factors. Quality & Quantity, 41(5), 673–690.