Stepwise Regression Explained: How to Run, Read and Report It

By Dr. Faheem Afzal Shah, Founder, StatFAS.com · Tool: Stepwise Regression Calculator

Stepwise regression is one of the most widely used and most criticised procedures in applied statistics. It is easy to describe: start with a model, add or drop predictors one at a time according to a rule, and stop when no change improves the criterion. It is also easy to misuse, because the final model looks like the result of a clean test when it is really the winner of a search. This guide explains how the procedure works, works through an example with checkable numbers, shows how to read the steps and the final model, and explains how to report it honestly.

What is stepwise regression?

Stepwise regression is an automated variable-selection method for regression models. There are three common variants. Forward selection starts with no predictors and adds the one that most improves the criterion at each step. Backward elimination starts with all candidate predictors and removes the least useful one at each step. Bidirectional (stepwise) selection allows both adding and removing at each step. The criterion can be a p-value threshold for entry and removal, or an information criterion such as AIC (Akaike, 1974) or BIC. The method sits in a long tradition of subset-selection procedures reviewed by Hocking (1976).

When should you use stepwise regression?

When not to use it. If you have a theory-driven model, specify it directly. If your aim is to estimate the effect of one exposure while adjusting for confounders, select covariates by subject-matter reasoning rather than by an algorithm. Stepwise selection is also a poor choice with small samples and many candidate predictors. Heinze et al. (2018) review the alternatives, including penalised regression, and the practical recommendations for choosing among them.

The mathematical foundation

For a linear model with k predictors fitted to n observations, three quantities drive most stepwise decisions.

Partial F test. Adding a predictor to a model with k predictors changes the residual sum of squares from RSSk to RSSk+1:

F = (RSSk − RSSk+1) / [RSSk+1 / (n − k − 2)]

which is compared with an F(1, n − k − 2) distribution (equivalently, the predictor's t test).

AIC for a Gaussian linear model, up to a constant:

AIC = n · ln(RSS / n) + 2p

where p is the number of estimated parameters. Lower is better, and the penalty 2p discourages needless predictors. BIC replaces 2p with p · ln(n) and is more conservative for n ≥ 8.

Adjusted R².

R²adj = 1 − (1 − R²)(n − 1)/(n − k − 1)

Assumptions and data requirements

Original research example (illustrative)

Background. A researcher wants to explain exam scores among 100 students using four candidate predictors: weekly study hours, class attendance rate, nightly sleep hours and commute time.

Objective and question. Which of the candidate predictors are retained by forward selection using AIC?

Data. n = 100. All values below are invented for teaching.

Step 1: Run forward selection by AIC

Illustrative forward selection path
StepActionModelAICDecision
0StartIntercept only620.4—
1Add study hoursStudy hours585.2Accept (−35.2)
2Add attendanceStudy hours + attendance574.9Accept (−10.3)
3Try sleep (best remaining)Study hours + attendance + sleep575.8Reject (+0.9)

Differences: 585.2 − 620.4 = −35.2; 574.9 − 585.2 = −10.3; 575.8 − 574.9 = +0.9. Because adding sleep raises AIC, the procedure stops. Commute time, which would have been tried only if sleep were accepted, is not in the model.

Step 2: Evaluate the final model

Final model: study hours and attendance, R² = .412, so k = 2 and n = 100.

Adjusted R² = 1 − (1 − .412)(99/97) = 1 − .588 × 1.0206 = 1 − .600 = .40.

Overall F(2, 97) = (.412/2) / (.588/97) = .206 / .00606 = 33.98, which is far beyond the 5% critical value of about 3.09, so p < .001.

Step 3: Interpret

The selected model explains about 40% of the variance in exam scores (adjusted R²). Study hours and attendance were retained, and sleep did not improve AIC enough to justify an extra parameter. Three cautions apply. First, the R², F test and p-values above are computed as if this model had been specified in advance, so they are optimistic. Second, a +0.9 AIC difference is small, and a different sample could easily have retained sleep: the selection is not strong evidence that sleep is irrelevant. Third, the result should be checked on new data or by bootstrap resampling of the whole selection procedure, which would show how often each predictor is selected.

How to run stepwise regression with the StatFAS Stepwise Regression Calculator

Open statfas.com/stepwise-regression-calculator.html.

  1. Prepare your data. One row per observation, one outcome column and one column per candidate predictor, with a header row and numeric codes. Handle missing values and check for outliers first.
  2. Load the data and choose the outcome and candidates. Exclude IDs and variables that are consequences of the outcome.
  3. Set options. Direction (forward, backward or both), criterion (p-value thresholds, AIC or BIC) and any entry and removal thresholds.
  4. Run and review. Read the step-by-step path first, then the final model, then diagnostics.
  5. Download results. Verify every number and sentence before it enters a manuscript.
Screenshot 1 (to be supplied): statfas-stepwise-regression-interface.png
Alt text: "StatFAS stepwise regression calculator interface before data entry."
Caption: The Stepwise Regression Calculator on StatFAS.com.
Screenshot 2: statfas-stepwise-regression-example-data.png · Alt: "Exam score and four candidate predictors loaded in the StatFAS stepwise regression calculator."
Screenshot 3: statfas-stepwise-regression-settings.png · Alt: "Direction and criterion settings for stepwise regression in StatFAS."
Screenshot 4: statfas-stepwise-regression-steps.png · Alt: "Step-by-step selection path and final model from StatFAS." Use only genuine captures of the tool.

Reading the output

APA 7 reporting example (based on the illustrative data)

Forward stepwise selection using the Akaike information criterion was applied to four candidate predictors of exam scores (N = 100): study hours, attendance, sleep and commute time. Study hours and attendance were retained, and the resulting model explained 41.2% of the variance (adjusted R² = .40), F(2, 97) = 33.98, p < .001. Because the model was selected from the same data used to estimate it, coefficients and p-values should be regarded as exploratory.

Essential elements: the candidate predictors and how they were chosen, the direction and criterion, the sample size, the selection path, the final model's coefficients with intervals, and a clear statement that inference after selection is optimistic. If you validated by resampling or on a hold-out sample, report those results.

Common mistakes and troubleshooting

Limitations and alternatives

The central problem is that the data are used twice, once to choose the model and once to assess it. Smith (2018) summarises the evidence against relying on stepwise methods, and Harrell (2015) argues for prespecified models or penalisation. Alternatives include the lasso and elastic net, which shrink coefficients while selecting, best-subsets with cross-validation, and expert-driven model building. If your concern is the shape of the outcome distribution rather than predictor choice, consider quantile regression. To examine relationships among predictors before selecting, use the correlation calculator, the heatmap guide and the descriptive statistics calculator. For a latent-variable approach to many correlated items, see PCA and the EFA guide.

Frequently asked questions

What is stepwise regression?

An automated procedure that adds or removes predictors one at a time according to a criterion, to arrive at a smaller regression model.

Why is it criticised?

The same data select and test the predictors, which inflates R², biases coefficients and makes p-values too small. Selections can also be unstable across samples.

Should I use AIC or p-values?

Information criteria are generally preferred to p-value thresholds, but neither removes the selection bias.

What should I do instead?

Prespecify the model from theory where possible; otherwise use penalised regression or validate the selection by resampling.

Conclusion

Stepwise regression can be a useful exploratory tool if you treat it as a search and not as a test. State the candidate set, direction and criterion, report the path, check stability, and validate before you trust the result. When you are ready to apply it to your data, the StatFAS Stepwise Regression Calculator provides a browser-based route, and its results should be checked by you before publication.

References

Author attribution: Dr. Faheem Afzal Shah, Plant Functional Genomics and Regulatory Genomics Researcher; StatFAS.com Founder.