Stepwise Regression Explained: How to Run, Read and Report It
Stepwise regression is one of the most widely used and most criticised procedures in applied statistics. It is easy to describe: start with a model, add or drop predictors one at a time according to a rule, and stop when no change improves the criterion. It is also easy to misuse, because the final model looks like the result of a clean test when it is really the winner of a search. This guide explains how the procedure works, works through an example with checkable numbers, shows how to read the steps and the final model, and explains how to report it honestly.
What is stepwise regression?
Stepwise regression is an automated variable-selection method for regression models. There are three common variants. Forward selection starts with no predictors and adds the one that most improves the criterion at each step. Backward elimination starts with all candidate predictors and removes the least useful one at each step. Bidirectional (stepwise) selection allows both adding and removing at each step. The criterion can be a p-value threshold for entry and removal, or an information criterion such as AIC (Akaike, 1974) or BIC. The method sits in a long tradition of subset-selection procedures reviewed by Hocking (1976).
When should you use stepwise regression?
- Exploratory work, where the aim is to generate hypotheses about which predictors might matter, not to confirm them.
- Prediction problems where a smaller model is wanted and performance will be checked on new data.
- Screening a moderately large pool of candidate predictors before building a theory-driven model.
When not to use it. If you have a theory-driven model, specify it directly. If your aim is to estimate the effect of one exposure while adjusting for confounders, select covariates by subject-matter reasoning rather than by an algorithm. Stepwise selection is also a poor choice with small samples and many candidate predictors. Heinze et al. (2018) review the alternatives, including penalised regression, and the practical recommendations for choosing among them.
The mathematical foundation
For a linear model with k predictors fitted to n observations, three quantities drive most stepwise decisions.
Partial F test. Adding a predictor to a model with k predictors changes the residual sum of squares from RSSk to RSSk+1:
F = (RSSk − RSSk+1) / [RSSk+1 / (n − k − 2)]
which is compared with an F(1, n − k − 2) distribution (equivalently, the predictor's t test).
AIC for a Gaussian linear model, up to a constant:
AIC = n · ln(RSS / n) + 2p
where p is the number of estimated parameters. Lower is better, and the penalty 2p discourages needless predictors. BIC replaces 2p with p · ln(n) and is more conservative for n ≥ 8.
Adjusted R².
R²adj = 1 − (1 − R²)(n − 1)/(n − k − 1)
Assumptions and data requirements
- All standard regression assumptions. Linearity, independent errors, constant variance and approximately normal residuals for inference. Check them on the final model.
- Adequate events per predictor. The candidate pool should be small relative to n. Harrell (2015) warns that selection from many candidates with a small sample is especially unreliable.
- Limited collinearity. Highly correlated predictors make the selected one arbitrary; check variance inflation factors.
- Complete cases. Missing values change n across candidate models; handle them before selection.
- A prespecified candidate set. Decide the pool, including any interactions or transformations, before looking at the outcome.
- A validation plan. Selection and testing on the same data are biased; plan to validate by hold-out data or resampling.
Original research example (illustrative)
Background. A researcher wants to explain exam scores among 100 students using four candidate predictors: weekly study hours, class attendance rate, nightly sleep hours and commute time.
Objective and question. Which of the candidate predictors are retained by forward selection using AIC?
Data. n = 100. All values below are invented for teaching.
Step 1: Run forward selection by AIC
| Step | Action | Model | AIC | Decision |
|---|---|---|---|---|
| 0 | Start | Intercept only | 620.4 | — |
| 1 | Add study hours | Study hours | 585.2 | Accept (−35.2) |
| 2 | Add attendance | Study hours + attendance | 574.9 | Accept (−10.3) |
| 3 | Try sleep (best remaining) | Study hours + attendance + sleep | 575.8 | Reject (+0.9) |
Differences: 585.2 − 620.4 = −35.2; 574.9 − 585.2 = −10.3; 575.8 − 574.9 = +0.9. Because adding sleep raises AIC, the procedure stops. Commute time, which would have been tried only if sleep were accepted, is not in the model.
Step 2: Evaluate the final model
Final model: study hours and attendance, R² = .412, so k = 2 and n = 100.
Adjusted R² = 1 − (1 − .412)(99/97) = 1 − .588 × 1.0206 = 1 − .600 = .40.
Overall F(2, 97) = (.412/2) / (.588/97) = .206 / .00606 = 33.98, which is far beyond the 5% critical value of about 3.09, so p < .001.
Step 3: Interpret
The selected model explains about 40% of the variance in exam scores (adjusted R²). Study hours and attendance were retained, and sleep did not improve AIC enough to justify an extra parameter. Three cautions apply. First, the R², F test and p-values above are computed as if this model had been specified in advance, so they are optimistic. Second, a +0.9 AIC difference is small, and a different sample could easily have retained sleep: the selection is not strong evidence that sleep is irrelevant. Third, the result should be checked on new data or by bootstrap resampling of the whole selection procedure, which would show how often each predictor is selected.
How to run stepwise regression with the StatFAS Stepwise Regression Calculator
Open statfas.com/stepwise-regression-calculator.html.
- Prepare your data. One row per observation, one outcome column and one column per candidate predictor, with a header row and numeric codes. Handle missing values and check for outliers first.
- Load the data and choose the outcome and candidates. Exclude IDs and variables that are consequences of the outcome.
- Set options. Direction (forward, backward or both), criterion (p-value thresholds, AIC or BIC) and any entry and removal thresholds.
- Run and review. Read the step-by-step path first, then the final model, then diagnostics.
- Download results. Verify every number and sentence before it enters a manuscript.
statfas-stepwise-regression-interface.pngAlt text: "StatFAS stepwise regression calculator interface before data entry."
Caption: The Stepwise Regression Calculator on StatFAS.com.
statfas-stepwise-regression-example-data.png · Alt: "Exam score and four candidate predictors loaded in the StatFAS stepwise regression calculator."Screenshot 3:
statfas-stepwise-regression-settings.png · Alt: "Direction and criterion settings for stepwise regression in StatFAS."Screenshot 4:
statfas-stepwise-regression-steps.png · Alt: "Step-by-step selection path and final model from StatFAS." Use only genuine captures of the tool.Reading the output
- The selection path. It shows which predictor entered or left at each step and what the criterion did. Mistake: reporting only the final model and hiding that a search took place.
- Criterion differences. An AIC difference below about 2 is a near tie. Mistake: treating a small gap as a clear winner.
- Coefficients. They are biased away from zero for selected predictors, because predictors with chance-inflated estimates are more likely to survive selection.
- Standard errors and p-values. They are too optimistic because they ignore the selection step.
- Adjusted R² and F. Descriptive in-sample quantities; validate to see out-of-sample performance.
- Collinearity. If two predictors are strongly correlated, which one survives may be arbitrary.
APA 7 reporting example (based on the illustrative data)
Forward stepwise selection using the Akaike information criterion was applied to four candidate predictors of exam scores (N = 100): study hours, attendance, sleep and commute time. Study hours and attendance were retained, and the resulting model explained 41.2% of the variance (adjusted R² = .40), F(2, 97) = 33.98, p < .001. Because the model was selected from the same data used to estimate it, coefficients and p-values should be regarded as exploratory.
Essential elements: the candidate predictors and how they were chosen, the direction and criterion, the sample size, the selection path, the final model's coefficients with intervals, and a clear statement that inference after selection is optimistic. If you validated by resampling or on a hold-out sample, report those results.
Common mistakes and troubleshooting
- Presenting the final model as if it had been prespecified. Disclose the selection procedure.
- Using stepwise selection to find "the causes". It selects predictive variables, not causal ones.
- Too many candidates for the sample size. Selection becomes unstable and overfits.
- Entering and removing on the same threshold. A predictor can enter and be removed in a loop; set the removal threshold above the entry threshold when using p-values.
- Ignoring collinearity. Check variance inflation factors and consider combining correlated predictors.
- Dropping a variable of known theoretical importance. Force-in key variables.
- Skipping validation. Use bootstrap selection frequencies, cross-validation or a hold-out sample.
- Different data across steps. Missing values can change n between candidate models and invalidate AIC comparisons.
Limitations and alternatives
The central problem is that the data are used twice, once to choose the model and once to assess it. Smith (2018) summarises the evidence against relying on stepwise methods, and Harrell (2015) argues for prespecified models or penalisation. Alternatives include the lasso and elastic net, which shrink coefficients while selecting, best-subsets with cross-validation, and expert-driven model building. If your concern is the shape of the outcome distribution rather than predictor choice, consider quantile regression. To examine relationships among predictors before selecting, use the correlation calculator, the heatmap guide and the descriptive statistics calculator. For a latent-variable approach to many correlated items, see PCA and the EFA guide.
Frequently asked questions
What is stepwise regression?
An automated procedure that adds or removes predictors one at a time according to a criterion, to arrive at a smaller regression model.
Why is it criticised?
The same data select and test the predictors, which inflates R², biases coefficients and makes p-values too small. Selections can also be unstable across samples.
Should I use AIC or p-values?
Information criteria are generally preferred to p-value thresholds, but neither removes the selection bias.
What should I do instead?
Prespecify the model from theory where possible; otherwise use penalised regression or validate the selection by resampling.
Conclusion
Stepwise regression can be a useful exploratory tool if you treat it as a search and not as a test. State the candidate set, direction and criterion, report the path, check stability, and validate before you trust the result. When you are ready to apply it to your data, the StatFAS Stepwise Regression Calculator provides a browser-based route, and its results should be checked by you before publication.
References
- Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723.
- Harrell, F. E., Jr. (2015). Regression modeling strategies: With applications to linear models, logistic and ordinal regression, and survival analysis (2nd ed.). Springer.
- Heinze, G., Wallisch, C., & Dunkler, D. (2018). Variable selection – A review and recommendations for the practicing statistician. Biometrical Journal, 60(3), 431–449.
- Hocking, R. R. (1976). The analysis and selection of variables in linear regression. Biometrics, 32(1), 1–49.
- Smith, G. (2018). Step away from stepwise. Journal of Big Data, 5, Article 32.