Lasso Regression Explained: How to Run, Read and Report It
Lasso (least absolute shrinkage and selection operator) regression shrinks coefficients like ridge regression, but its penalty can set some coefficients exactly to zero. It therefore performs estimation and variable selection in one step, which makes it popular when you have many candidate predictors and expect only some to matter.
What is lasso regression?
Lasso adds an L1 penalty to the least-squares criterion (Tibshirani, 1996):
β̂lasso = argmin [ (1 / 2n) Σ(yi − xi′β)² + λ Σ|βj| ]
(The 1/2n scaling is that of the scikit-learn and glmnet convention; other software scales λ differently, so λ values are not comparable across programs.) As λ increases, more coefficients are driven exactly to zero. Predictors are standardised first so that the penalty is applied fairly, and the intercept is not penalised. λ is chosen by cross-validation (Friedman et al., 2010).
When should you use it?
Use it when you have many predictors relative to observations, want a sparse, interpretable model, or need to reduce overfitting. Compare it with ridge regression, which keeps all predictors, and with ordinary multiple regression when predictors are few and well understood.
Assumptions and caveats
- Linear-model assumptions apply, and predictors must be standardised.
- Among highly correlated predictors lasso tends to pick one somewhat arbitrarily, and the choice can change with a different sample. Elastic net (an L1 + L2 mix) is more stable in that case.
- Selected variables are not guaranteed to be the true causal ones, and classical p-values and intervals are not valid after selection. Post-selection inference methods or bootstrap-based stability checks are needed.
Original example (illustrative)
Question. Which of eight candidate measures (x₁ to x₈) predict a performance score? Data for n = 40 (invented). In this simulated dataset I set only x₁, x₂ and x₃ to have real effects, with x₂ moderately correlated with x₁, and the others have none; real data come without that knowledge.
Show the full illustrative dataset (CSV, copy into the tool)
x1,x2,x3,x4,x5,x6,x7,x8,score 51.3,49.5,56.4,51.0,44.6,53.6,63.0,59.5,65.8 43.0,35.5,43.8,50.4,26.7,47.8,37.5,42.7,38.8 44.6,44.5,54.1,60.4,48.7,63.7,43.3,53.5,47.1 59.0,55.3,42.6,40.8,45.4,52.2,39.9,47.9,69.8 48.4,53.9,52.1,53.6,43.5,48.7,57.8,64.9,67.6 37.4,56.9,63.5,57.8,52.6,46.9,64.6,69.6,58.2 68.0,70.4,53.6,37.9,50.0,56.6,37.1,54.0,92.6 54.3,58.2,38.2,43.4,45.6,38.3,67.4,45.0,61.2 53.3,49.4,65.8,63.2,56.3,28.0,50.5,56.8,70.1 60.0,49.6,68.2,36.8,43.4,59.4,50.5,70.0,75.3 51.9,45.4,46.2,39.1,37.2,56.3,55.8,62.9,59.0 42.5,60.9,47.1,65.7,45.7,42.6,52.5,60.3,49.4 51.6,45.7,36.6,36.0,55.0,59.9,48.4,39.3,56.7 58.7,43.2,42.9,56.2,27.5,53.9,44.2,51.1,59.2 49.2,51.4,56.9,42.4,64.2,57.3,58.4,61.6,65.4 57.9,61.3,50.8,35.7,48.6,42.3,35.8,52.6,72.2 44.3,38.2,39.6,52.7,53.6,63.2,49.9,60.4,45.0 64.0,67.0,26.3,62.3,53.4,54.2,53.7,53.8,65.8 53.2,48.5,31.0,48.9,42.0,60.8,47.1,50.8,60.5 41.5,41.3,49.9,35.1,53.0,48.9,38.1,26.0,36.9 55.1,50.0,44.7,47.6,68.2,49.5,50.9,35.1,56.0 66.5,66.2,60.7,50.5,59.2,53.7,56.1,48.5,90.8 35.3,51.6,30.7,47.6,48.0,39.6,56.1,48.0,47.8 45.6,52.3,45.2,63.9,53.5,45.3,30.6,36.9,57.0 60.9,55.0,47.2,66.4,37.2,44.1,45.3,55.9,70.0 43.4,41.3,33.9,57.3,58.1,45.2,51.6,37.1,33.1 45.3,59.6,51.4,73.1,42.1,55.8,48.0,55.7,61.0 49.9,45.1,41.3,80.7,49.2,29.8,43.5,56.8,53.0 45.0,59.3,60.0,48.5,45.3,40.0,43.0,35.3,75.8 62.0,69.9,37.4,38.2,32.3,40.4,18.9,38.6,80.8 63.0,53.5,58.5,45.1,67.6,52.0,46.2,75.5,68.4 46.8,37.8,52.0,49.6,60.7,40.8,58.0,58.5,62.8 43.3,48.1,41.7,73.5,43.0,45.5,39.3,46.5,46.7 49.9,56.7,43.9,48.1,35.8,41.7,77.6,60.4,55.9 42.2,34.5,40.2,49.8,50.3,42.6,37.1,64.2,48.1 54.5,49.0,47.8,44.7,20.6,51.2,39.3,40.0,62.2 43.6,53.2,38.3,35.7,56.4,57.5,40.4,55.6,48.7 47.1,51.2,37.4,58.3,62.0,56.4,55.6,12.3,53.7 52.6,51.1,48.5,43.7,50.6,54.1,47.4,45.4,67.4 62.3,46.5,60.3,51.8,42.0,47.1,40.8,56.8,79.6
Five-fold cross-validation selects λ = 0.774 (scikit-learn scale). The table shows standardised coefficients for OLS and for lasso at this λ.
| Predictor | OLS | Lasso |
|---|---|---|
| x1 | 7.08 | 6.86 |
| x2 | 4.85 | 4.32 |
| x3 | 4.59 | 4.04 |
| x4 | −1.13 | −0.21 |
| x5 | −0.35 | 0 (dropped) |
| x6 | −0.77 | 0 (dropped) |
| x7 | −0.27 | 0 (dropped) |
| x8 | 0.34 | 0 (dropped) |
| λ | Non-zero |
|---|---|
| 6.189 | 2 |
| 1.547 | 3 |
| 0.774 (CV choice) | 4 |
| 0.193 | 8 |
Lasso keeps 4 predictors (x1, x2, x3, x4). Cross-validated RMSE is 6.11 for lasso and 7.23 for OLS with all eight predictors; the in-sample R² of the lasso fit is 0.83 (in-sample, so optimistic).
Interpretation. The lasso keeps the three predictors that were given real effects (x₁ to x₃) and also retains x4, which has no true effect (a false positive that selection methods can produce), and drops the other 4 predictors, with coefficients shrunk toward zero compared with OLS. The shrinkage is deliberate bias that buys lower variance. Retention of a variable means it helped predict held-out data under this penalty, not that it has a proven effect, and the selected set would likely change somewhat in a new sample. Treat the output as a data-driven shortlist to be checked in new data.
How to run lasso regression with StatFAS
Open the Lasso Regression Calculator. Arrange one outcome column and several predictor columns.
Understanding the output
- Non-zero coefficients: the selected predictors and their shrunken sizes.
- λ (and its scale): report with the selection method.
- Coefficient path: shows the order in which predictors enter as λ decreases.
- Cross-validation curve: minimum error and the one-standard-error choice (a sparser model within one SE of the minimum; here the SE is about 8.75 on the MSE scale).
- Out-of-sample error: the proper measure of performance.
APA 7 reporting example
Lasso regression with standardised predictors and λ = 0.774 (five-fold cross-validation; scikit-learn scale) retained 4 of 8 predictors (x1, x2, x3, x4). Cross-validated RMSE was 6.11 for lasso and 7.23 for the full OLS model.
Common mistakes and troubleshooting
- Not standardising predictors before penalisation.
- Reporting p-values or classical CIs for selected variables as if selection had not occurred.
- Interpreting selection as causal importance.
- Over-trusting one split. Use repeated cross-validation or stability selection to see how stable the chosen set is.
- Applying lasso to a handful of well-motivated predictors where theory-driven regression is more transparent.
Limitations and alternatives
With groups of correlated predictors, elastic net is usually preferable. If you do not need selection, ridge regression often predicts as well. For theory-driven entry of predictors use hierarchical regression; stepwise methods are generally less reliable than penalised regression (stepwise regression). See also the regression overview.
Conclusion
Lasso offers selection and shrinkage together. Standardise, tune λ by cross-validation, evaluate on held-out data, and treat the selected variables as hypotheses rather than established effects.
1.
statfas-lasso-regression-interface.png, alt: "StatFAS Lasso Regression Calculator interface"2.
statfas-lasso-regression-example-data.png, alt: "Example data entered in the StatFAS Lasso Regression Calculator"3.
statfas-lasso-regression-settings.png, alt: "Lasso Regression Calculator settings in StatFAS"4.
statfas-lasso-regression-result.png, alt: "Lasso Regression Calculator results generated in StatFAS"Frequently asked questions
Does lasso regression give p-values?
Not valid ones in the usual sense. Standard errors and p-values do not account for the selection step. Use post-selection inference methods, bootstrap stability checks or validate the selected model on new data.
When should I use lasso instead of ridge?
Use lasso when you expect only a subset of predictors to matter and want a sparse model. Use ridge when many predictors each contribute a little or when predictors are strongly correlated and you want to keep all of them.
What is the one-standard-error rule?
It chooses the largest λ whose cross-validated error is within one standard error of the minimum, giving a simpler model with similar predictive performance.
References
- Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B, 58(1), 267–288.
- Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1–22.
- Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning (2nd ed.). Springer.