Hypothesis Analysis Suggester Explained: Choose the Right Statistical Test
Choosing the wrong test is one of the most common and most avoidable errors in applied research. A hypothesis such as "students who use the new method score higher" can be tested in several ways, and the right one depends on how the data were collected and measured. This guide explains the decision logic behind choosing a test, walks through a worked example, shows how to read and report the result, and explains where an automated suggester helps and where your own judgement must take over.
What is a hypothesis analysis suggester?
A hypothesis analysis suggester takes a description of your research question and data (for example the outcome type, number of groups and study design) and proposes an appropriate statistical analysis. It does not replace statistical reasoning. It encodes the same decision tree that a statistician would walk through, so that you can start from a defensible choice and then verify it.
When should you use it?
- Planning an analysis before collecting data, so the design supports the test.
- Checking that a test you already selected matches your hypothesis and data type.
- Teaching or learning how study design maps to analysis.
- Finding a non-parametric or alternative option when assumptions look doubtful.
Suggestion versus decision. A suggested test is a starting point. Clustering, repeated measurements, confounding and missing data are design features that a short list of questions cannot fully capture. Treat the suggestion as a draft analysis plan and confirm it against your design.
The logic behind the choice
Four questions narrow the choice for most common designs.
- What is the aim? Compare groups, measure association, or predict an outcome.
- What is the outcome's measurement level? Continuous, ordinal, binary or counts.
- How many groups or predictors? Two groups, three or more, or several predictors.
- Are observations independent or paired? Different participants in each group, or the same participants measured repeatedly.
| Aim and design | Outcome | Typical parametric test | Common alternative |
|---|---|---|---|
| Two independent groups | Continuous | Welch's t-test | Mann–Whitney U |
| Two paired measurements | Continuous | Paired t-test | Wilcoxon signed-rank |
| Three or more independent groups | Continuous | One-way ANOVA (Welch) | Kruskal–Wallis |
| Association of two variables | Continuous | Pearson correlation | Spearman correlation |
| Association of two categorical variables | Categorical | Chi-square test | Fisher's exact test |
| Predict from one or more predictors | Continuous | Linear regression | Robust or quantile regression |
| Predict from one or more predictors | Binary | Logistic regression | Exact or penalised logistic |
The mathematical foundation
For two independent groups with unequal variances, Welch's t statistic is:
t = (M1 − M2) / √(s1²/n1 + s2²/n2)
The degrees of freedom are estimated by the Welch–Satterthwaite approximation:
df = (s1²/n1 + s2²/n2)² / [ (s1²/n1)²/(n1 − 1) + (s2²/n2)²/(n2 − 1) ]
The standardised effect size is Cohen's d, the mean difference divided by the pooled standard deviation. Welch's version is recommended by default because it performs well whether or not the variances are equal (Delacre et al., 2017).
Assumptions to check after the suggestion
- Independence. Observations in one group should not influence observations in another. If participants are nested in classes or clinics, a simple t-test is not appropriate.
- Measurement level. Means and standard deviations assume at least interval-like data. For ordinal scales with few categories, rank-based tests are safer.
- Distribution. For t-tests and ANOVA the sampling distribution of the mean difference should be roughly normal. With moderate sample sizes this is usually satisfied unless there are extreme outliers or strong skew. Inspect plots rather than relying only on a significance test of normality.
- Homogeneity of variance. Welch-type tests relax this assumption.
- Adequate sample size. Small samples have low power; plan the sample size before data collection.
Original research example (illustrative)
Background. An instructor wants to know whether a new active-learning method improves final exam scores compared with traditional lectures.
Hypothesis. Students taught with the active-learning method score differently on the final exam than students taught with lectures (two-sided).
Data. Two independent groups, n = 20 each, outcome scored 0–100. The values below are invented for teaching.
Step 1: Describe the design and outcome
Aim: compare groups. Outcome: continuous. Groups: two. Observations: independent (different students). The decision table points to Welch's independent-samples t-test, with Mann–Whitney U as the sensitivity analysis if the distribution looks unsuitable.
Step 2: Summarise the data
| Group | n | Mean | SD |
|---|---|---|---|
| Active learning | 20 | 72.4 | 8.5 |
| Lecture | 20 | 67.1 | 9.2 |
Step 3: Compute the test
Standard error: √(8.5²/20 + 9.2²/20) = √(3.6125 + 4.2320) = √7.8445 = 2.80. Mean difference = 72.4 − 67.1 = 5.3. So t = 5.3 / 2.80 = 1.89.
Degrees of freedom: 7.8445² / (3.6125²/19 + 4.2320²/19) = 61.53 / (0.6869 + 0.9426) = 61.53 / 1.6295 ≈ 37.8. The two-tailed p for t(37.8) = 1.89 is about .066.
The 95% confidence interval for the difference is 5.3 ± 2.025 × 2.80, i.e. [−0.37, 10.97]. Pooled SD = √((72.25 + 84.64)/2) = 8.86, so Cohen's d = 5.3 / 8.86 = 0.60.
Step 4: Interpret
The observed difference is about 5 points with a medium standardised effect (d ≈ 0.60), but the interval includes zero and p > .05, so these illustrative data do not give clear evidence of a difference. Note the distinction: a non-significant result is not proof of no effect. The confidence interval is compatible with anything from a small negative difference to a large positive one, which suggests the study was underpowered for an effect of this size. A larger sample or a replication would be the sensible next step.
How to use the StatFAS Hypothesis Analysis Suggester
Open statfas.com/hypothesis-analysis-suggester.html.
- State your hypothesis. Write the research question in one sentence and note whether it is directional.
- Describe the outcome and predictors. Identify the measurement level and number of groups or variables.
- Describe the design. Independent or paired observations, and any repeated measures or clustering.
- Review the suggestion. Read the proposed test and any listed assumptions or alternatives.
- Run the analysis. Take the suggestion to the matching StatFAS calculator or your own software, and check assumptions with plots.
statfas-hypothesis-suggester-interface.pngAlt text: "StatFAS hypothesis analysis suggester interface before input."
Caption: The Hypothesis Analysis Suggester on StatFAS.com.
statfas-hypothesis-suggester-example-input.png · Alt: "Two-group exam score hypothesis entered in the StatFAS suggester."Screenshot 3:
statfas-hypothesis-suggester-result.png · Alt: "Suggested statistical test and assumptions shown by the StatFAS suggester." Use only genuine captures of the tool.Reading the output
- The suggested test. This is the main answer. Check that it matches the design you actually have, not the design you wish you had.
- Listed assumptions. Treat them as a checklist. Mistake: assuming they are satisfied because the tool did not warn about them.
- Alternatives. Use a non-parametric or robust alternative as a sensitivity analysis, and report both if they disagree.
- Effect size and interval. Prefer these over a bare p-value (Wasserstein & Lazar, 2016).
APA 7 reporting example (based on the illustrative data)
An independent-samples Welch's t-test compared final exam scores between students taught with active learning (M = 72.4, SD = 8.5, n = 20) and students taught with lectures (M = 67.1, SD = 9.2, n = 20). The difference was not statistically significant, t(37.8) = 1.89, p = .066, mean difference = 5.30, 95% CI [−0.37, 10.97], Cohen's d = 0.60.
Essential elements: the test used and why, descriptive statistics for each group, the test statistic with degrees of freedom, the exact p-value, an effect size and a confidence interval. State whether the test was planned before seeing the data.
Common mistakes and troubleshooting
- Choosing the test after seeing which gives p < .05. Decide the analysis from the design and record it in advance.
- Ignoring pairing or clustering. Repeated measures and nested data need paired or multilevel methods.
- Running many pairwise t-tests instead of one ANOVA. This inflates the Type I error rate; use an omnibus test with appropriate follow-ups.
- Using normality tests as the sole gate. In large samples they flag trivial deviations; in small samples they have little power. Use plots as well.
- Treating a non-significant result as evidence of no effect. Examine the interval and the power.
- Dichotomising continuous variables. Splitting at the median discards information and reduces power.
Limitations and alternatives
A suggester can only work from what you tell it. It cannot detect confounding, measurement problems or a mismatch between your question and your design. For complex designs, consult a statistician. Related guides: stepwise regression and quantile regression for prediction questions, and the descriptive statistics calculator and correlation calculator for first-pass checks.
Frequently asked questions
How do I choose the right statistical test for my hypothesis?
Identify the aim, the outcome's measurement level, the number of groups or predictors, and whether observations are independent or paired. These four facts usually leave one or two reasonable tests.
Should I test normality before choosing a t-test?
Look at plots of the data or residuals first. t-tests are fairly robust with moderate samples and no extreme outliers, and a formal normality test adds little on its own.
Is a suggested test always the correct one?
No. It is a starting point. Clustering, repeated measures and missing data can all change the appropriate analysis.
What if my data violate the assumptions?
Use a robust or non-parametric alternative, transform the data where this is justified, or switch to a model that fits the data structure. Report what you did and why.
Conclusion
A good analysis starts with the design, not the software. Define the aim, the outcome and the structure of your observations, choose the test that fits, verify the assumptions, and report effect sizes with intervals. The StatFAS Hypothesis Analysis Suggester offers a browser-based starting point for that decision, and the suggestion should always be checked by you before it goes into a manuscript.
References
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
- Delacre, M., Lakens, D., & Leys, C. (2017). Why psychologists should by default use Welch's t-test instead of Student's t-test. International Review of Social Psychology, 30(1), 92–101.
- Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, Article 863.
- Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133.
- Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika, 34(1–2), 28–35.