In specialty thesis defenses, answers to the jury's statistics questions usually fall into two categories: "I noticed this while working on the topic and resolved it this way" or "I hadn't thought about that." The second answer leaves a trace of doubt over months of work.
The errors below were compiled from actual defenses and reviewer letters. Each one looks small on the surface but produces major consequences during the defense or the publication process.
Error 1: Determining Sample Size Without a Power Analysis
"We looked at the existing patients in our clinic, 60 was enough", this justification is no longer accepted by any jury or reviewer.
A study without a power analysis carries two dangers at once. Either the study includes too few patients and cannot detect a real difference (Type II error), or it includes too many patients with no ethical justification for it.
During the defense, this question inevitably comes up: "How did you determine your sample size?" Every question left unanswered weakens confidence in the methodology.
Error 2: Using a Parametric Test Without a Normality Test
Before deciding on a t-test or ANOVA, the distribution of the data must be examined. If this step is skipped and the data are not normally distributed, the entire analysis becomes invalid.
Even more dangerous: the normality test was performed but the result was not reported. When the jury asks "Why did you choose a t-test?", the answer "It seemed appropriate" is not accepted.
The article must clearly state under what conditions and with what method the normality test was performed, and how its result affected the choice of test.
Error 3: Choosing the Wrong Post-Hoc Test
After ANOVA comes out significant, a post-hoc test is needed to find out which groups differ from one another. But which post-hoc test?
Tukey when variance homogeneity is met, Games-Howell when it is not, Bonferroni for small samples, the researcher who cannot justify this choice struggles during the defense.
A more common error: reaching the conclusion "ANOVA was significant, there is a difference between the groups" without performing any post-hoc test at all. This is a statistically incomplete conclusion.
Error 4: Ignoring Confounding Factors
The difference between two groups came out statistically significant. But does this difference really stem from the intervention, or from the groups differing in age, sex, or comorbidity?
Univariate findings presented without a multivariate analysis are considered a serious methodological weakness, especially in retrospective studies. This is the most frequent reason reviewers give for major revision.
The statement "the groups were similar at baseline" does not solve this problem. Similarity must be demonstrated statistically, and the analysis must be conducted accordingly.
Error 5: Ignoring Missing Data
No clinical dataset is complete. Yet how missing data were handled is often absent from most theses.
In how many patients are which values missing? Is this missingness random or systematic? If missing patients were excluded from the analysis, does the remaining group retain its representativeness?
A thesis written without answering these questions receives the reviewer note "Risk of selection bias not assessed."
Error 6: Confusing the P Value with Clinical Significance
P=0.03 came out. "There is a significant difference between the groups" was written and passed over. But what is the magnitude of the difference? Does this difference mean anything in clinical practice?
Theses that do not report effect size (Cohen's d, OR, absolute risk difference) increasingly draw criticism. In a large sample, a small, clinically unimportant difference can produce statistical significance. A thesis that does not discuss this situation appears to lack methodological depth.
Error 7: Presenting Results Without a Confidence Interval
"OR = 2.4, p = 0.02", this reporting is incomplete. OR = 2.4 (95% CI: 1.1–5.3) and OR = 2.4 (95% CI: 1.9–3.1) tell very different stories.
In the first example the confidence interval is very wide, which shows that the estimate is uncertain. In the second there is a narrow and precise estimate. The p value may be below 0.05 in both, but the clinical interpretation is entirely different.
Every statistic presented without a reported confidence interval means an incomplete picture for the reviewer.
Where Do People Get Stuck Most in This Analysis?
The points researchers struggle with most are these: performing a power analysis retroactively and presenting it with a reasonable justification. Reflecting the normality test result in the write-up by linking it to test selection. Constructing a multivariate model and defending which variables were included and why.
Let us assess together whether your thesis statistics are methodologically defensible. Request a 30-minute free consultation, before or after your defense.