How many athletes are enough? Small-sample instability of reliability indices and the minimum sample size for a trustworthy minimal detectable change in two-trial sport field tests: a Monte Carlo study

Vollständiger Abstract

Worum geht es in dieser Arbeit?

Abstract Background Test-retest reliability of sport and rehabilitation field tests is usually reported from very small samples, yet how unstable the resulting indices are, and how many participants are needed, is rarely quantified. We used Monte Carlo simulation to evaluate relative (intraclass correlation coefficient, ICC) and absolute (minimal detectable change, MDC₉₅) reliability statistics under sport-realistic small-sample conditions. Methods Two-trial data were simulated for a counter-movement-jump-like measure (mean 35 cm, SD 6 cm) in a fully crossed design: true ICC (0.60–0.90), sample size N (8-100) and a systematic between-trial shift (0–1 cm), 2000 replications per cell. For each dataset we computed ICC(2,1) and ICC(3,1) with 95% confidence intervals and absolute indices (standard error of measurement, typical error, coefficient of variation, MDC₉₅), evaluating bias, relative root-mean-square error (RMSE), interval coverage and qualitative-category instability, each with Monte Carlo standard errors. A trustworthy MDC was defined a priori as relative RMSE ≤ 15% (MDC within ~ 15% of truth; also 20%, 10%, 5%); robustness to heteroscedastic error was tested. Analyses used R. Results ICC(2,1) confidence-interval coverage was nominal (~ 0.95), validating the design. The ICC(2,1) showed small-sample downward bias (up to -0.05 at N = 8, negligible by N = 100). The MDC₉₅ was far more volatile, its relative RMSE falling from ~ 26% ( N = 8) to ~ 13% ( N = 30) and ~ 7% ( N = 100). The minimum N was ~ 15, ~30, ~ 75 and > 100 for the 20%, 15%, 10% and 5% tolerances. The qualitative ICC category was unstable in roughly half of small samples and, at the good/excellent boundary (0.90), remained ~ 50% unstable even at N = 100. A between-trial shift barely affected MDC instability but lowered the absolute-agreement ICC(2,1). The simulated MDC₉₅ curve matched its closed-form value; conclusions held under heteroscedastic error. Conclusions For two-trial reliability studies resembling these conditions, about 30 participants were a practical minimum for a stable MDC₉₅ (15% tolerance), and more for stricter tolerances (model-specific, not universal). A high ICC did not guarantee a stable MDC; our findings suggest reporting absolute-error indices with their confidence intervals and sizing reliability studies around the precision of the MDC that drives individual decisions.