Auditing GenAI–Student Grade Claims on Public Datasets: Nested Controls, Frozen Thresholds, and Claim Labels

Vollständiger Abstract

Worum geht es in dieser Arbeit?

Public generative artificial intelligence (GenAI)–student datasets invite contested links between AI intensity, usage style, and grades, yet many analyses treat predictive accuracy or significant coefficients as sufficient evidence while skipping prior achievement, co-outcome leakage checks, and absolute effect-size thresholds. This paper presents a construct-audit protocol that treats associational claim survival as a reproducible labeling task: a feature-role taxonomy, forbidden-feature gates, nested out-of-fold change-in-R2 materiality thresholds, and operational labels (stable, vanished, artifact-born—the last defined but not positively observed here), with predictive models used as instruments rather than as the scientific product. On the public ai_student_impact_dataset, treated as a construct-audit sandbox (possibly synthetic or engineered; no campus-population or causal claims), a five-seed Ridge-primary run is used to validate those rules rather than to estimate GenAI effects: all eight primary intensity and style claims are non-material under locked absolute gates (AI joint change-in-R2≈0.0064 versus prior grade-point-average lift ≈0.859), while a kitchen-sink OLS significance foil stars 12/19 coefficients that the inventory does not promote. A report-only Random Forest check shows that style and joint-block clearance can depend on the modeling instrument; inventory labels remain Ridge-primary under the locked metric. The protocol can therefore withhold GenAI–GPA claims when absolute gates fail, and a six-step laptop workflow is specified so educational researchers can apply the same checks without reproducing the full validation schedule.