Data Availability for Critical Appraisal Tools for Evaluating Artificial Intelligence in Clinical Studies: Dataset for a Scoping Review

Vollständiger Abstract

Worum geht es in dieser Arbeit?

Abstract Background AI is increasingly used in clinical research, generating a growing need for robust critical appraisal tools to evaluate methodological quality, reporting standards, and potential biases. While traditional instruments exist for conventional clinical studies, specific tools designed for AI-based research are still emerging. Objective This dataset accompanies a scoping review that aimed to identify and describe existing critical appraisal tools, reporting frameworks, and bias classification systems applicable to clinical studies using AI, including chatbot-based interventions. Methods We systematically searched MEDLINE, Embase, CINAHL, PsycINFO, and IEEE Xplore from inception to April 2024. Eligible studies included those proposing or using tools for critical appraisal, reporting, quality assessment, or risk of bias in AI-related clinical research. Screening and extraction followed JBI and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) recommendations. Data extraction combined human review with a supervised GPT-4o–based retrieval-augmented generation (RAG) process to enhance transparency and reproducibility. All AI-assisted outputs were verified independently by two reviewers. Results Seventy records were included: 46 reporting guidelines (comprising 26 guides for reporting AI studies, 16 critical appraisal tools, 2 quality assessment instruments, and 2 risk-of-bias tools), 9 bias classification or bias mitigation studies, and 15 chatbot evaluation studies. All datasets, extraction templates, and RAG prompts are publicly available to facilitate validation and reuse. Conclusions This dataset provides a comprehensive overview of critical appraisal and reporting tools for AI-based clinical research. It may support the development of standardized evaluation frameworks and promote transparency in future AI-assisted health studies.