Query-based multimodal interaction and adaptive gated fusion for Alzheimer's disease detection

Vollständiger Abstract

Worum geht es in dieser Arbeit?

Early detection of Alzheimer's disease is crucial for delaying disease progression and improving patient outcomes. Speech has emerged as a promising biomarker for Alzheimer's disease detection because it is convenient, non-invasive, and low-cost. However, existing studies often rely on a single modality or fail to fully exploit complementary information across modalities. To address these limitations, we propose a query-based multimodal interaction and adaptive gated fusion framework for Alzheimer's disease detection. The proposed framework jointly models multi-view acoustic and textual information derived from speech. Specifically, audio and spectrogram features are extracted from preprocessed speech at the segment level, while the full speech recording is transcribed by an automatic speech recognition system to derive textual features. A shared-query multimodal interaction strategy is employed, in which the same set of learnable queries independently extracts task-relevant information from the audio, spectrogram, and textual representations into a unified query space, followed by an adaptive gated fusion strategy that dynamically integrates the resulting query representation with the original modality-specific representations. Experiments were conducted on the ADReSSo and Pitt datasets, comprising speech recordings from participants with Alzheimer's disease and cognitively normal controls. Experimental results demonstrate that the proposed method outperforms existing approaches, achieving accuracies of 87.14% and 90.20% on the ADReSSo and Pitt datasets, respectively. Furthermore, ablation studies and representation visualizations confirm the effectiveness of the proposed interaction and fusion mechanisms. These findings suggest that the proposed framework effectively exploits complementary multimodal information and learns more discriminative representations for Alzheimer's disease detection, highlighting its potential for speech-based cognitive impairment screening.