Immersive Web Virtual Environment for Language Learning: Multimodal Interaction and Error Correction Based on Speech, Gesture and Eye Tracking

Vollständiger Abstract

Worum geht es in dieser Arbeit?

In order to solve the problems of existing Web language learning tools, such as single interaction dimension, lagging error correction feedback and insufficient attention perception, this paper designs and verifies a multimodal immersive Web virtual environment optimization scheme which integrates voice, gesture and eye tracking. The core breakthroughs are as follows: (1) a dual-mechanism adaptive fusion algorithm of “attention weight+dynamic threshold” is proposed, which solves the problem of error correction failure of the traditional fixed threshold scheme in high-fluctuation interactive scenarios (such as voice discontinuity, gesture occlusion, attention drift); (2) a four-stage closed-loop control mechanism of perception-decision-feedback-adjustment” is constructed, which realizes the end-side lightweight deployment. Based on WebXR, MediaPipe and other native technologies, the cross-end compatible architecture is constructed, and the comparative experiments on three data sets and 10 typical working conditions show that, compared with the traditional Web platform, the collaborative error correction accuracy of the scheme is 93. 2% (improved by 41%), and the response time is compressed to 0.15 seconds (shortened by 62%). End-side resource occupancy is reduced to 16.8%, and learning efficiency is 4.8 times per minute. The research confirms that the scheme effectively balances the immersion, error correction accuracy and end-to-end adaptability, and provides technical support for the large-scale application of immersive Web language education.