Exploring Subjective and Objective Performance of Multimodal Interactions in Different Physical Environments

With the advancement of multimodal human-computer interaction, its application scenarios are expanding beyond leisure and work to complex situations like disaster relief and outdoor exploration. This study focuses on simulating four real physical environments (vibration, noise, low-illumination, and normal) that users may encounter. By combining four interaction channels (gesture, voice, touch, and keyboard), the study evaluates the coupling of different physical environments and interaction modes based on users' subjective and objective performance in completing map zoom tasks. The findings also reveal an interesting phenomenon - despite higher frustration and fatigue levels, the preference for the gesture channel is significantly higher in low-illumination environments. These experimental results provide empirical evidence and guidance for improving multimodal interaction applications in external physical environments and identifying optimal matches between physical environment-channel-task in the interaction process, contributing to the future development of multimodal human-computer interaction.