Towards User-Aware VR Learning Environments: Combining Brain-Computer Interfaces with Virtual Reality for Mental State Decoding

User-aware adaptive systems can greatly benefit from brain-computer interface (BCI) technologies. BCIs allow continuous monitoring of users’ mental states and tailoring of the system to their individual skills and needs. We conducted a feasibility study integrating a BCI using functional near-infrared spectroscopy (fNIRS) into a virtual reality (VR) environment for realistic industrial learning scenarios. Using a fNIRS-based BCI allowed us to a) identify learning progress of individuals based on their working memory load across multiple learning sessions and b) investigate the underlying brain patterns. Our results showed a non-linear relationship between task difficulty and brain responses in the prefrontal cortex (PFC). Finally, we were able to draw four major conclusions regarding architecture components and vital research perspectives, to progress towards a vision of user-aware adaptive system design.

CHI EA ‘23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems