Physiological Measures for Games and VR: Novel Tools and Approaches

Tuesday, September 12, 2023
by Sid Heeg

On August 9, 2023, Dr. Ifi Mavridou and Dr. John E. Muñoz (J&F Alliance, Adjunct) spoke at a panel on the use of physiological monitoring, biofeedback equipment, and tools for VR applications and research. Both researchers are experts in this field, with Mavridou talking about the creation and design of hardware and Muñoz about design and the use of software, they presented their experiences on what these tools can offer for research in games and more. The applications that both Mavridou and Muñoz work with are cutting edge and provide researchers with a toolkit on how they can design and approach their studies. This approach personalizes and tailors the study design not only to make it easier for researchers but also for study participants to jointly design and study immersive experiences.

Mavridou shared her experience in working with EmTeq Labs where she works in the design and creation of extended reality (XR) technologies and wearables. These wearables, much designed like VR headsets, involve biometric equipment that measure how users respond and feel through facial expressions, combining a wide variety of biometric sensors into one device. This would reduce the overall need for excess equipment and wires that often weigh down participants.

Mavridou also noted an overall lack of specialized equipment but a need for it in research. In the past, the types of behaviours and affects of biometric technologies were captured as high and low energy states. The struggle with this equipment is assessing negative and positive states of valance—or how things like approval or disapproval are measured (ie. happy or unhappy). Knowing the valence state can inform researchers on the causation of high or low energy states and what possibly caused that reaction. One of the examples Mavridou brought up was a public speaking training simulation in VR. The participants involved all had a fear of public speaking and through using these types of technologies, researchers could see their physiological reactions and coach the participants through the exercise, helping them build their confidence.

Muñoz opened his remarks by highlighting the importance of physiological signals in research. These signalssuch as pupil response, skin temperature, blood pressure, and much more—are used to extract unconscious player states that can’t be accessed through other methodologies like stress management.

Muñoz provided examples on how researchers can gather this data and the different types of methods that are used in studies that require biofeedback as a measurement (questionnaires, interviews, game metrics, and physiological metrics). He noted the positives and drawbacks for each method. For example, questionnaires are quick and easy to analyze but are often dishonest and it’s hard to convey emotions when filling them out. With Muñoz’s focus primarily being on physiological metrics, he finds that interpreting findings is challenging because of the nature of the data being collected.

To Muñoz, what’s most important is to experiment. No one methodology is perfect but rather, these methodologies are complementary and should be used in conjunction with each other to get the most accurate results. He then focused on how physiological metrics can be used specifically to collective player feedback in games-based studies. He noted several types of body signals that can be captured in studies and the data they capture, including: 

  • Heart rate as an indication of psychological stress; 

  • Eye tracking and pupil behaviour as an indication of cognitive load; 

  • Activity of sweat glands as an indication of physiological stress, and; 

  • Brainwaves measured through EEG as an indication of cognitive states.

Once Muñoz collects this raw data, he runs it through a variety of software tools that assist in extrapolation and interpretations such as iMotions, OpenSignals, and NeuroKit2. The reason why he cycles through various software, is largely due to compatibility issues between the hardware being used for the study and the software used to examine the raw data. One of the solutions to this compatibility problem is part of the work Muñoz does with J&F Alliance and creation of new tools—like the Excite-O-Meter, a Unity friendly tool that integrates the physiological metrics into interactive applications that quantify the responses during and after an immersive experience. This would allow researchers more time to spend with the data and understand the results they are getting from the study as opposed to trouble-shooting.

Stay up to date with Mavridou and Muñoz by following them on social media.