Exploring the Potential of EEG for Real-Time Interactions in

Immersive Virtual Reality

MIKKO KORKIAKOSKI1, THEODOROS ANAGNOSTOPOULOS2, OSKARI RAJALA1,

MIKAEL SARKINIEMI1, MARKUS HIRSIMAKI1, JERE KINNUNEN1, PAULA ALAVESA1,

PANOS KOSTAKOS1

University of Oulu,

Pentti Kaiteran Katu1, 90570, Oulu

Abstract: - Brain-computer interfaces (BCIs) can use data from non-invasive electroencephalogram (EEG) to

transform different brain signals into binary code, often aiming to gain control utility of an end-effector (e.g

mouse cursor). In the past several years, advances in wearable and immersive technologies have made it

possible to integrate EEG with virtual reality (VR) headsets. These advances have enabled a new generation of

user studies that help researchers improve understanding of various issues in current VR design (e.g.

cybersickness and locomotion). The main challenge for integrating EEG-based BCIs into VR environments is

to develop communication architectures that deliver robust, reliable and lossless data flows. Furthermore, user

comfort and near real-time interactivity create additional challenges. We conducted two experiments in which a

in addition, discuss the findings to help improve the design of user research studies that deploy similar EEG-

based BCIs in VR environments.

Key-Words: - electro-encephalography, EEG, interaction, locomotion, user research study, virtual reality

Received: October 29, 2022. Revised: January 21, 2023. Accepted: February 16, 2023. Published: March 27, 2023.

Immersive experiences offered by VR and

augmented reality (AR) are gaining traction in areas

outside gaming and simple simulations [1].

VR devices are physical controllers and hand or

body tracking [4], [5]. In addition, the most

common feedback modalities are visual, audio, and

haptics (vibration). Capturing the user’s attention

using options derived from what is available can be

challenging in six degrees of freedom (6DoF) or

explored the use of multimodal feedback to help

govern the attention of the user while BCIs have

also shown promise with respect to the potential use

correct placement and they also remain susceptible

to inference. Against this backdrop, our

experimental setup consists of a popular head

exposed to closed-loop biofeedback given through a

virtual heads up display (HUD) shown in VR. The

pilot study consisted of a simple task of re-scaling

an object in the VR realm using only the user

generated EEG signals. In the second experiment,

the system architecture was simplified to lower the

latency caused by the EEG data conversion. This

experiment featured two tasks: telekinesis and

teleportation. In the telekinesis task, users were

instructed to move objects to predetermined

had a control group who wore the EEG device,

however their EEG values were not used as a

control mode.

This article presents two experiments, pilot

experiment (see Subsection 3.1) and study

experiment (see Subsection 3.2), in which some of

the controls are bypassed with a commercially

available non-invasive EEG based BCI, enabling

users to have direct agency in the simulated

environment. Strict policy for social distancing was

researcher running the experiments. The protocol

for testing was decided well in advance to keep the

procedure uniform. Still, when conducting testing in

different locations and with a different set of

equipment, the comparability of the results is a

concern [13]. For all testing, the VR equipment and

Muse 2 headsets were the same models, but not the

same exact units. The PCs used for running the

simulations were very light weight, all the

computers were able to run at the intended

analysis a non-parametric Kruskal-Wallis test [14]

was used for results of the questionnaires while the

Fig. 1: User research study progress.

The purpose of this study was to explore the

feasibility of a consumer grade EEG as a controller

in immersive VR and gain some early insights on

user experience (sense of agency and cyber

sickness) when using EEG for locomotion and

interactions. Our findings show that a simplified

approach can be adequate for non-serious use.

architecture of our system, procedure and the

demographics of our test users. In Section 4 we go

through the results and analysis methods of this

mixed methods study. Discussion of the observed

results is performed in Section 5, while the research

effort is concluded in Section 6.

Naturally occurring forms of interaction, that bypass

and physical discomfort [20], [21], [22], brain

activity during navigational tasks [21], and as an

attention enhancement [11].

Although EEG systems are mostly used in

conjunction with other data gathering methods [20],

[21], [22], [23], [24], EEG data has been

instrumental in measuring the cognitive load of

users [23], [24]. Consequently, this body of research

walking etc. [26], [27]. Within this context, prior

VR research has leveraged portable EEG devices as

a noninvasive BCI for evaluating virtual interactions

including locomotion [10], [28]. Similarly, proof-of-

devices and measurement solutions that would

provide usable EEG data as close as possible to real-

mental effort [40]. This suggests EEG can improve

the engagement and focus on the users in VR. In

this research effort we were interested in the little

explored connection between VR sickness and

an even less explored aspect in using EEG as a

mode of interaction, for a sense of agency. We did

this based on the assumption that due to higher

immersion and focus there might be observable

differences in this context of user experience.

This has severely hampered user testing for the

devices and simulations. Covid-19 has also raised

new concerns especially regarding the cleanliness of

the equipment, when conducting experiments where

the devices are passed on from person to person

numerous times in short periods of time [13].

We conducted a user research study to explore the

potential of using EEG data as a controller i.e., a

mode for interaction in immersive VR. The study

experiment (see Subsection 3.2) was preceded by a

frequency in this window is from 1Hz to 128 Hz

with a time resolution of one second. The resulting

connection. This architecture in the pilot experiment

was more complex than in the study experiment (see

Subsection 3.2). This is because in the pilot

experiment equipment consisted of a controller PC,

a Raspberry Pi, a Muse 2, a HDM, and a Polar H10

heart-rate sensor. It was simplified for the actual

study by using Bluetooth to connect Muse 2 to the

PC instead of Raspberry Pi, and adjusting the

calculations for transforming EEG signals into

usable data. The Polar H10 heart-rate monitor was

to the chest using a strap. Oculus Quest was used as

the HMD. The participants were asked to perform

an object scaling task. Three participants, group 1

(G1), were exposed to their data through a virtual

wrist-mounted heads-up display (HUD). The other

three participants in the second group, (G2), were

treated similarly, but without the exposure to the

HUB. Both groups consisted of one female and two

male subjects, aged between 22 to 40 years of age.

All subjects were students with a computer science

Following a short guidance session in the VR play

area, the participants were asked to relax for 60

seconds with their eyes closed while the threshold

values were being collected. The participants in the

first group were asked to open their eyes, and while

focused, instructed to scale the ball. The group was

provided with a virtual wrist-worn HUD, displaying

data from Muse 2 (EEG) and the heart-rate monitor.

The participants in the second group repeated the

same tasks, but without having the HUD data

groups were asked to fill out a questionnaire. The

pilot experiment was conducted at the university

removing the Raspberry Pi and using Lab Streaming

Layer (LSL), thereby Muse 2 could stream the EEG

data to Unity directly. BrainVision LSL Viewer [43]

was used to observe the EEG channels

simultaneously. It was also used in calibrating Muse

participants wore Muse 2 under the HMD, the

calibration phase was as interference-free as

possible.

participants went through the tasks in the same order

they are in this table. The only difference between

experimental and control groups was that the control

group generated fake focus values.

We use a simple way of calculating the focus

values by taking the four data channels from Muse

2, first summing up the channels and then dividing

the sum by four with a frequency of 10Hz. These

averages are then transformed into the focus values

by using standard deviation for every 100 average

from a new batch of data every time could induce

very abrupt changes in the value.

The participants were divided into groups based

on study experiment conditions. Due to the limited

availability of test users, we conducted a within-

subjects experiment in regards to the VR setup,

however the experimental and control groups were

separated. The experimental group tested with

actual focus values that were generated from the

users’ EEG data, while the control group did the

Table 2. Completion of mean focus values of

control and experimental groups. Values were

The VR scenes tested were presented in Table 1,

they are able to move the objects by focusing. In the

Puzzle room, the users used the telekinesis system

to place certain shaped objects to their

corresponding positions. These positions were

indicated by table-shaped pedestals. The pedestals

changed their color to green when the correct object

was placed on them. When all objects were in their

correct places the task was completed. The

Teleportation scene is a hallway with five nodes. In

this scene the users had to use the teleportation

The collected material consisted of EEG data,

completion times, general observations and

form, demographic questions, and questions on

susceptibility to motion sickness. The post

questionnaire (SSQ) [44] and questions about sense

The EEG for the participants was recorded from

mean focus values were <0.050. For completion

Table 3. Statistically significant results on mean

completion times and variance in seconds.

As with the focus values and EEG, the

completion times were collected for all of the tasks.

The only statistically significant results were found

in completion times for Playroom and Timed

Teleportation as well as total completion for the

whole course. The whole course consists of all the

tasks including the Playroom scene, as presented in

Table 3.

After conducting a Kruskal-Wallis test [14] on

Table 4. Statistically significant results from SSQ.

VR. This questionnaire was loosely based on [45]

and the analysis was conducted using a Kruskal-

Wallis test [14]. The purpose of this questionnaire

on agency was to observe differences mainly in the

telekinesis use case. However, the only close to

significant results were found in the Teleportation

task, as presented in Table 5. Concretely,

questionnaire given to users had five specific

questions, such as: (1) “Q1 = I was able to interact

with the environment the way I wanted to”, (2) “Q2

my performance in the teleportation room”.

as a mode of interaction, more sophisticated

solutions may be needed as even in this study

focused, while the control group was able to pass

the tasks with the help of the “fake” focus values

i.e., by chance. In this case it is unfortunate that we

cannot also show statistically significant results

between the experimental and the control group in

the post-questionnaire measuring sense of agency.

In the Teleportation task, only agency had a p-value

of 0.071, with the experiment group faring higher.

The reason for comparing the Teleportation, with

and without a time limit, was to see if it would be

without the time limit was run fist giving the

participants an opportunity to not just learn the route

but also to acquaint themselves better with the

teleport system. Then, when it came to running the

time limited version of the task, the participants

were already aware of how the whole scene worked,

and did not have to learn the teleport mechanics nor

the route. This led to the users scoring better

completion times in the time limited scene. The

original idea for measuring completion times was to

Since the results from SSQ were inclusive, we

cannot say if there were differences between the

groups. However, we did not observe much VR

sickness with the participants to begin with. This is

possibly due to teleportation being the most user-

friendly mode of locomotion in VR [46]. This was

also expected since in the designated tasks, other

than the teleportation, the user did not need to move.

different locations, even with mostly identical

equipment, the comparability of the results is of

and the experiments were run in private. Conducting

the testing like this can have positive and negative

effects on the results. Positive in the way that the

participants might feel more relaxed in the situation,

which could make the collection of biometric data,

such as EEG, more reliable. However, at the same

time there is a chance of acquiescence bias. The

participants were informed their honesty was valued

more than favorable comments, but this source of

bias cannot be completely ruled out. To mitigate this

very simple way of calculating the focus value. We

also noticed that the EEG data recorded from Muse

2 is very sensitive to muscle movement. Muscle

movement, blinking as well as eye movement, had a

noticeable effect on the amplitude of the raw EEG

sent by Muse 2 and it was easily observable from

the recorded data. The method used for calculating

the focus values was also simpler compared to the

one used in the pilot experiment. Using standard

deviation of averages instead of fast Fourier

EEG and the needed calculations for using it in

real-time interactions in VR have their limitations.

simple calculations might be useful for

entertainment purposes, but other solutions are

needed for more serious applications. Muscle

movements on the forehand and around eyes caused

disturbance to the signal and as we specified in the

results the users were able to cause a peak in

measurements by blinking. We can suggest the use

of the current solution in certain types of future

studies such as the intuitiveness levels of EEG

controls in gameplay. Despite the inaccuracy in the

Acknowledgements:

Authors would like to thank the following computer

References:

[3] Oscar Ariza, Gerd Bruder, Nicholas Katzakis, and

[4] Panos Kostakos, Paula Alavesa, Mikko

[5] Max B. Schafer, Kent W. Stewart, Nico Losch,

[7] Marc Philipp Dietrich, Gotz Winterfeldt, and

2021, pp. 523 – 536.

Grogorick, Paul Maximilian Bittner, Maryam

Mustafa, and Marcus Magnor, Immersive eeg:

evaluating electroencephalography in virtual reality,

Jang, J. S. Kim, In-Young Kim, Jang-Han Lee, and

[12] Stephan Hertweck, Desiee Weber, Hisham Alwanni,

Fabian Unruh, Martin Fischbach, Marc Erich

Latoschik, and Tonio Ball, Brain activity in virtual

reality: assessing signal quality of high-resolution

eeg while using head-mounted displays, In

Conference on Virtual Reality and 3D User

Interfaces (VR), Osaka, Japan, 23 March, 2019, pp.

970 – 971.

[13] Anthony Steed, Francisco R. Ortega, Adam S.

Williams, Ernst Kruijff, Wolfgang Stuerzlinger,

Vol. 1, 2010, pp. 1 – 10.

[15] Mathew Turk, Multimodal interaction: A review,

– 195.

[16] Microsoft: Introducing instinctual interactions,

Available online: https://learn.microsoft.com/en-

fundamentals (accessed on 17 October 2022).