Change search
ReferencesLink to record
Permanent link

Direct link
Reality-based brain-computer interaction
Umeå University, Faculty of Science and Technology, Department of Computing Science.
2011 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Recent developments within human-computer interaction (HCI) and cognitive neuroscience have come together to motivate and enable a framework for HCI with a solid basis in brain function and human reality. Human cognition is increasingly considered to be critically related to the development of human capabilities in the everyday environment (reality). At the same time, increasingly powerful computers continuously make the development of complex applications with realistic interaction easier. Advances in cognitive neuroscience and brain-computer interfaces (BCIs) make it possible to use an understanding of how the brain works in realistic environments to interpret brain measurements and adapt interaction in computer-generated virtual environments (VEs). Adaptive and realistic computer applications have great potential for training, rehabilitation and diagnosis. Realistic interaction environments are important to facilitate transfer to everyday reality and to gain ecological validity. The ability to adapt the interaction is very valuable as any training or learning must be done at the right level in order to optimize the development of skills.

The use of brain measurements as input to computer applications makes it possible to get direct information about how the brain reacts to aspects of a VE. This provides a basis for the development of realistic and adaptive computer applications that target cognitive skills and abilities. Theories of cognition and brain function provide a basis for how such cognitive skills develop, through internalization of interaction with the current environment. By considering how internalization leads to the neural implementation and continuous adaptation of mental simulations in the brain it is possible to relate designed phenomena in a VE to brain measurements.

The work presented in this thesis contributes to a foundation for the development of reality-based brain-computer interaction (RBBCI) applications by combining VR with emerging BCI methods based on an understanding of the human brain in human reality. RBBCI applications can be designed and developed to interact directly with the brain by interpreting brain measurements as responses to deliberate manipulations of a computer-generated reality. As the application adapts to these responses an interaction loop is created that excludes the conscious user. The computer interacts with the brain, through (the virtual) reality.

Abstract [sv]

Den senaste tidens utveckling inom människa-dator-interaktion (MDI) och kognitiv neurovetenskap har samverkat till att motivera och möjliggöra ett ramverk för MDI med en stabil grund i hjärnfunktion och människors verklighet. Mänsklig kognition anses till allt högre grad vara kritisk beroende av hur människors förmågor utvecklas i den vardagliga miljön (verkligheten). Samtidigt har ständigt kraftfullare datorer gjort det allt lättare att utveckla komplexa applikationer med realistisk interaktion. Framsteg inom kognitiv neurovetenskap och hjärna-dator-gränssnitt (brain-computer interface, BCI) gör det möjligt att dra nytta av en förståelse av hur hjärnan fungerar i realistiska miljöer för att tolka hjärnmätningar och anpassa interaktion i datorgenererade virtuella miljöer (virtual environment, VE). Adaptiva och realistiska datorapplikationer har stor potential för träning, rehabilitering och diagnostik. Realistiska interaktionsmiljöer är viktiga för att underlätta överföring (transfer) till vardagen och för att nå ekologisk validitet. Möjligheten att anpassa interaktion är mycket värdefull eftersom träning och lärande måste ske på rätt nivå för att optimera effekten.

Genom att använda sig av hjärnmätningar som indata till datorprogram blir det möjligt att få direkt information om hur hjärnan reagerar på olika aspekter av en VE. Detta ger en grund för utveckling av realistiska och adaptiva datorprogram som riktar in sig på kognitiva färdigheter och förmågor. Teorier om kognition och hjärnan ger en bas för att förstå hur sådana kognitiva färdigheter utvecklas genom att interaktion med omgivningen internaliseras. Genom att ta hänsyn till hur internalisering leder till ständig utveckling av mentala simuleringar i hjärnan är det möjligt att relatera designade fenomen i en VE till hjärnmätningar.

Det arbete som presenteras i denna avhandling lägger en grund för utveckling av verklighets-baserad hjärna-dator-interaktions (reality-based brain-computer interaction, RBBCI) applikationer genom att kombinera VR med nya BCI metoder, baserat på en förståelse av den mänskliga hjärnan i människans verklighet. RBBCI-program kan designas och utvecklas för att interagera direkt med hjärnan genom att tolka hjärnmätningar som respons på avsiktliga manipulationer av den datorgenererade verkligheten. När programmet anpassar sig till denna respons uppstår en interaktionsloop som exkluderar den medvetna användaren. Datorn interagerar med hjärnan, genom (den virtuella) verkligheten.

Place, publisher, year, edition, pages
Umeå: Department of Computing Science, Umeå University , 2011. , 55 p.
Series
Report / UMINF, ISSN 0348-0542 ; 11.05
Keyword [en]
reality-based brain-computer interaction, virtual reality, reality-based interaction, brain-computer interface, activity theory, the free-energy principle, grounded cognition
National Category
Human Computer Interaction Computer Science
Research subject
människa-datorinteraktion
Identifiers
URN: urn:nbn:se:umu:diva-44904ISBN: 978-91-7459-245-0OAI: oai:DiVA.org:umu-44904DiVA: diva2:423116
Presentation
2011-06-20, MC413, MIT-building, Umeå University, Umeå, 13:15 (English)
Opponent
Supervisors
Available from: 2011-06-15 Created: 2011-06-14 Last updated: 2011-06-15Bibliographically approved
List of papers
1. Using brain imaging to assess interaction in immersive VR
Open this publication in new window or tab >>Using brain imaging to assess interaction in immersive VR
2009 (English)In: Challenges in the evaluation of usability and user experience in reality based interaction / [ed] Georgios Christou, Effie Lai-Chong Law, William Green, & Kasper Hornbæk, Boston, MA, USA: ACM , 2009, 23-27 p.Conference paper (Refereed)
Abstract [en]

We have developed a system where the combination of functional brain imaging (fMRI) and Virtual Reality (VR) can be used to study and evaluate user experience based on brain activation and models of cognitive neuroscience. The ability to study the brain during natural interaction with an (ecologically valid) environment has great potential for several areas of research and development, including evaluation of Reality-Based Interaction (RBI). The RBI concept of tradeoffs is of particular interest since we want to further explore the relation between how the brain works with an accepted reality and what happens when this reality is disrupted. We present the system with an overview of conducted studies to illustrate capabilities and feasibility. In particular, feasibility is supported by the fact that the brain activations seen in these studies match expectations based on existing literature. Further discussion elaborates on the relation to RBI and evaluation; and finally some possible future work is presented.

Place, publisher, year, edition, pages
Boston, MA, USA: ACM, 2009
Keyword
Virtual reality, fMRI, reality-based interaction, VRfMRI, brain imaging
National Category
Computer Science
Identifiers
urn:nbn:se:umu:diva-40038 (URN)978-2-917490-09-9 (ISBN)
Conference
CHI 2009 Workshop, 5 April, Boston, USA
Available from: 2011-02-15 Created: 2011-02-15 Last updated: 2011-12-28Bibliographically approved
2. Effects of interactivity and 3D-motion on mental rotation brain activity in an immersive virtual environment
Open this publication in new window or tab >>Effects of interactivity and 3D-motion on mental rotation brain activity in an immersive virtual environment
Show others...
2010 (English)In: Proceedings of the 28th international conference on Human factors in computing systems, Association for Computing Machinery (ACM), 2010, 869-878 p.Conference paper (Refereed)
Abstract [en]

The combination of virtual reality (VR) and brain measurements is a promising development of HCI, but the maturation of this paradigm requires more knowledge about how brain activity is influenced by parameters of VR applications. To this end we investigate the influence of two prominent VR parameters, 3d-motion and interactivity, while brain activity is measured for a mental rotation task, using functional MRI (fMRI). A mental rotation network of brain areas is identified, matching previous results. The addition of interactivity increases the activation in core areas of this network, with more profound effects in frontal and preparatory motor areas. The increases from 3d-motion are restricted to primarily visual areas. We relate these effects to emerging theories of cognition and potential applications for brain-computer interfaces (BCIs). Our results demonstrate one way to provoke increased activity in task-relevant areas, making it easier to detect and use for adaptation and development of HCI.

Place, publisher, year, edition, pages
Association for Computing Machinery (ACM), 2010
Keyword
Virtual reality, fMRI, Reality-based interaction, VRfMRI, brain imaging, BCI
National Category
Computer Science
Identifiers
urn:nbn:se:umu:diva-40034 (URN)10.1145/1753326.1753454 (DOI)000281276700097 ()978-1-60558-929-9 (ISBN)
Conference
CHI 2010 - ACM Conference on Human Factors in Computing Systems, 10-15 Apri, Atlanta, GA, USA
Available from: 2011-02-15 Created: 2011-02-15 Last updated: 2015-10-09Bibliographically approved
3. Cognitive Training and the Need for Reality-Based Brain-Computer Interaction: Theoretical Background and Potential Applications
Open this publication in new window or tab >>Cognitive Training and the Need for Reality-Based Brain-Computer Interaction: Theoretical Background and Potential Applications
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Computer-aided cognitive training has the potential to be an important tool in the increasingly important fight against dementia and cognitive decline. In order for this potential to be realized a number of challenges must be addressed. This paper presents an overview of the current state of related research with a focus on theoretical results that motivate the use of realistic and adaptive computer applications to address these challenges. This includes consideration of previous research on cognitive training within cognitive neuroscience, recent developments within human-computer interaction (HCI) and previous work relating to brain activity in virtual reality environments. The increasing popularity of reality-based interaction (RBI) and brain-computer interfaces (BCIs) is discussed and the unified concept of reality-based brain-computer interaction (RBBCI) is presented to capture the principles needed for the development of realistic and adaptive systems for cognitive training and to serve as a cornerstone concept for cross-disciplinary development.

Keyword
cognitive training, reality-based brain-computer interaction, brain-computer interface, the free-energy principle, adaptive psychophysiological computing
National Category
Computer Science Human Computer Interaction
Research subject
människa-datorinteraktion
Identifiers
urn:nbn:se:umu:diva-44902 (URN)
Available from: 2011-06-14 Created: 2011-06-14 Last updated: 2011-06-15
4. Tying activity theory to brain function: theoretical foundations for reality-based brain-computer interaction
Open this publication in new window or tab >>Tying activity theory to brain function: theoretical foundations for reality-based brain-computer interaction
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Recent developments within human-computer interaction and cognitive neuroscience point to the need for interaction theory with a solid basis in theories of brain function, and present the opportunity to develop such foundations. The rising popularity of brain-computer interfaces (BCIs) in particular calls out for such theoretical grounding. Activity theory has proved itself capable of supporting human-computer interaction (HCI) research and development over the last decades. Relating activity theory to modern theories of brain function emerges as a desirable goal. The inherent importance of complex realistic contexts in activity theory supports the development of applications combining BCIs with realistic interaction environments. The developmental focus of activity theory is also valuable when constructing applications for training or rehabilitation. In this paper we show how activity theory can be related to modern frameworks for brain function such as grounded cognition and the free-energy principle.

Keyword
activity theory, brain-computer interfaces, reality-based brain-computer interaction, brain function, grounded cognition, the free-energy principle
National Category
Computer Science Human Computer Interaction
Research subject
Computing Science; människa-datorinteraktion
Identifiers
urn:nbn:se:umu:diva-44901 (URN)
Available from: 2011-06-14 Created: 2011-06-14 Last updated: 2011-06-15

Open Access in DiVA

fulltext(3878 kB)602 downloads
File information
File name FULLTEXT02.pdfFile size 3878 kBChecksum SHA-512
8155c3ca569a8a8f1c297ae5049c8a057c21d276d96ee7fd934a3871d7b310143137be506f5ed323846c1959e98bab670b011971e17370f7c57f38fe07405557
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Sjölie, Daniel
By organisation
Department of Computing Science
Human Computer InteractionComputer Science

Search outside of DiVA

GoogleGoogle Scholar
Total: 602 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Total: 441 hits
ReferencesLink to record
Permanent link

Direct link