Figure 1. Design-based research mapped to the MESH360 design principles (adapted from McKenney and Reeves 2012). EOR, ecology of resources;
AR, augmented reality; VR, virtual reality; DP, design principle.
ORIGINAL RESEARCH ARTICLE
Claudio Aguayoa*, Cristóbal Dañobeitiab, Thomas Cochranea, Stephen Aielloc, Stuart Cookc and Alberto Cuevasb
aCentre for Learning and Teaching, Auckland University of Technology, Auckland, New Zealand; bEmbodied Group, Santiago, Chile; cParamedicine Department, Auckland University of Technology, Auckland, New Zealand.
(Received: 5 September 2018; final version received: 7 November 2018; Published 28 November 2018)
This paper is based on the second stage of a design-based research (DBR) project encompassing the initial prototyping of virtual reality (VR) simulation in paramedicine education using self-reported and biometric feedback data. In this discussion paper we present the range of reflections and theoretical possibilities that arose from the piloting experience and their implications in redesigning practice in paramedicine education. We focus on the foundational literature and epistemological understandings coming from neurophenomenological cognitive science applied in technology-enhanced learning, using mixed reality (MR) in paramedicine simulation learning as a case. We do so following the logic of a DBR methodological framework, in part demonstrating the usefulness of DBR when reflecting on applied practice to inform newer theoretical developments, leading to further integrated solutions in future practice. In addition, we also put attention on a conceptual shift from a focus on VR, to a focus on MR with emphasis on the associated benefits offered by MR learning situations within paramedicine education. Finally, we discuss the benefits of incorporating self-reported and biometric feedback data in paramedicine education in particular, and in technology-enhanced learning in general, for the design of meaningful learning experiences informed by emotional and physiological responses of learners.
Keywords: mixed reality; biometric feedback; self-report; clinical simulation;
This paper is part of the special collection Mobile Mixed Reality Enhanced Learning, edited by Thom Cochrane, Fiona Smart, Helen Farley and Vickel Narayan. More papers from this collection can be found here.
*Corresponding author. Email: claudio.aguayo@aut.ac.nz
Research in Learning Technology 2018. © 2018 C. Aguayo et al. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.
Citation: Research in Learning Technology 2018, 26: 2150 - http://dx.doi.org/10.25304/rlt.v26.2150
In this section we present the theoretical design process of a mixed reality (MR) learning experience in paramedicine education, informed by participant self-reported and biometric feedback data, which we define as ‘embodied report’ data. This MR learning design is part of the Multiple Environment Simulation Hub (#MESH360) project, which originally was set up during 2016 to explore the use of virtual reality (VR) to enhance paramedicine education, focusing on authentic mobile VR simulation scenarios and exercises (Cochrane et al. 2016). Since its original conceptualisation, the MESH360 project has evolved to now include the exploration of the use of MR learning scenarios informed by embodied reports data. Such a process is fundamented on a neurophenomenological approach to cognition and emotional experience (Dañobeitia 2017; Varela 1996) and follows a design-based research (DBR) methodology to design solutions to practical problems. DBR consists of constant feedback loops between theory, reflection, testing and applied practice, while accounting for the uptake of newer theoretical knowledge, technologies, methodologies and contextual challenges (Amiel and Reeves 2008).
Following the recent attention to the use of immersive technologies in education and professional training simulation, such as VR and augmented reality (AR) (Aguayo, Cochrane, and Narayan 2017), focus has been put on the combination of AR and VR with real environment (RE) situations (i.e. non-digital learning elements) to produce MR learning affordances. The rationale behind this is to maximise the learning affordances offered by the integration of different digital elements of a MR learning scenario, so learners can benefit from a range of learning possibilities available within the digital continuum of MR learning experiences (Jowsey and Aguayo 2017; Milgram and Kishino 1994). Such an approach seeks not only to benefit from the individual learning opportunities offered by the different learning affordances of each independent element present in a MR learning intervention but also to address the unique social, cultural and cognitive particularities of individual learners to produce enhanced learning outcomes across broad audiences (Aguayo 2017).
The research is also informed by the exploration of subjective experiences correlated to empirical biophysiological data coming from recent developments in enactive cognitive science (Froese and Di Paolo 2011; Varela 1996). This permits a better understanding of how users experience a given learning situation based on self-reported subjective data and empirical biometric feedback data (Brouwer et al. 2015). Basically, by assessing self-reported and biometric information of users during a learning experience, it is possible to extrapolate their emotional and physiological responses, that is, their ‘embodied reports’, to specific emotional stimulus (Dañobeitia 2017). This has led us to explore how the use of embodied reports data can inform and improve the redesign of learning scenarios in paramedicine education using MR learning scenarios. For such a data-driven process, DBR is a useful methodology, as constant feedback loops between initial theoretical solutions tested in practice permit the generation of newer practical solutions informed by evolving theory, methodologies and technologies (Cochrane et al. 2017).
The following sections explore the literature surrounding the foundational theoretical frameworks and contexts that underpin the project.
Clinical simulation is a widely accepted and utilised element of healthcare education (Cook et al. 2013; Pike and O’Donnell 2010). Traditional forms of health education simulation involve the use of high-fidelity mannequins for students to practise clinical techniques within a safe environment for themselves in a manner that does not endanger live patients (Cook et al. 2012). The MESH360 project was born out of the need to find more flexible and cost-effective simulation experiences (Zendejas et al. 2013) for paramedicine students who are predominantly working professionals scattered around the country. The goal is to enhance traditional simulation experiences through the use of mobile MR, while also redefining the boundaries of traditional simulation pedagogies that are predominantly competency focused and consequently informed by behaviourist learning theories (Hamstra et al. 2014). We are interested in redefining healthcare education to enable the development of student critical thinking, diagnosis, problem-solving and creative approaches to critical care, and therefore we leverage pedagogies that are informed by learning theories such as constructivism, connectivism, experiential learning and authentic learning, for example heutagogy (Hase and Kenyon 2001, 2007). This led to the development of an implementation strategy that also aims to keep the project costs low and enables rapid and agile development.
We chose a DBR methodology to guide the development and iterative redesign of the MESH360 project. DBR provides a structured iterative four-stage approach to curriculum redesign and critically informs the integration of new technologies into the curriculum through a thorough investigation of the literature as a first stage. Through our initial literature review we identified five key design principles to guide the development of the MESH360 project (Cochrane et al. 2016, 2017).
This was followed by our rapid prototyping and redesign stages of an MR enhanced simulation (Cochrane et al. 2018). We used McKenney and Reeves’ (2012) model of DBR mapped to our identified design principles, as shown in Figure 1.
Figure 1. Design-based research mapped to the MESH360 design principles (adapted from McKenney and Reeves 2012). EOR, ecology of resources;
AR, augmented reality; VR, virtual reality; DP, design principle.
In the following sections we explore the literature informing the integration of embodied reports into the MESH360 project.
‘Cognitive science’ refers to the scientific study of the mind, and it brings together different disciplines such as neuroscience, psychology, artificial intelligence, linguistics, anthropology, philosophy and cybernetics. Cognitive science allows us to understand human and animal behaviour from a multidisciplinary perspective considering biological, cultural, social and philosophical aspects. From a biological perspective, each type of organism, from the simplest virus to the largest tree or the complicated human being, has its own functional physiological characteristics. This refers to the set of underlying physical and chemical mechanisms responsible for the origin, development and progression of life and living organisms.
The science of human physiology, in particular, tries to explain the characteristics and specific mechanisms of the human body that make it a living being (Hall 2017). Human physiological responses, as well as those in many invertebrates, are commanded by the nervous system. The nervous system is anatomically divided between the central and peripheral nervous systems (CNS and PNS). The CNS is made up of the human brain and the spinal cord, and the PNS comprises all nerves that are placed outside the brain and the spinal cord. The SNP comprises somatic, autonomic and enteric nervous systems (Hall 2017). In this regard, it has been recorded that the behaviour of the nervous system during its homeostasis, that is, in its self-regulating function to maintain a stable internal condition against a stimulus, is linked to different emotions and cognitive functions (Guyton and Hall 2003).
Of particular interest for physiological biometric feedback in cognitive science are the two subsystems of the autonomic system, which is divided into the sympathetic and parasympathetic systems. The sympathetic system is supporting the stress response (‘fight or flight’) to threatening events; it is a central component of emotional experience and, by extension, manifests itself at the cognitive and behavioural level (Christopoulos, Uy, and Yap 2016). The sympathetic system, in turn, supports the mobilisation of the body towards appetitive or gratifying stimuli. The typical measurable manifestation of the sympathetic system is the control, mostly involuntary, of physiological responses, such as heart rate (HR), sweating, breathing and eye blinking (Christopoulos, Uy, and Yap 2016).
There are several theories about what is meant by an emotion, so that an exact definition of it remains unfinished. From Descartes to Darwin, the assumption has been that the perception of danger produces a condition, fear, which in turn is expressed in the body, for example as vasomotor discharge and behaviour (perception → emotion → physiology). In this perspective, emotions are expressed bodily; the corporal expression is a manifestation of something ‘internal’, of a non-corporal experience (Haye and Carballo 2017). Tomkins (1962), cited in Ekman (1992), states that emotions are sets of organised responses that ‘are capable, when activated, of simultaneously capturing organs as widely distributed as the face, heart and endocrine, and imposing a specific pattern of correlated responses’ (p. 2).
James’s theory, on the other hand, holds that emotion, far from being an expression of an internal state, is a bodily modification (physiological in a broad sense) that is felt or perceived ‘inside’, during the process of this bodily change (Cannon 1927; Haye an Carbayo 2017). The causal scheme perception → physiology → emotion corresponds properly to the theory of Lange (Schioldann 2001), whose familiarity with James has come to be called ‘James-Lange hypothesis’ from Cannon (1927) (Haye and Carbayo 2017). The James-Lange theory was subject to much criticism, however, until the 1980s: ‘neuroscience research on the role of peptides and receptor cells established that (both) James-Lange and his critics were right in that the emotions originate as much in the corporal changes as in the mental perceptions’ (Coleman and Snarey 2011). This means that both theories are considered correct.
Today in the literature we can find numerous studies where it has been possible to relate the autonomic nervous system (ANS) with emotional states. There is evidence of distinctive patterns of activity of the ANS with emotions (Ekman, Levenson, and Friesen 1983; Kreibig 2010), such as anger, fear, disgust, sadness (Ekman 1992; Levenson, Carstensen, Friesen and Ekman 1991). These correspond to negative emotions, as well as joy, which corresponds to a positive emotion (Ekman, Levenson, and Friesen 1983), and surprise, which does not have a clear valence (Kreibig 2010). These six emotions have been called ‘basic emotions’, and also ‘primary’ or ‘fundamental emotions’, because these emotions are manifested with the same patterns in all cultures, as well as in some animals (Ortony and Turner 1990).
The different emotional states are a reflection of the activity of the nervous system, which can be assessed through different methods. Some methods measure changes in neural activity, such as electroencephalogram (Ciorciari 2012), magnetoencephalography and topography of steady state (Harris, Ciorciari, and Gountas 2018). Other measure changes in metabolic activity, such as functional magnetic resonance imaging, positron emission tomography and functional transcranial Doppler ultrasound (Ahlert, Kenning, and Plassmann 2006; Harris, Ciorciari, and Gountas 2018). On the other hand, biometric measurement methods measure the response of the sympathetic system, which is associated with the control of physiological responses, most of them involuntary. All these different methods, that is, biometric measurement, neural activity measurement and metabolic activity measurement of the neural system, are methods used to study cognitive and affective processes (Harris, Ciorciari, and Gountas 2018).
Relevant sympathetic physiological responses, for the purpose of this article, are heart frequency rate and sweating, as these can be correlated to emotional responses during particular cognitive and affective conditions, such as for example during a learning process. These two physiological responses can be assessed through the measurement of HR and Galvanic skin response (GSR), respectively.
HR represents one of the most promising markers. It is believed that HR can be a great indicator of stress; in fact, it has been reported that salivary cortisol levels, which are an indicator of stress, increase when there is a high HR (Michels et al. 2013). Although there is a wide range of factors that affect HR, the ANS is the most prominent one. The heart is doubly innervated by the ANS so that relative increases in sympathetic activity are associated with increases in HR and relative increases in parasympathetic activity are associated with decreases in HR. Therefore, the relative sympathetic increases cause the time between heartbeats (the interval between beats) to shorten and increase the relative parasympathetic (Thayer et al. 2012).
The most frequent method used to measure the HR is the electrocardiogram, which measures the HR and the electrical wave potential detected by two electrodes, one on each side of the heart (Andreassi 2010). Another way to measure HR is through pulse photoplethysmography, introduced by Hertzman (1938), which is a simple and useful method to measure the relative changes of blood volume in the microvascular bed of peripheral tissues and evaluate the peripheral circulation. This signal is obtained through non-invasive pulse oximetry systems and is based on the absorption of light by the bloodstream (Gil et al. 2010).
Within the study of emotions, it has been observed that there is an increase in the acceleration of the HR against pleasant patterns, and an increase in the deceleration of the HR against unpleasant patterns (Balconi, Vanutelli, and Finocchiaro 2014). It has also been seen that there are variations according to the intensity of the stimulus. Stimuli with low intensity are related to a deceleration of the HR, while intense stimuli have been seen to activate defence responses associated with the acceleration of the HR (Balconi, Vanutelli, and Finocchiaro 2014).
Sweating is an indicator of stress or arousal. An arousal has been conceptually defined as a ‘driving state or an energizer of non-specific behavior, something that describes the intensity of an experience, but not its quality’ (Dillon et al. 2000, p. 2); that is, arousal is an indicator of intensity (Duffy 1962). GSR is a frequent method used to measure the activation of the sympathetic nervous system through sweating. This gives direct information about the electrical conductance of the skin, which is related to the levels of sweating in sweat glands (Gil et al. 2010; LaBarbera and Tucciarone 1995). The neural control of the sweat glands in humans is entirely under the sympathetic control. An increase in sympathetic nervous system activation is used as a large indicator of arousal.
Several studies using the image view paradigm have shown that the GSR is highly correlated with the emotional arousal self-report. GSR can be used as an indicator of different processes such as activation, attention and significance of tasks or affective intensities of a stimulus as experienced by a subject (LaBarbera and Tucciarone 1995; Montagu and Coles 1966; Ohme et al. 2009). However, it is not a good indicator of valence; its great limitation is that the valence of the stimulus cannot be measured – that is, it cannot be said whether it is a positive or negative stimulus (LaBarbera and Tucciarone 1995; Ohme et al. 2009).
For many theorists, an important characteristic of emotions is their coherence of response, variously labelled as coherence of the response system (Ekman 1992), concordance (Nesse et al. 1985; Wilhelm and Roth 2001) or organisation of trends of answers (Lazarus and Lazarus 1991; Levenson 1994). Although there are reports showing a correlation between emotional experience and physiology (Kreibig 2010; Shu et al. 2018; Siedlecka and Denson 2018), there is a small niche studying this, which calls for further explorative work.
In recent decades much importance has been given to the use of first-person methods in cognitive sciences, in particular to the science of consciousness, where it has become a need to ask participants whether they perceived the stimulus presented or not (Kriegel 2013). The Chilean neurobiologist Francisco Varela proposed a study model to study consciousness called ‘neurophenomenology’ (Varela 1996). Neurophenomenology emphasises the importance of collecting data in the first person of phenomenologically trained subjects as a strategy to describe and quantify physiological processes relevant to consciousness. The general approach, at the methodological level, is to obtain richer data in the first person through disciplined phenomenological explorations of experience and to use this original data in the first person to discover new data in the third person through physiological processes crucial for consciousness. Therefore, a central objective of neurophenomenology is to generate new data by incorporating refined and rigorous phenomenological explorations of experience in the experimental protocols of neuroscientific cognitive research on consciousness (Lutz and Thompson 2003).
Self-reporting is one of the most common procedures for collecting data in psychology; it is also an important tool for the social sciences in general. It is possibly the most traditional way to measure emotional experiences in these areas. Strictly speaking, self-report refers to the act of a person giving a detailed account or statement describing their actions or any personal event (Hersen et al. 2004). Yet one of the main problems in accessing the experience of an individual lies precisely in that current methodologies explain the behaviour of subjects through traditional instruments, such as surveys and interviews, which are unsatisfactory in terms of not taking measurements at the same time that they are living the experience of a visual and/or sound content.
This produces a bias because the interviewees must remember past experiences and bring them to the present. In other words, they only collect personal reports and past memories, or projections of possible or hypothetical actions (Babbie 2013). Even more, not all of these methods are capable of generating data because the report is vulnerable to variables that can distort the description of those who are actually experiencing it. Within these variables, it is found that (1) the subject is not willing to do the introspective exercise required to accurately report their experience or (2) the subject feels pressured to respond according to expectations that may differ from their real experience (Dañobeitia 2017).
From the preceding exploration of the literature, we see an opportunity for methodologies that measure the factors that determine the subjective experience of individuals by accounting for their enjoyment, emotional involvement and retention of content, in addition to assessing their behaviour and physiological responses to a given stimuli. From a science of experience perspective, it becomes necessary to know, identify and characterise people according to their needs and tastes when confronted with given content. In this sense, emotional involvement of individuals with content is related to the degree of retention of the same. The more intense the emotional involvement of the participants with content, the more they remember it (Baraybar-Fernández et al. 2017).
Inspired by the enactive neurophenomenological approaches to cognition proposed by Varela (1996), and the epistemological underpinnings derived from the Santiago School of Cognition (Hallowell 2009; Luisi 2016), the Embodied Group from Chile (www.embodiedgroup.com) developed a multimodal data approach methodology (Lahat, Adali, and Jutten 2015), coined ‘embodied reports’, to assess emotional experiences via physiological and self-reported data (Dañobeitia 2017). The embodied reports approach constitutes a methodology to elucidate the level of emotional engagement, based on the subjective experience and differential biological responses of individuals faced with a particular content presentation and/or emotional experience. The embodied reports methodology offers an innovative and technological solution to assessing the emotional response of individuals by providing a non-invasive way of measuring a subjective experience through a physiological and emotional analysis of the participants, pointing to a greater possibility of contrasts compared with other methodologies (Dañobeitia 2017).
At the core of the methodology is the integration of both the self-reported subjective experience of an individual (‘first-person data’) with physiological correlates that reflect autonomic activity that accompanies a subjective state under study (‘third-person data’) (Dañobeitia 2017). For this, the embodied reports methodology uses an expression pedal for obtaining self-reported first person data, and modern wearable physiological sensor devices that are able to measure physiological responses of the body as third-person data, such as GSR and HR.
The expression pedal works like a gas pedal in a motor vehicle, where the subject presses it, obtaining numerical answers from 1 to 100, with 1 meaning a low value and 100 a high value. The use of the pedal allows for a real-time report of the preferences of the subjects, which can be contrasted with the physiological correlates obtained by the wearable sensor devices, thus permitting the researcher to know at each moment what emotional state a particular subject was in. This, complemented with rational reflection and sample statements (subjective elements), allows for the measuring of participants’ experiences rather than rational behaviour from them, thus providing indicative information about the emotional states of individual participants. Therefore, the embodied reports methodology integrates conscious information with the unconscious data of people participating in the measurement process.
The research team identified the synergy between the embodied reports methodology and the MESH360 project, and thus we integrated the embodied reports methodology within the MESH360 DBR project. This resulted in the development of a transdisciplinary DBR team (Table 1).
As shown in Table 1 the research project evolved to incorporate the collaboration of four teams, each with specific expertise required to design and develop the theoretical and practical elements of the project. These included three teams based at the university: discipline experts (paramedic lecturers); academic advisors as educational technology experts; an mobile mixed reality (MMR) development team providing the MMR platform choice and integration advice; and a team of enactive biometric researchers based in Chile (Embodied Group), who visited the main project team for 2 weeks in 2017, providing prototype embodied biometric data collection instruments and methods.
Our initial research questions for the 2017 MESH360 project included:
In this paper we focus upon the 2018 redevelopment of the project to better address the second research question. Participants for the study included volunteers from first year to third year bachelor’s of Health Science undergraduate degree students, as well as invited professional (employed) paramedics, with a total of 32 participants in the first iteration in 2017 and a further 21 participants in 2018. Ethics consent was gained through the university ethics committee, and all participants were supplied with an information sheet and consent forms for the research element of the study, with all participant data anonymised.
During August 2017 we piloted the embodied reports methodology using the MESH360 project in paramedicine education as a case study. The aim of the pilot was to triangulate self-reported subjective student feedback and objective HR and GSR biometric data with student learning, while engaged in a virtual paramedicine scenario. For this purpose, students were presented a critical care 360-degree scenario with hidden points of interests (using a Google Cardboard-compatible head-mounted display) and were asked to explore and identify risks within the virtual environment. The virtual scene represented a building site with multiple potential hazards, including chemicals, exposed electrical and water pipes and people hiding (Cochrane et al. 2018).
The chosen 360-degree online visualisation platform, SeekBeak, allowed for the background recording of the viewing of the hidden point of interests in the form of a ‘hotspot heat map’ (Figure 2a). To record a hotspot in the heat map, students had to stare at a given potential hazard for more than 2 s. Students were also connected to an embodied reports unit recording first-person data through a self-reporting pedal, and third-person data through HR and GSR sensors (Figure 2b). Students were presented the 360-degree scenario for a minute and were asked to increasingly press the pedal when they felt emotionally engaged with the scene. Note that all participants from the pilot study (n = 10) reported not being distracted by the task of pressing the pedal while visualising the 360-degree scenario. However we acknowledge such task may alter the biofeedback response, which calls for future methodological improvements to eliminate and/or isolate this potential issue. Finally, the embodied reports data was processed and plotted against time for a global representation of first- and third-person data sets in relation to the duration of the experience (Figure 2c; for details see Cochrane et al. 2018).
Figure 2. Experimental piloting set-up using an embodied reports unit to record first- and third-person data during a paramedicine virtual reality scenario: (a) hotspot heat map representing the immersive experience of a participant; (b) experimental set-up including the self-reporting pedal (green) and HR and GSR sensors (blue and red, respectively); and (c) analytical representation of embodied reports data sets against time of the virtual reality experience.
The piloting of the embodied reports methodology with the MESH360 project was a data-driven process. Only by measuring and interpreting first- and third-person data across participants with different degrees of contextual awareness knowledge in paramedicine could we make sense of the data in relation to the pedagogical aims of the project. We did expect to be able to inform and redesign consequent practice based on embodied reports findings; yet analytical reflection and discussion sessions between the researchers and practitioners were required first. Once again, the DBR methodology proved to be adequate in accommodating new data-informed findings with prior theoretical guidelines tested in practice during the piloting, which led to reflection to produce a new set of theoretical solutions to be tested in contextual practice (Amiel and Reeves 2008; Cochrane et al. 2017). In the case of the MESH360 project, a whole range of new theoretical avenues emerged from the piloting of embodied reports data in paramedicine education. These are presented next.
The MESH360 piloting during 2017 based on the embodied reports methodology provided a range of emerging methodological considerations for the research team. These principally related to methodological enhancements, hardware and software improvements, characterisation of participants in relation to their stress levels prior to, during and post-event, design considerations in relation to VR and MR, and new research pathways. Table 2 presents a summary of such emerging key points regarding the use of embodied reports data in paramedicine education.
| Domain | Emerging key points |
| Methodology enhancement |
|
| Participant characterisation |
|
| VR/MR experience design |
|
| Emerging research pathways |
|
| HR, heart rate; GSR, Galvanic skin response. | |
The set of key points and considerations that arose from the MESH360 piloting event during August 2017 allowed the research team to reconsider the methodological procedures employed during the piloting. Following the DBR methodology (Figure 1), the research team carried out different reflection and brainstorming sessions to consider how the data-driven experience from 2017 could inform a new methodology and redesign of the original MESH360 2017 intervention, in preparation for the following data collection event to occur during September 2018.
Findings from the MESH360 2017 piloting event, in combination with recent MR literature and experiences (e.g. Akçayır and Akçayır 2017; Jowsey and Aguayo 2017) and VR/MR hardware and associated development tools (e.g. use of OculusGO standalone VR headset), directed the design of a new MESH360 clinical simulation MR experience in paramedicine education. From a methodological point of view, more emphasis was put on determining previous states of stress of participating students through both a prequestionnaire and ‘baseline’ (i.e. pre-VR simulation) biometric feedback recording. This particular aspect emerged from an anecdotal finding from the MESH360 piloting event, where one student presented a relatively high HR (120 bpm) when entering the simulation, yet seemed calm to the lecturers. This led the research team to reflect on the type of stress levels students bring into clinical simulations and how that can affect their learning, and potentially their future clinical practice. This prompted the research team to explore and give attention to prior levels of stress before entering clinical simulation events; and beyond that, to consider longitudinal studies in relation to students’ stress level management throughout their academic journey and the implications that pedagogical practices can have on students.
From a user experience (UX) point of view, and in consideration of theoretical insights underpinning the embodied reports methodology, it became clear that the VR simulation needed to be longer than the original ‘1 min experience’ from the piloting event. This led the research team to implement longer exposures to VR simulation experiences to allow for longer embodied reports data collection. Related to this point, and in consideration of the principle of making experiences as immersive as possible, a shift from a static 360-degree scenario to a dynamic 360-degree scenario (i.e. use of 360-degree video) was adopted. The combination of these UX-related factors, that is, longer exposure to VR content and associated recording of baseline stress levels, led to the implementation of a series of VR scenes composing the VR simulation. The starting VR scenes presented ‘calming’ scenarios (e.g. a forest) aiming to record baseline data, and subsequent scenes presented ‘stressing’ content (e.g. an ambulance ride), permitting recording of individual responses to specific and increasingly stressing stimuli.
In addition, a shift from an isolated VR experience to an MR experience was implemented by combining the VR experience with a ‘RE’ component, as described by Milgram and Kishino (1994) in reference to the digital continuum of MR experiences. The RE component involved clinical simulation work with a high-fidelity simulation mannequin (about 10–15 min long), which presented the symptoms described to students over a ‘radio call’ during the VR ambulance ride. This was also intended to present students with a more authentic environment and was complemented with environmental sounds such as paramedic monitoring equipment sounds and ‘crying people’. This UX continuum between the VR experience and the RE experience is what permitted the implementation of a MR learning simulation. In addition, a mid-MR experience assessment was also implemented by asking students to provide a ‘working diagnosis’ and ‘action plan’ between the VR experience and the RE simulation room work with the mannequin.
Finally, from a research point of view, the MESH360 2017 piloting provided a range of novel research considerations and pathways for the research team, including the investigation of potential correlations between embodied reports data patterns with demographic data of participants; the feasibility of using embodied reports recording equipment during the RE mannequin simulation work (currently limited by the physical nature of the recording equipment); the investigation of cognitive and embodied synchronisation between team crew members and between paramedics and patients (Konvalinka et al. 2011); and research into the ‘appropriate’ levels of stress exposure during clinical simulations, as well as emotional triggers and stress management practices in paramedicine education.
Here we present and discuss the design process of the MR learning experience in paramedicine education, informed by embodied reports data. Findings indicate that the theoretical underpinnings coming from neurophenomenological approaches derived from the Santiago School of Cognition (Varela 1996), informing the embodied reports methodology, and the methodology itself, are adequate to inform the research process related to emotional engagement of students during VR/MR clinical simulation in paramedicine education. The insights provided by the embodied reports methodology during the MESH360 2017 piloting event, in relation to the emotional experiences of paramedicine students during simulation tasks accessed via physiological and self-reported data, were useful in directing the redesign of a new methodology for upcoming events using MR learning scenarios. In addition, the different research pathways that emerged were only accessible through the embodied reports methodology, not only highlighting its usefulness in paramedicine education but also its value in advancing our knowledge in the clinical simulation field using VR and MR. Overall, we believe the embodied reports methodology can advance and improve students’ learning outcomes and professional practice in paramedicine education.
During this process we also valued the usefulness and adequacy of the reflective feedback loops of DBR as a research methodology to investigate innovation in practice, therefore confirming previous conclusions in designing mobile VR learning environments (Cochrane et al. 2017), here tested in relation to mobile MR environments (Cochrane et al. 2018). In particular, the reflection process advocated by DBR, in consideration of findings from the first MESH360 piloting stage, allowed for the redesign of newly theory-informed solutions (Amiel and Reeves 2008; McKenney and Reeves 2012), yet to be tested in practice during the upcoming MESH360 events.
This set of findings permitted the research team to address the MESH360 research questions exploring ways of enhancing clinical simulation learning environments using mobile MR and therefore more authentic and real-world scenarios through technology-enhanced learning. In particular, findings allowed assessment of whether first-person (enactive) and biometric feedback data, that is, embodied reports data, could be used to triangulate and evidence students’ learning using mobile MR scenarios. In this regard, the embodied reports methodology largely exceeded the initial expectations of the MESH360 research team. In this sense we see a real potential for the embodied reports methodology across educational contexts and sectors to inform learning outcome enhancements. However, we also recognise that methodological improvements to the embodied reports equipment and associated tools are still required when working within VR, MR and, by extension, other emerging learning technology affordances such as AR. Future iterations of the research will explore participant feedback and the prototyping of the embodied reports methodology within other clinical simulation contexts beyond paramedicine education.
Based on the initial prototyping use of VR simulation in paramedicine education using embodied reports data equipment (Cochrane et al. 2018), in this discussion paper we have presented the range of theoretical and practical implications that emerged from the piloting experience and their influence in redesigning practice in paramedicine education. The embodied reports methodology (Dañobeitia 2017) is grounded on appropriate epistemological understandings coming from neurophenomenological cognitive science (Lutz and Thompson 2003; Varela 1996), which proved to be applicable within paramedicine clinical simulation based on VR and MR learning experiences. By extension, we also see potential for such theoretical underpinnings to be applied within other technology-enhanced learning situations and contexts (Aguayo 2018), yet this remains to be explored.
Findings also demonstrated the usefulness of a DBR framework for designing mobile VR learning environments (Cochrane et al. 2017), particularly in relation to reflecting on applied practice to inform newer theoretical developments and practical solutions. In our case, such solutions involved a conceptual shift from a focus on VR to a focus on MR, with emphasis on the associated benefits offered by MR learning situations within paramedicine education. Finally, we see clear benefits in incorporating self-reported and biometric feedback data in paramedicine education and potentially across other technology-enhanced learning situations involving the design of meaningful and authentic learning experiences.
Aguayo, C. (2017) ‘AR/VR/MR & RR in learning and teaching digital projects’, THETA 2017 Conference: connecting Minds. Creating The Future, Auckland, New Zealand. (online proceedings with no page number)
Aguayo, C. (2018) ‘Exploring autopoietic principles in technology-enhanced learning’, Scholarship of Technology-Enhanced Learning (SoTEL) 2018 Inaugural Symposium, Auckland, New Zealand.
Aguayo, C., Cochrane, T. & Narayan, V. (2017) ‘Key themes in mobile learning: prospects for learner-generated learning through AR and VR’, Australasian Journal of Educational Technology, vol.33, no. 6, pp. 27–40. doi: 10.14742/ajet.3671
Ahlert, D., Kenning, P. & Plassmann, H. (2006) ‘A window to the consumer’s mind: application of functional brain imaging techniques to advertising research’, in International Advertising and Communication, eds S. Diehl & R. Terlutter, Springer, pp. 163–178.
Akçayır, M. & Akçayır, G. (2017) ‘Advantages and challenges associated with augmented reality for education: a systematic review of the literature’, Educational Research Review, vol. 20, pp. 1–11. doi: 10.1016/j.edurev.2016.11.002
Amiel, T. & Reeves, T. C. (2008) ‘Design-based research and educational technology: rethinking technology and the research agenda’, Journal of Educational Technology & Society, vol 11, no. 4, p. 29.
Andreassi, J. L. (2010) Psychophysiology: Human Behavior and Physiological Response, Psychology Press, New York.
Babbie, E. R. (2013) The Basics of Social Research, Cengage Learning, Belmont.
Balconi, M., Vanutelli, M. E. & Finocchiaro, R. (2014) ‘Multilevel analysis of facial expressions of emotion and script: self-report (arousal and valence) and psychophysiological correlates’, Behavioral and Brain Functions, vol. 10, no. 1, p. 32. doi: 10.1186/1744-9081-10-32
Baraybar-Fernández, A., Baños-González, M., Barquero-Pérez, Ó., Goya-Esteban, R. & de-la-Morena-Gómez, A. (2017) ‘Evaluation of emotional responses to television advertising through neuromarketing’, Comunicar: Media Education Research Journal, vol. 25, no. 52, pp. 19–28.
Brouwer, A. M., Hogervorst, M., Reuderink, B., van der Werf, Y. & van Erp, J. (2015) ‘Physiological signals distinguish between reading emotional and non-emotional sections in a novel’, Brain-Computer Interfaces, vol. 2, no. 2–3, pp. 76–89. doi: 10.1080/2326263X.2015.1100037
Cannon, W. B. (1927) ‘The James-Lange theory of emotions: a critical examination and an alternative theory’, The American journal of psychology, vol. 39, no. 1/4, pp. 106–124.
Christopoulos, G. I., Uy, M. A. & Yap, W. J. (2016) ‘The body and the brain: measuring skin conductance responses to understand the emotional experience’, Organizational Research Methods, vol. 27, no. 1, pp.1–27. doi: 10.1177/1094428116681073
Ciorciari, J. (2012) Bioelectrical Signals: the Electroencephalogram, Physiology, Biophysics, and Biomedical Engineering, p. 539, Boca Raton.
Cochrane, T., et al., (2016) ‘Designing virtual reality environments for paramedic education: MESH360’, ASCILITE 2016 Adelaide, pp. 125–135.
Cochrane, T., et al., (2017) ‘A DBR framework for designing mobile virtual reality learning environments’, Australasian Journal of Educational Technology, vol. 33, no. 6, pp. 27–40. doi: 10.14742/ajet.3613
Cochrane, T., et al., (2018) ‘Designing immersive mobile learning mixed reality for paramedic education’, IEEE-TALE 2018: Education and Technology Conference, In Press.
Coleman, A. E. & Snarey, J. (2011) ‘James-Lange theory of emotion’, in Encyclopedia of Child Behavior and Development, eds S. Goldstein & J.A. Naglieri, Springer, Boston, MA, pp. 844–846.
Cook, D., et al., (2012) ‘Comparative effectiveness of technology-enhanced simulation versus other instructional methods: a systematic review and meta-analysis’, Simulation in Healthcare, vol. 7, no. 5, pp. 308–320. doi: 10.1097/SIH.0b013e3182614f95
Cook, D. A., et al., (2013) ‘Comparative effectiveness of instructional design features in simulation-based education: systematic review and meta-analysis’, Medical Teacher, vol. 35, no. 1, pp. 867–898. doi: 10.3109/0142159X.2012.714886
Dañobeitia, C. (2017) ‘Embodied reports: developing first and third-person methods for a science of experience’, [online] Available at: https://www.researchgate.net/publication/327390873_Embodied_Reports_Developing_First_and_Third-person_Methods_for_a_Science_of_Experience
Dillon, C., et al., (2000) ‘Aroused and immersed: the psychophysiology of presence’, Proceedings of 3rd International Workshop on Presence, Delft University of Technology, Delft, The Netherlands, pp. 27–28.
Duffy, E. (1962) Activation and Behavior, Wiley, Oxford.
Ekman, P. (1992) ‘An argument for basic emotions’, Cognition & Emotion, vol. 6, no. 3–4, pp. 169–200. doi: 10.1080/02699939208411068
Ekman, P. & Friesen, W. V. (1975) Unmasking the Face: a Guide to Recognizing Emotions from Facial Cues, Prentice Hall, Englewood Cliffs.
Ekman, P., Levenson, R. W. & Friesen, W. V. (1983) ‘Autonomic nervous system activity distinguishes among emotions’, Science, vol. 221, no. 4616, pp. 1208–1210. doi: 10.1126/science.6612338
Froese, T. & Di Paolo, E. A. (2011) ‘The enactive approach: theoretical sketches from cell to society’, Pragmatics & Cognition, vol. 19, no. 1, pp. 1–36. doi: 10.1075/pc.19.1.01fro
Gil, E., et al., (2010) ‘Photoplethysmography pulse rate variability as a surrogate measurement of heart rate variability during non-stationary conditions’, Physiological measurement, vol. 31, no. 9, p. 1271. doi: 10.1088/0967-3334/31/9/015
Gilissen, R., et al., (2008) ‘Parent–child relationship, temperament, and physiological reactions to fear-inducing film clips: further evidence for differential susceptibility’, Journal of Experimental Child Psychology, vol. 99, no. 3, pp. 182–195. doi: 10.1016/j.jecp.2007.06.004
Guyton, A. C. & John, E. (2003) Hall, Medical Physiology, 10th edn, Elsevier Health Sciences, Philadelphia.
Hall, J. E. (2017) Guyton E Hall Tratado De Fisiologia Médica, Elsevier, Brasil.
Hallowell, R. (2009) ‘Humberto Maturana and Francisco Varela’s contribution to media ecology: autopoiesis, The Santiago School of Cognition, and Enactive Cognitive Science’, Proceedings of the Media Ecology Association, Media Ecology Association, Saint Louis, MO, pp. 143–158.
Hamstra, S. J., et al., (2014) ‘Reconsidering fidelity in simulation-based training’, Academic Medicine, vol. 89, no. 3, pp. 387–392. doi: 10.1097/ACM.0000000000000130
Harris, J. M., Ciorciari, J. & Gountas, J. (2018) ‘Consumer neuroscience for marketing researchers’, Journal of Consumer Behaviour, vol. 17, no. 3, pp. 239–252. doi: 10.1002/cb.1710
Hase, S. & Kenyon, C. (2001) ‘From andragogy to heutagogy’, ultiBASE Articles, vol. 5, no. 3, pp. 1–10.
Hase, S. & Kenyon, C. (2007) ‘Heutagogy: A child of complexity theory’, Complicity: An International Journal of Complexity and Education, vol. 4, no. 1, pp. 111–118.
Haye, A. & Carballo, M. (2017) ‘James’s theory of emotion and affective turn in psychology/La teoría de las emociones de James y el giro afectivo en psicología’, Estudios de Psicología, vol. 38, no. 1, pp. 115–139. doi: 10.1080/02109395.2016.1268390
Hersen, M., et al., (eds) (2004) Comprehensive Handbook of Psychological Assessment, Volume 3: Behavioral Assessment, John Wiley & Sons, New Jersey.
Hertzman, A. B. (1938) ‘The blood supply of various skin areas as estimated by the photoelectric plethysmograph’, American Journal of Physiology-Legacy Content, vol. 124, no. 2, pp. 328–340.
Jowsey, S. & Aguayo, C. (2017) ‘O-tu¯-kapua (‘what clouds see’): a mixed reality experience bridging art, science, technology in meaningful ways’, Teachers and Curriculum, vol. 17, no. 2, pp. 95–102. doi: 10.15663/tandc.v17i2.166
Kreibig, S. D. (2010) ‘Autonomic nervous system activity in emotion: a review’, Biological psychology, vol. 84, no. 3, pp. 394–421. doi: 10.1016/j.biopsycho.2010.03.010
Kriegel, U. (2013) ‘A hesitant defense of introspection’, Philosophical Studies, vol. 165, no. 3, pp. 1165–1176. doi: 10.1007/s11098-013-0148-0
Konvalinka, I., et al., (2011) ‘Synchronized arousal between performers and related spectators in a fire-walking ritual’, Proceedings of the National Academy of Sciences, 108(20), pp. 8514–8519. doi: 10.1073/pnas.1016955108
LaBarbera, P. A. & Tucciarone, J. D. (1995) ‘GSR reconsidered: a behavior-based approach to evaluating and improving the sales potency of advertising’, Journal of Advertising Research, vol. 35, no. 5, pp. 33–54.
Lahat, D., Adali, T. & Jutten, C. (2015) ‘Multimodal data fusion: an overview of methods, challenges, and prospects’, Proceedings of the IEEE, vol. 103, no. 9, pp. 1449–1477.
Lazarus, R. S. & Lazarus, R. S. (1991) Emotion and Adaptation, Oxford University Press on Demand, Oxford.
Levenson, R. W. (1992) ‘Autonomic Nervous System Differences among Emotions’, Psychological Science, vol. 3, no. 1, pp. 23–27. doi: 10.1111/j.1467-9280.1992.tb00251.x.
Levenson, R. W. (1994) ‘Human emotion: a functional view’, The Nature of Emotion: fundamental Questions, vol. 1, pp. 123–126.
Levenson, R. W., Carstensen, L. L., Friesen, W. V. & Ekman, P. (1991) ‘Emotion, physiology, and expression in old age’, Psychology and aging, vol. 6, no. 1, pp. 28–35. doi: 10.1037/0882-7974.6.1.28
Luisi, P. L. (2016) The Santiago school: autopoiesis and the biologics of life. [online] Available at: https://wsimag.com/science-and-technology/19657-the-santiago-school
Lutz, A. & Thompson, E. (2003) ‘Neurophenomenology integrating subjective experience and brain dynamics in the neuroscience of consciousness’, Journal of consciousness studies, vol. 10, no. 9–10, pp. 31–52.
McKenney, S. & Reeves, T. C. (2012) Conducting Educational Design Research. Routledge, London.
Michels, N., et al., (2013) ‘Children’s heart rate variability as stress indicator: association with reported stress and cortisol’, Biological psychology, vol. 94, no. 2, pp. 433–440. doi: 10.1016/j.biopsycho.2013.08.005
Milgram, P. & Kishino, F. (1994) ‘A taxonomy of mixed reality visual displays’, IEICE TRANSACTIONS on Information and Systems, vol. 77, no. 12, pp. 1321–1329.
Montagu, J. D. & Coles, E. M. (1966) ‘Mechanism and measurement of the galvanic skin response’, Psychological Bulletin, vol. 65, no. 5, p. 261. doi: 10.1037/h0023204
Nesse, R. M., et al., (1985) ‘Endocrine and cardiovascular responses during phobic anxiety’, Psychosomatic Medicine, vol. 47, no. 4, pp. 320–332.
Ohme, R., et al., (2009) ‘Analysis of neurophysiological reactions to advertising stimuli by means of EEG and galvanic skin response measures’, Journal of Neuroscience, Psychology, and Economics, vol. 2, no. 1, p. 21. doi: 10.1037/a0015462
Ortony, A. & Turner, T. J. (1990) ‘What‘s basic about basic emotions?’, Psychological Review, vol. 97, no. 3, p. 315.
Pike, T. & O’Donnell, V. (2010) ‘The impact of clinical simulation on learner self-efficacy in pre-registration nursing education’, Nurse Education Today, vol. 30, no. 5, pp. 405–410. doi: 10.1016/j.nedt.2009.09.013
Schioldann, J. (2001) In Commemoration of the Centenary of the Death of Carl Lange: the Lange Theory of ‘Periodical Depressions’-A Landmark in the History of Lithium Therapy, Adelaide Academic Press, Burnside.
Shu, L., et al., (2018) ‘A review of emotion recognition using physiological signals’, Sensors, vol. 18, no. 7, p. 2074. doi: 10.3390/s18072074
Siedlecka, E. & Denson, T. F. (2018) ‘Experimental methods for inducing basic emotions: a qualitative review’, Emotion Review, vol. 11, no. 1, p. 1754073917749016. doi: 10.1177/1754073917749016
Thayer, J. F., et al., (2012) ‘A meta-analysis of heart rate variability and neuroimaging studies: implications for heart rate variability as a marker of stress and health’, Neuroscience & Biobehavioral Reviews, vol. 36, no. 2, pp. 747–756. doi: 10.1016/j.neubiorev.2011.11.009
Tomkins, S. (1962) Affect Imagery Consciousness: Volume I: the Positive Affects, Springer publishing company, New York.
Varela, F. J. (1996) ‘Neurophenomenology: a methodological remedy for the hard problem’, Journal of Consciousness Studies, vol. 3, no. 4, pp. 330–349.
Wilhelm, F. H. & Roth, W. T. (2001) ‘The somatic symptom paradox in DSM-IV anxiety disorders: suggestions for a clinical focus in psychophysiology’, Biological Psychology, vol. 57, no. 1–3, pp. 105–140. doi: 10.1016/S0301-0511(01)00091-6
Zendejas, B., Wang, A. T., Brydges, R., Hamstra, S. J. & Cook, D. A. (2013) ‘Cost: the missing outcome in simulation-based medical education research: a systematic review’, Surgery, vol. 153, no. 2, pp. 160–176.