ORIGINAL RESEARCH ARTICLE

Exploring mobile mixed reality in healthcare higher education: A systematic review

Todd Stretton^a*, Thomas Cochrane^b and Vickel Narayan^b

^aDepartment of Physiotherapy, School of Clinical Sciences, Faculty of Health and Environmental Sciences, Auckland University of Technology, Auckland, New Zealand; ^bCentre for Learning and Teaching, Auckland University of Technology, Auckland, New Zealand.

(Received: 15 August 2018; final version received: 8 October 2018; Published 27 November 2018)

Abstract

Background: The evolution of technology and simulation has had a significant impact on clinical education. However, it remains grounded in traditional teaching paradigms, limiting potential for enhanced learning. Furthermore, the impact of mixed reality enabled mobile devices remains underexplored.

Purpose: The aim of this article was to investigate mobile learning and mixed reality in healthcare higher education.

Method: A search of six databases from the earliest available date to 30 February 2018 and a hand search of journals and included studies was performed. Inclusion criteria focused on ‘healthcare’, ‘higher education’, ‘mobile learning’ and ‘mixed reality’. All study designs were included, though they were limited to the English language. The checklist of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis was used as a framework for the review, with included studies critiqued using the mixed methods appraisal tool.

Results: The search generated 1484 studies, with 18 meeting inclusion criteria. The majority of studies utilised mobile mixed reality (mMR) for teaching procedural skills with established mobile platforms; anatomy; and clinical assessment. mMR demonstrated benefits in skill competency and knowledge scores when compared to control. Users were favourable towards future use of mMR.

Conclusion: While mMR successfully delivered some clinical skills; the pedagogical impact of engagement with higher order clinical reasoning remains a challenge for future studies.

Keywords: mobile learning; medical; clinical skills; technology enhanced learning; pedagogy.

This paper is part of the special collection Mobile Mixed Reality Enhanced Learning, edited by Thom Cochrane, Fiona Smart, Helen Farley and Vickel Narayan. More papers from this collection can be found here.

^*Corresponding author. Email: Todd.stretton@aut.ac.nz

Research in Learning Technology 2018. @ 2018 T. Stretton et al. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.

Citation: Research in Learning Technology 2018, 26: 2131 - http://dx.doi.org/10.25304/rlt.v26.2131

Introduction

Pressures on healthcare higher education (HHE) has led to exploration of different ways to engage students. Pressures include a rise in student cohort numbers to meet industry demand with corresponding reduction in staff to student ratio, classroom space and resources (American Association of College of Nursing 2017). Concurrently, students find it increasingly difficult to balance study, work and life, especially with curricula that are delivered at a certain time, place and mode (Sauve et al. 2018).

The evolution of information technologies has been advocated to assist the pressures on HHE (Houghton et al. 2014). However, despite the advent of clinical simulation, curriculum remains predominantly embedded in behaviourist and teacher-directed learning rather than a student-directed and constructivist pedagogy to develop clinical reasoning and a more active role in learning (Houghton et al. 2014). As access to HHE evolves with the development of low-cost technology and use of high-fidelity clinical simulation, educators are encouraged to consider pedagogy that embraces student-directed and technology-enabled learning (TEL). While no single mode of TEL will meet the needs of all students, a multitude of methods should be considered (Mayer 2014).

Mixed reality, mobile learning and mobile mixed reality

According to Milgram and Kishino (1994), mixed reality (MR) is the convergence of the real and virtual worlds. Mixed reality exists on a reality–virtuality continuum, with augmented reality (AR) and augmented virtuality (AV) in between (Figure 1). Fidelity in HHE has been defined as ‘the degree to which a simulation looks, feels, and acts like a human patient’ (Hamstra et al. 2014). The perceived need for high fidelity (closer to real environment) has been regularly debated, though it is limited by expense, on-site equipment and training (Zendejas et al. 2013). However, the value of low fidelity can be advocated for (Hamstra et al. 2014) as long it maintains a relation between functional aspects of the simulation (e.g. tactile, visual and auditory features) and the human functional factors for education (e.g. context and complexity of task, stage of student learning and appropriate pedagogy). Mixed reality offers the opportunity to rethink and design innovation and clinical simulation spaces for learning and teaching in HHE (Magana 2014).

Figure 1. Adapted from Milgram’s reality–virtuality continuum.

Mobile devices – in particular, smartphones – are increasingly becoming embedded into the socio-cultural activities of modern life (International Telecommunication Union 2015) as sensors and communication technology enable location of affordances nearby and the student’s social network (Olsson et al. 2013). Mobile learning (mLearning) has become de rigueur in secondary education (OECD 2015) and is progressively being integrated into higher education to enable flexibility and student-directed learning while providing social interactions using personal mobile devices (Bujak et al. 2013; Crompton 2013). Mobile mixed reality (mMR) has been documented to improve competency of physical tasks, support memory-related activities and develop situated cognition within social, cultural and physical contexts in education (Sharples 2002).

The aim of this review is to explore the integration of mMR informed by learning theory and new pedagogical strategies in HHE. Specifically, it investigates the possibilities presented by the rise of mLearning and MR to enable new pedagogical strategies in HHE.

Methods

This systematic review followed the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) (Moher et al. 2009). A comprehensive literature search was performed on the use of mMR in HHE in the following health and education databases: Scopus; PsychINFO (OVID); ERIC (OVID); Medline (OVID); Web of Science and AMED (OVID). Search terms used to identify articles included the following: ‘higher education’, ‘healthcare’, ‘mobile learning’ and ‘mixed reality’ (Appendix 1). While ‘extended reality’ was not included in the search terms, subsets along the reality–virtuality continuum as identified by Milgram and Kishino (1994) were included in the search. Publisher sites were searched for relevant educational technology journals (Appendix 2), and reference lists of included studies were hand-searched. Databases were searched from the earliest available date to 30 February 2018, with only English-language articles included.

Study selection

Studies were imported into EndNote software (Clarivate Analytics) and duplicate articles removed. Two assessors performed a first-level selection based on titles and abstracts by following the inclusion and exclusion criteria independently.

Population–intervention–comparison–outcome (PICO) inclusion criteria were as follows: population – HHE (i.e. undergraduate healthcare education), including, though not limited to, medicine, nursing, physiotherapy, paramedicine, podiatry, midwifery, oral health and occupational therapy; intervention – MR-capable mobile devices (such as mobile phones, tablets or non-tethered head display units; HDUs) for teaching and learning; and outcome – the impact on clinical skill competency, which was defined as measurement of clinical skill ability as determined by a competency checklist. The secondary outcome was reporting of mMR usability. Exclusion criteria were the following: primary or secondary education (Year 1–13) settings, postgraduate or clinical settings; use of laptops, tethered HDUs or simulation suites; did not include AR, AV or virtual environment as a mode of teaching; or were not original empirical study designs.

The full texts of all the included or potentially relevant articles were imported into EndNote. A second-level selection of these articles were performed by the same two assessors independently by reading the full text of the articles and following the same inclusion and exclusion criteria. Inter-rater reliability percentage agreement at second-level selection was 91.30%. Discrepancies were resolved by consensus or when needed by a third reviewer, both at the first and second levels of selection. A PRISMA flowchart describing the number of studies included and excluded at the first and second levels, as well as the reason for their exclusion at the second level of selection, were prepared (Figure 2).

Figure 2. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) flowchart of studies.

The quality of included studies was determined by the mixed methods appraisal tool (MMAT) (Hong et al. 2018). The MMAT is designed for systematic reviews that include qualitative, quantitative and mixed-methods studies and is efficient as it allows the use of one tool for concomitantly appraising the most common types of empirical studies. After responding to two screening questions, each included study was rated in the appropriate category of criteria as either ‘yes’, ‘no’ or ‘can’t tell’. While calculation of an overall score is discouraged, the MMAT provides a more detailed presentation of the ratings of each criterion to better inform the quality of included studies. The MMAT is based on constructionist theory and has already been used by more than 100 systematic mixed study reviews (http://mixedmethodsappraisaltoolpublic.pbworks.com/). The MMAT inter-rater reliability between the two raters was 99.96% (Appendix 4).

Data collection process

Qualitative data were extracted from the included studies by one reviewer and cross-checked by a second reviewer. As quantitative data was not being collected or analysed, this review did not include items 10–16 on the PRISMA checklist.

Results

The search returned 1480 papers, with four additional papers included after hand-searching appropriate journal publication sites and source references (total 1484 articles). After duplicates were removed, 961 papers were available. After titles and abstracts were screened, 938 papers were excluded. Following the review of 23 papers retrieved for full text evaluation, an additional five studies failed to meet the inclusion criteria; therefore, 18 papers were included in the review. For each included study, study design, healthcare discipline (participants), intervention, measure and key findings were extracted (Table 1). See Appendix 3 for a summary of papers excluded after review of the full text.

Quality

A variety of study designs were employed in the included studies – qualitative studies (n = 1); randomised controlled trials (RCTs) (n = 5); non-RCTs (n = 1); quantitative descriptive studies (n = 3); mixed methods (n = 7); and design-based research (DBR) (n = 1). All studies that included qualitative design performed well on the MMAT; however, those that included RCT in their mixed methods were more variable, with only three meeting all criteria in the MMAT (Aebersold et al. 2018; Cowling and Birt 2018; De Oliveira et al. 2013). The purpose of the four descriptive studies was to establish validity of the TouchSurgery® modules (Khelemsky, Hill, and Buchbinder 2017; Kowalewski et al. 2017; Morone et al. 2017; Sugand, Mawkin, and Gupte 2015), therefore they included survey and cross-sectional analytic study in their design.

Participants

The included studies comprised 1009 participants – 300 in the experimental groups and 318 in the control groups of the RCTs; 149 novices and 82 experts in the cross-sectional studies; and 160 in the cohort study. Not all studies identified the participant age and/or gender. Two studies included undergraduate nursing students (Aebersold et al. 2018; Garrett, Jackson, and Wilson 2015); 12 included medical students (Albrecht et al. 2013; Amer et al. 2017; Bartlett et al. 2017; De Oliveira et al. 2013; John et al. 2015; Kowalewski et al. 2017; Küçük, Kapakin, and Göktaş 2016; Moro et al. 2017; Morone et al. 2017; Noll et al. 2014, 2017; Sugand, Mawkin, and Gupte 2015); 3 included paramedicine students (Birt et al. 2017a, 2017b; Cowling and Birt 2018) and 1 included dentistry students (Khelemsky, Hill, and Buchbinder 2017).

Intervention

Common skills included

mMR was utilised for procedural clinical skills, assessment skills and anatomy knowledge. Twelve studies focused on clinical skills, with the majority focused on catheter or scope insertion (Aebersold et al. 2018; Bartlett et al. 2017; Birt et al. 2017a, 2017b; Cowling and Birt 2018; De Oliveira et al. 2013); surgical procedures (Amer et al. 2017; John et al. 2015; Khelemsky, Hill, and Buchbinder 2017; Kowalewski et al. 2017; Morone et al. 2017); familiarisation with general clinical skills and equipment (Garrett, Jackson, and Wilson 2015); wound and dermatology management (Albrecht et al. 2013; Noll et al. 2014, 2017); and aiding clinical anatomy knowledge (Küçük, Kapakin, and Göktaş 2016; Moro et al. 2017). One study used mMR for augmenting a respiratory failure case scenario (Garrett, Jackson, and Wilson 2015).

Mobile device included

Some mobile devices required Internet connectivity to download the mMR application while others were preinstalled prior to the study commencing. The majority of studies included iPhone and/or Android mobile phones (Albrecht et al. 2013; Amer et al. 2017; Bartlett et al. 2017; Birt et al. 2017a, 2017b; Cowling and Birt 2018; De Oliveira et al. 2013; Garrett, Jackson, and Wilson 2015; John et al. 2015; Khelemsky, Hill, and Buchbinder 2017; Küçük, Kapakin, and Göktaş 2016; Morone et al. 2017; Noll et al. 2014, 2017), with some also including tablets alongside the phones or as the sole means of mMR. The iPad was mentioned in the majority of tablet-led studies (Aebersold et al. 2018; De Oliveira et al. 2013; Garrett, Jackson, and Wilson 2015; John et al. 2015; Khelemsky, Hill, and Buchbinder 2017; Kowalewski et al. 2017; Küçük, Kapakin, and Göktaş 2016; Noll et al. 2014, 2017), while others used the iPad alongside other tablets or did not specify the brand or type (Garrett, Jackson, and Wilson 2015; Khelemsky, Hill, and Buchbinder 2017; Küçük, Kapakin, and Göktaş 2016; Moro et al. 2017). Google Cardboard or ColorCross were also used with mobile phones in three studies (Birt et al. 2017a, 2017b; Cowling and Birt 2018).

Software

Three studies (Albrecht et al. 2013; Noll et al. 2014, 2017) evaluated a dermatology module on the mARble® platform – a mobile AR-based simulation application used to access interactive text, graphic, audio and video using off-the-shelf AR markers (http://marble-app.weebly.com/). Küçük, Kapakin and Göktaş (2016) used the Aurasma platform to embed AR markers to create a MagicBook® that students accessed on their mobile devices to view the interactive, augmented or superimposed object. One study (Garrett, Jackson, and Wilson 2015) included Layar (www.layar.com) to provide interactive markers for the available educational resources.

Six studies (Amer et al. 2017; Bartlett et al. 2017; Khelemsky, Hill, and Buchbinder 2017; Kowalewski et al. 2017; Morone et al. 2017; Sugand, Mawkin, and Gupte 2015) evaluated various modules on the TouchSurgery platform – an interactive AR application that is freely available for download on mobile phones and tablets. The app aims to familiarise the student with surgical procedural steps with a ‘tutorial’, then a ‘test’ module to assess understanding of the techniques. TouchSurgery calculates total scores as a percentage of correct decisions, appropriate swipe interactions and the time taken to complete each step.

The remaining studies involved external developers to provide an appropriate mMR platform (Aebersold et al. 2018; Birt et al. 2017a, 2017b; Cowling and Birt 2018; De Oliveira et al. 2013; John et al. 2015), some of which used 3D printed equipment in the augmented environment (Birt et al. 2017a, 2017b; Cowling and Birt 2018).

Outcome measures

The majority of studies measured competency of skill or knowledge using author-developed Likert scale questionnaires or procedural-based checklists (Aebersold et al. 2018; Albrecht et al. 2013; Amer et al. 2017; Bartlett et al. 2017; Birt et al. 2017a, 2017b; Cowling and Birt 2018; De Oliveira et al. 2013; John et al. 2015; Khelemsky, Hill, and Buchbinder 2017; Kowalewski et al. 2017; Küçük, Kapakin, and Göktaş 2016; Moro et al. 2017; Morone et al. 2017; Noll et al. 2014, 2017; Sugand, Mawkin, and Gupte 2015) or surveys developed to measure the learning experience or usability of the mMR (Aebersold et al. 2018; Albrecht et al. 2013; Birt et al. 2017a, 2017b; Cowling and Birt 2018; Garrett, Jackson, and Wilson 2015; John et al. 2015; Khelemsky, Hill, and Buchbinder 2017; Kowalewski et al. 2017; Moro et al. 2017; Morone et al. 2017; Noll et al. 2017). The participants’ emotional state was measured using the Profile of Mood States in two studies (Albrecht et al. 2013; Noll et al. 2017) and ‘cognitive load’ was considered in another (Küçük, Kapakin, and Göktaş 2016). Only one study (Garrett, Jackson, and Wilson 2015) included a focus group.

Benefits of mMR

Reported benefits from the use of mMR in healthcare education included improvement in procedural skills, such as scope and tube placement, and anatomy knowledge (Aebersold et al. 2018; Albrecht et al. 2013; Amer et al. 2017; Birt et al. 2017a; Cowling and Birt 2018; De Oliveira et al. 2013; John et al. 2015; Küçük, Kapakin, and Göktaş 2016). There was a general acceptance of mMR as a learning tool and recommendation for further use in courses (Amer et al. 2017; Birt et al. 2017a; Cowling and Birt 2018). Usability surveys highlighted that mMR demonstrated a statistically significant decrease in fatigue, numbness and boredom compared to control groups (Albrecht et al. 2013; Moro et al. 2017) and lowered the cognitive load of learning for students (Küçük, Kapakin, and Göktaş 2016). Students also favoured access to video content of skill demonstration when AR markers linked to external sources (Garrett, Jackson, and Wilson 2015).

Constraint of mMR

Some studies found no statistically significant difference in skill competency between experimental and control groups (Bartlett et al. 2017; Birt et al. 2017b; Noll et al. 2014; Noll et al. 2017), nor when comparing AV (Oculus), AR (3D objects on tablet) and tablet-based tuition without AR (Moro et al. 2017). Noll et al. (2017) reported some ‘irritability’ in the use of the mARble platform, and Garrett, Jackson, and Wilson (2015) commented that technical issues, slow response time, phone incompatibility, Internet connection issues and instability of the platform negatively influenced students’ perception of mMR. Students did not value AR marker links to text or PDFs and found the use of scanning markers disrupted the flow of learning.

Discussion

Quality

While this review included a range of study designs, the majority were either one-off case studies; did not specify mMR exposure time; or exposure time was not reflective of that in a tutorial or self-directed learning – seeming to negate the premise that access to mMR should enable flexibility in student-directed learning (Bujak et al. 2013). However, some studies did encourage flexibility in student learning by offering the exposure to mMR throughout the course (Birt et al. 2017a, 2017b; Cowling and Birt 2018; Küçük, Kapakin, and Göktaş 2016) or time similar to that in a 60 min tutorial (Bartlett et al. 2017). In an effort to embrace pedagogy before technology, Cowling and Birt (2018) employed the DBR methodology to analyse the problem, develop a solution in an initial phase, then test and refine in a second phase (McKenney and Reeves 2012). Solutions to practical and complex educational problems are iteratively developed and impact on real-world problems in a framework for experimental yet rigorous inquiry (Cook and Santos 2016; McKenney and Reeves 2012). Design-based research may better enable the simultaneous pursuit of pedagogy and practical innovation in the emerging field of mMR.

Two studies (Noll et al. 2014, 2017) included a follow-up period at 14 and 20 days, respectively. However, longitudinal studies across a semester would better determine the impact of mMR in retention of knowledge. Furthermore, as included studies were techno-centric with only one study involving a focus group (Garrett, Jackson, and Wilson 2015), future studies would benefit with interview-based data to understand the social process of learning.

Participants

The use of mMR in TEL was restricted to four health disciplines. However, the interventions included could be translated to use with physiotherapy, optometry, midwifery, oral health or occupational therapy students. Garrett, Jackson and Wilson (2015) integrated AR markers into a nursing clinical scenario requiring some decision-making regarding a patient in respiratory failure. A recent systematic review by Carvalho et al. (2017) highlighted that problem-based learning was the most used strategy in undergraduate nursing programmes to facilitate clinical reasoning – a crucial process in enabling autonomous healthcare practice. The use of mMR to develop similar scenarios could be utilised when designing for authentic learning and furthering high-order analysis.

Intervention

The TouchSurgery application was the most investigated form of mMR in this review with studies aimed at establishing validity with surgical modules. The Aurasma, Layar and mARble platforms were also utilised. These established platforms make development, delivery and marketing cost-effective compared to employing external developers to quantify mMR usability. Augmented reality procedural skills may also be less time-consuming to develop compared to the development of AV environments with multiple options in a complex case scenario. Some studies adopted a bring-your-own-device (BYOD) approach, especially those using the TouchSurgery application. While tethered AV devices are commercially available, they are not considered mobile and affordable for the general higher education population at the time of this review. One included study compared AR-based anatomy on a tablet to AV-based anatomy using an Oculus Rift (Moro et al. 2017). This study found no difference in mean anatomy knowledge score, learner immersion and engagement. However, the AV participants exhibited more symptoms (e.g. headaches, dizziness, nausea, blurred vision) than the less immersive AR group.

The included studies did not engage with higher order critical analysis or take advantage of interactions that link social media with mMR (Crompton 2013). This may be a result of current software limitations and accessibility in the included mobile devices. Recently, AV using tethered HDUs has been used to investigate behaviours of general practitioners and trainees when confronted with a patient demanding antibiotics prescription (Nifakos and Zary 2014; Pan et al. 2016). The scenario demonstrated that the virtual environment was immersive enough to present a real-life scenario and that experienced GPs are more likely to use clinical reasoning and rationalise withstanding the pressure to prescribe antibiotics unnecessarily. mMR problem-based TEL could be utilised in similar behavioural scenarios.

There was an absence of using mMR in interprofessional collaboration, which may prompt future integrated practice, understanding of discipline-specific roles, development of common language and potentially overcoming barriers of distance between course venues (Cochrane et al. 2017). Future studies may also include interprofessional students interacting with each within the mMR scenario, as well as less linear procedural learning in an effort to enhance clinical reasoning and engagement – which was only investigated by Moro et al. (2017) in the included studies.

Limitations of the review

The limitations of any systematic review are the scope of the search terms and the databases searched. The search was also restricted to papers published in the English language, resulting in studies from Australia, Canada, England, Germany, Turkey and the United States only. The age of the participants in the included studies was limited to a younger (<25 years) university population when indicated. Future inclusion of other disciplines (e.g. midwifery and occupational therapy) may reflect an age differential and perception of use of mMR in HHE.

Conclusions

The majority of studies in this review demonstrated a positive impact to the development of skill competency and knowledge by utilising mMR in HHE. While the included studies focused on four health disciplines, there is opportunity to extend mMR using available BYOD devices, developed software and skills-based measures to other areas of health education (see Table 2).

However, there is a demonstrable lack of longitudinal studies and a general lack of engagement with learning theory in the design of the studies, interventions and evaluation methods. Future studies in mMR would benefit from utilising DBR, which enables inquiry into the transformative possibilities in this emerging field of mMR in HHE. The identified key drivers for the studies are clinical skill competency and flexible delivery for students or in response to limited clinical simulation facilities. There is also little engagement with higher-order critical analysis or capacity building for non-specific scenarios. This stems from a gap in embedding the studies in learning theory and higher-order critical thinking, which presents a challenge to future studies in HHE.

References

Aebersold, M., et al., (2018) ‘Interactive anatomy-augmented virtual simulation training’, Clinical Simulation in Nursing, vol. 15, pp. 34–41. doi: 10.1016/j.ecns.2017.09.008.

Albrecht, U.-V., et al., (2013) ‘Effects of mobile augmented reality learning compared to textbook learning on medical students: randomized controlled pilot study’, Journal of Medical Internet Research, vol. 15, no. 8, pp. 195–207.

Amer, K. M., et al., (2017) ‘A mobile-based surgical simulation application: a comparative analysis of efficacy using a carpal tunnel release module’, Journal of Hand Surgery, vol. 42, no. 5, pp. 389.e1–389.e9. doi: 10.1016/j.jhsa.2017.02.008.

American Association of College of Nursing. (2017) Fact sheets. Nursing faculty shortage, [online] Available at: http://www.aacnnursing.org/News-Information/Fact-Sheets/Nursing-Faculty-Shortage

Bartlett, R. D., et al., (2017) ‘A pilot study to assess the utility of a freely downloadable mobile application simulator for undergraduate clinical skills training: a single-blinded, randomised controlled trial’, BMC Medical Education, vol. 17, no. 1, 247. doi: 10.1186/s12909-017-1085-y.

Birt, J. & Cowling, M. (2017) ‘Toward future ‘mixed reality’ learning spaces for steam education’, International Journal of Innovation in Science and Mathematics Education, vol. 25, no. 4, pp. 1–16.

Birt, J., Moore, E. & Cowling, M. (2017a) ‘Improving paramedic distance education through mobile mixed reality simulation’, Australasian Journal of Educational Technology, vol. 33, no. 6, pp. 69–83. doi: 10.14742/ajet.3596.

Birt, J., Moore, E. & Cowling, M. A. (2017b) ’Piloting mobile mixed reality simulation in paramedic distance education’, in eds N. Rodrigues, et al., 2017 IEEE 5th International Conference on Serious Games and Applications for Health (SeGAH). Perth, Australia, Institute of Electrical and Electronics Engineers Inc. doi: 10.1109/SeGAH.2017.7939270.

Bujak, K. R., et al., (2013) ‘A psychological perspective on augmented reality in the mathematics classroom’, Computers and Education, vol. 68, pp. 536–544. doi: 10.1016/j.compedu.2013.02.017.

Carvalho, D. P. S. R. P., et al., (2017) ‘Strategies used for the promotion of critical thinking in nursing undergraduate education: a systematic review’, Nurse Education Today, vol. 57, pp. 103–107. doi: 10.1016/j.nedt.2017.07.010.

Cochrane, T., et al., (2017) ‘Developing virtual collaborative health team educational environments’, in Me, Us, IT! Proceedings ASCILITE2017: 34th International Conference on Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education, eds H. Partridge, K. Davis & J. Thomas, Toowoomba, Australia, pp. 37–41.

Cook, J. & Santos, P. (2016) ‘Three phases of mobile learning state of the art and case of mobile help seeking tool for the health care sector’, in Mobile learning design, eds D. Churchill, et al., Springer International, Singapore, pp. 315–333. doi: 10.1007/978-981-10-0027-0_19.

Cowling, M. & Birt, J. (2018) Pedagogy before technology: A design-based research approach to enhancing skills development in paramedic science using mixed reality. Information (Switzerland), 9(2), p. 29. doi: 10.3390/info9020029.

Crompton, H. (2013) ‘A historical overview of mobile learning: toward learner-centered education’, in Handbook of mobile learning, eds Z. L. Berge & L. Y. Muilenburg, Routledge, New York, pp. 80–107.

De Oliveira, G. S., et al., (2013) ‘Virtual airway simulation to improve dexterity among novices performing fibreoptic intubation’, Anaesthesia, vol. 68, no. 10, pp. 1053–1058. doi: 10.1111/anae.12379.

Garrett, B. M., Jackson, C. & Wilson, B. (2015) ‘Augmented reality m-learning to enhance nursing skills acquisition in the clinical skills laboratory’, Interactive Technology and Smart Education, vol. 12, no. 4, pp. 298–314. doi: 10.1108/ITSE-05-2015-0013.

Hamstra, S. J., et al., (2014) ‘Reconsidering fidelity in simulation-based training’, Academic Medicine, vol. 89, no. 3, pp. 387–392. doi: 10.1097/ACM.0000000000000130.

Hong, Q. N., et al., (2018) ‘Mixed methods appraisal tool (MMAT), version 2018’, in ed I. C. Canadian Intellectual Property Office, Canada.

Houghton, C. E., et al., (2014) ‘Students’ experiences of implementing clinical skills in the real world of practice’, Journal of Clinical Nursing, vol. 22, no. 13/14, pp. 1961–1969. doi: 10.1111/jocn.12014.

International Telecommunication Union. (2015) The world in 2015: ICT facts and figures. [online]. Available at: https://www.itu.int/en/ITU-D/Statistics/Documents/facts/ICTFactsFigures2015.pdf. [Accessed 3 Aug. 2018).

John, N. W., et al., (2015) ‘A tablet-based virtual environment for neurosurgery training’, Presence: Teleoperators and Virtual Environments, vol. 24, no. 2, pp. 155–162. doi: 10.1162/PRES_a_00224.

Khelemsky, R., Hill, B. & Buchbinder, D. (2017) ‘Validation of a novel cognitive simulator for orbital floor reconstruction’, Journal of Oral and Maxillofacial Surgery, vol. 75, no. 4, pp. 775–785. doi: 10.1016/j.joms.2016.11.027.

Kowalewski, K. F., et al., (2017) ‘Validation of the mobile serious game application touch surgery™ for cognitive training and assessment of laparoscopic cholecystectomy’, Surgical Endoscopy and Other Interventional Techniques, vol. 31, no. 10, pp. 4058–4066. doi: 10.1016/j.joms.2016.11.027

Küçük, S., Kapakin, S. & Göktaş, Y. (2016) ‘Learning anatomy via mobile augmented reality: effects on achievement and cognitive load’, Anatomical sciences education, vol. 9, no. 5, pp. 411–421. doi: 10.1002/ase.1603.