ORIGINAL RESEARCH ARTICLE

Research into effective gamification features to inform e-learning design

Indrel Doney^*

Abstract

(Received: 5 June 2018; final version received: 23 October 2018; Published 31 January 2019)

Game-based learning is one of the main trends currently in e-learning, and while opinion is still divided on its merits a number of studies have been published that highlight its benefits in enhancing learning and increasing motivation. This paper aims to analyse existing research to identify some of the key approaches and pedagogical factors that make learning through games effective and engaging (considering adult learners in particular), with the purpose of creating a list of features that can be used to inform the inclusion of gamification elements into e-learning activities. Forty-one case studies of serious games, game-based learning and gamification in learning from the last 10 years were reviewed in order to identify the elements that contributed to their success. From this analysis a list of suggested features was produced that may be of use to those wishing to embed gamification elements when designing e-learning activities.

Keywords: game-based learning; gamification; serious games; e-learning; e-learning heuristics

^*Corresponding author. Email: idoney.elearning@gmail.com

Research in Learning Technology 2019. © 2019 I. Doney. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.

Citation: Research in Learning Technology 2019, 27: 2093 - http://dx.doi.org/10.25304/rlt.v27.2093

Introduction

Game-based learning is currently one of the main trends in e-learning (Beetham 2013) and looks to become increasing popular; it was identified by the NMC Horizon Project (2012) as one of six technologies that are likely to have a major impact on learning.

According to de Freitas and Maharg (2011), advances in technology mean that there are now greater opportunities for game-based learning across education and training. Several studies have been completed highlighting its advantages, including the benefits of serious games in problem-based learning (Kim, Park, and Baek 2009), the opportunities for collaborative learning (Nickel and Barnes 2010) and how games can increase motivation (Woo 2014). However, there are a number of factors that limit their implementation, including development costs (Moreno-Ger, Burgos, and Torrente 2009), a lack of design models (Hess and Gunter 2013) and uncertainty about their educational benefits (Tsekleves, Cosmas, and Aggoun 2016).

The creation of a set of heuristics (requirements) to inform educational game design was proposed by Malone (1980); however, it was based on elements considered to make games fun rather than to have value for learning. This paper attempts to outline a list of key approaches, design features and attributes that a review of available research shows as being of benefit to learning and motivation. This list could then be used to inform the design and development of e-learning activities for those wishing to include gamification elements without the technical challenges and costs involved in developing a video game or fully immersive 3D world.

Literature review

A sample of case studies was identified by searching various journals and repositories for papers on serious games, game-based learning and the use of gamification approaches in learning. These were restricted to those working with adults, dating from the last 10 years, that included evaluation or results of a specific learning activity or game.

The studies were located through references from meta-studies and literature reviews, as well as through searches on a university journal database using combinations of keywords – ‘serious games’ OR ‘game-based learning’ OR ‘simulation games’ OR ‘gamification’ AND ‘results’ OR ‘evaluation’ OR ‘outcomes’. The papers selected were primarily peer reviewed papers or conference proceedings. As the focus is on the features that make serious games or game-based learning effective, studies were selected where there was an effect on learning or motivation or both as well as a description of the game or learning activity and its features. In total 41 case studies were identified and the games or learning experiences outlined were analysed against key categories in order to provide the basis for the suggested approaches and design features.

Case studies analysis and categorisation

Several different classifications of specific game attributes in relation to learning have been suggested, including by Malone and Lepper (1987), who identified challenge, curiosity, control and fantasy. Garris, Ahlers and Driskell (2002) outline what they consider to be the key gaming features required for learning: fantasy, rules and/or goals, sensory stimuli, challenge, mystery and control; these were expanded further by Wilson et al. (2009) to include adaptation, assessment, interaction and conflict. According to Bedwell et al. (2012), there is considerable overlap in many of these; they produced a simplified list that includes nine categories: action language, assessment, conflict or challenge, control, environment, game fiction, human interaction, immersion and rules and/or goals. Carenys, Moya and Perramon (2017) included additional categories such as competition, telling stories, engagement, multimedia representation, feedback and transfer of skills.

Seven headings from different classification schemes (and an additional category, ‘Reflection’) have been used to attempt to categorise the data – the categories selected were those that appeared to be most prevalent as contributing factors to successful learning in the case studies reviewed. These were selected mainly from the lists by Bedwell et al. (2012) and Carenys, Moya and Perramon (2017); some generalisation has been used (e.g. environment, game fiction and multimedia representation have been amalgamated into a more general ‘Representation’ category) in order to streamline the analysis.

The categorisation and analysis are subjective as they are based on personal interpretation of the information available, and for most of the studies there appear to have been a number of contributing factors. However, when reviewing the learning elements of the case studies the factors that seemed to have the most impact included the following: challenge, the level of difficulty and ability to stretch the learner; competition against the game or other players, which can be a motivating factor and encourage learners to repeat tasks in order to improve; control, which relates to the ability of the learners to manipulate their environment; feedback, a crucial element of the learning cycle, enabling users to learn from actions and errors; interaction with game characters or other players (and in some cases tutors or moderators); representation, which can relate to the game’s environment, the realism of scenarios and the use of visuals and media; rules and/or goals, which allow learners to understand how to play the game and what they need to achieve; and finally, reflection. The latter category was not present in the classification schemes outlined; however, providing learners with the chance to reflect on learning (something often not present in games, where the emphasis may be more on speed or scores) appeared to be a positive factor in encouraging learning in five of the case studies.

Prioritising these categories is again a subjective exercise and would vary depending on the game style or format, topic and audience; however, in general, it would appear that having clear rules and achievable goals, the right level of challenge for the audience and providing meaningful feedback are some of the most crucial factors. Many of the categories connect with each other and can be usefully considered in relation to the game as a whole. For example, the level of challenge and achievable goals are closely linked; interaction with other players is often combined with competition mechanisms such as scoreboards. Providing feedback and allowing time to reflect on it are both key factors in a positive learning experience, and when planning the game environment a number of aspects need to be considered including the level of realism (representation), how much control the learner has and how he or she might interact with other players or characters.

A summary of the categories, suggested priority ratings and links between categories is included in Appendix A, and in the following sections each has been discussed in more detail.

Challenge

Challenge relates to the level of difficulty of activities (Garris, Ahlers, and Driskell 2002); these should stretch the user but be achievable in order to avoid frustration (Carenys, Moya, and Perramon 2017). The element of challenge was present in many of the case studies; several included increasing levels of difficulty, which allowed learners to build on knowledge from one level to the next (Anderson and Barnett 2011; Boeker et al. 2013; Domínguez et al. 2013; Gold 2016), and there may be advantages in providing options for learners to choose their own level (Eagle and Barnes 2009; Ebner and Holzinger 2007). Having time limits for activities (Ebner and Holzinger 2007) can add to the challenging aspect but can mean that people don’t have time to reflect on their choices (Kiili 2007). Challenges should be sufficiently engaging so that learners are motivated to solve them (Coller and Scott 2009), but it is important to ensure challenges are not too difficult for learners to overcome (Huang 2011).

The level of challenge needs to be considered during the process of setting the rules and/or goals to ensure that the goals are achievable but will still stretch the learner. This also relates to the competition category; for example, the increasing difficulty of different levels can be acknowledged by matching them to appropriate badges, rewards or scores.

Competition

Competition is common in games and can take different forms: players can compete against themselves, the game or with other users (Carenys, Moya, and Perramon 2017). Success is often recognised with rewards such as points or feedback messages (Wang and Sun 2011). The balance between intrinsic and extrinsic motivation is a concern when considering the use of reward systems; Malone and Lepper (1987) argue that although we do learn when extrinsically motivated (e.g. in order to achieve high grades), learning may be more effective when we are intrinsically motivated, for example by our own curiosity or desire to achieve a goal for our own satisfaction. However, they also emphasise the importance of recognition as a motivating factor – the enjoyment provided by having our efforts acknowledged by others. According to a study by Rieber and Noah (2008), too much emphasis on scores and competition can have a detrimental effect, with some users becoming obsessed by trying to improve their score and losing sight of the learning outcome.

Badges or points were used in several of the examples (Charles, Bustard, and Black 2009; Moreno 2012) and can be used to recognise different types of effort and engagement (Hakulinen, Auvinen, and Korhonen 2013). Virtual prizes were used in one case study (Hannig et al. 2012), and this was seen as a motivating factor for students to better their previous performance. Leaderboards where scores are compared against others may be motivating to some (Ebner and Holzinger 2007; Misfeldt 2015) and can encourage friendly competition and collaboration (Hannig et al. 2012); however, others may find it off-putting (Domínguez et al. 2013).

Competition needs to be considered in relation to challenge and goals, matching rewards, scores or other recognition devices to the level of difficulty and to achievement of set goals. It also provides a method of interaction with the game and other players.

Control

‘Control’ can refer to the ability of players to manipulate (Bedwell et al. 2012) or influence or control elements in a game (Wilson et al. 2009). In the study by Barab et al. (2009), players could change their environment (applying strategies to clean up a river), and in the study by Li, Cheng and Liu (2013) students were able to construct a railway track by applying computer programming concepts; these types of transformational activities may assist in allowing learners to contextualise information. Several of the case studies were scenario type activities where learners could make choices and then observe the results (Guillén-Nieto and Aleson-Carbonell 2012; Kikot et al. 2014). In the case study by Lancaster (2014), decisions were taken by a group vote; however, feedback from the users highlighted the importance of independent problem-solving.

Building in the opportunity for learners to have some sort of control within the game needs to be considered when designing the game environment or representation, deciding how learners will interact with the game and when setting the rules and/or goals. Another consideration is how learners will get feedback from the result of their actions and whether or not this will involve human interaction from peers or a facilitator.

Feedback

Feedback can relate to providing users with information on choices or providing an update of progress. It allows users to learn from actions and adjust their choices (Carenys, Moya, and Perramon 2017). The importance of clear feedback that provides reasons for incorrect choices as a development tool is highlighted in the studies by Antonaci et al. (2014), Erhel and Jamet (2013) and Sward et al. (2008). There are advantages to real-time (Berns, Gonzalez-Pardo, and Camacho 2013) and delayed feedback provided at the end of a challenge or activity (Knight et al. 2010), depending on the format and challenge level.

Another way to provide information is to provide ‘hints’ to allow learners to make more informed choices (Kanthan and Senger 2011). In the study by Moshirnia and Israel (2010), it was demonstrated that providing feedback and information by interaction with game characters (rather than via pop-up boxes or information displays) could be beneficial by allowing this to be seen as a more seamless part of the game. In complex games having feedback provided by a facilitator or teacher can allow for more detailed and individualised information (Yalabik, Howard, and Roden 2012). The study by Davidovitch, Parush and Shtub (2008) emphasised the importance to the learning process of having a mechanism for recording of history or progress.

Feedback is strongly linked to interaction; decisions about interaction methods will impact on how feedback will be presented to the learner. It may also link to challenge levels; how much feedback is provided and when may vary as difficulty increases.

Interaction

Interaction may relate to human interaction (Bedwell et al. 2012) or interaction with equipment, such as manipulating controls in a flight simulator game (Wilson et al. 2009). Many of the case studies were conducted in an educational environment, which allowed for team play (Hamalainen, Oksanen, and Hakkinen 2008; Kikot et al. 2014), competition against others (Charles, Bustard, and Black 2009; Hannig et al. 2012) or intervention by a supervisor or facilitator (Yalabik, Howard, and Roden 2012). These options may not always be possible, especially when using games outside of an instructional programme.

The ability to communicate and interact with other users was highlighted as a positive by Antonaci et al. (2014); it may allow for the development of additional interpersonal skills (Ranchhod et al. 2014) and encourage peer learning (Sindre, Natvig, and Jahre 2009). In more complex learning games (such as the business management simulation in the study by Yalabik, Howard, and Roden 2012), interaction with a facilitator or tutor can provide more detailed and individualised information and feedback than would be possible otherwise. Moderation by a facilitator may also be required when discussion boards and chat facilities are provided in order to ensure communication between learners is appropriate and to correct any misunderstandings.

As well as human interaction, there are also benefits to taking on a role and interacting with other game characters in order to provide a more contextualised learning environment (Boeker et al. 2013). It can also allow for a more engaging way of presenting information (Beckem and Watkins 2012; Hainey et al. 2011). In the study by Halpern et al. (2012), those taking on the role of a ‘tutor’ to an ‘avatar student’ produced the highest achievement rates.

The choice of interaction method (which may include interacting with the game, other players or a facilitator) is key when considering how feedback will be given. It also links in with competition features, especially if including ways of comparing scores with other players.

Representation

According to Wilson et al. (2009), representation relates to the user’s perceptions of the game’s reality. Bedwell et al. (2012) refer separately to a game’s environment (the location in which the game is set) and its immersion levels, which they class as the player’s perception of his or her place within the game – if successful the user will accept the temporary reality and engage with the game; if not it can lead to distractions and frustrations and have a negative effect on learning. Carenys, Moya and Perramon (2017) include multimedia representation as an attribute; it can also relate to narrative or telling stories (Carenys, Moya, and Perramon 2017) and fantasy versus reality (Garris, Ahlers, and Driskell 2002).

A few of the case studies employed a narrative (Hess and Gunter 2013; Woo 2014) and/or took place within a ‘fantasy’ environment (Boeker et al. 2013; Halpern et al. 2012). These may lead to greater learner engagement and a more immersive experience. However, in an experiment comparing different games (Adams et al. 2012) no evidence was found that students learn better when games have a strong narrative theme, and a study by Echeverría et al. (2012) testing different games to teach physics with schoolchildren found no significant differences in learning outcomes between fantasy and non-fantasy versions.

In other studies the importance of realism was highlighted; this was often related more to the perceived realism of the situations or scenarios than the visual representation of the environment (Guillén-Nieto and Aleson-Carbonell 2012; Kiili 2007; Kikot et al. 2014). According to Misfeldt (2015), scenarios can be simplified as long as the reaction to choices is seen as sufficiently realistic. Berns, Gonzalez-Pardo and Camacho (2013) highlighted the importance of a realistic environment to contextualise learning (i.e. teaching vocabulary about food in a supermarket). One of the benefits of a sufficiently realistic game environment is that learners are able to test out choices safely, which they could not in the real world; for example, in the study by Hamalainen, Oksanen and Hakkinen (2008) learners have to put out fires, and in the study by Lancaster (2014) nursing students have to choose the correct actions to deal with an overdose.

Visuals were used to help explain conceptual information in the studies by Anderson and Barnett (2011), Eagle and Barnes (2009) and Tüysüz (2009). There may be benefits to presenting information in different formats (Beckem and Watkins 2012), and the use of multimedia (animation, images, audio) was linked to engagement by Carenys, Moya and Perramon (2017). However, although advanced graphics and animations (e.g. virtual reality environments or high definition visuals) can create a more immersive experience, according to a study by Ranchhod et al. (2014), simple and abstracted interfaces still allow for effective learning. Simpler options could include the use of simple 2D representational graphics or cartoon-style visuals. The results of the study by Ritterfeld et al. (2009) showed that, although interactivity and multimodality both had a positive impact on knowledge gains when tested immediately, the effect was not demonstrated in tests of deeper learning or longer-term retention levels. It is important therefore to ensure that the focus when designing games is on activities that allow for deeper engagement with the subject such as those that provide opportunities for reflection, that allow learners to participate in discussions or problem-solving exercises, rather than those focusing on short-term memorisation.

Mayer (2009) conducted several experiments to identify the most effective ways of using multimedia for learning based on the principle that people can only process a certain amount of information through each channel (auditory and visual) at any one time (cognitive load). Although highlighting the benefits of appropriate graphics and other multimedia components, his coherence and redundancy principles specify that learning is more effective when extraneous elements are excluded. The study by Woo (2014) showed a correlation between cognitive load and performance. A learning game was designed with the aim of reducing cognitive overload – this included providing pretraining materials covering background knowledge, using a single perspective for visuals, zooming in and out on images to focus on elements as required and presenting the information in small segments.

The representation style and design of the game environment are also linked to how users interact with, and within, the game and the amount of control they have to affect their virtual surroundings.

Rules and goals

According to Garris, Ahlers and Driskell (2002), games happen within a ‘fixed space and time period with precise rules governing game play’ (p. 448). Games often have a main goal that is achieved through the completion of smaller goal-oriented tasks (Carenys, Moya, and Perramon 2017), and this was demonstrated in the examples by Barab et al. (2009) and Guillén-Nieto and Aleson-Carbonell (2012). How the game is presented (as learning or entertainment) can impact on learning (Erhel and Jamet 2013), and linking goals to learning outcomes can help learners focus. As shown by Rieber and Noah (2008), when this focus is lost at the expense of concentrating on game play elements (such as scores) it can have an adverse effect on learning.

According to Ebner and Holzinger (2007), games should be easy to play and require minimal instruction; however, this needs to be balanced with ensuring the instructions are clear enough to avoid misunderstandings (Yalabik, Howard and Roden 2012). The importance of clear rules and goals is further illustrated in the examples by Carenys, Moya and Perramon (2017) and Woo (2014).

The goals and challenge levels need to be considered together to ensure goals are achievable but at the correct level of difficulty to engage the learner.

Reflection

One further category that was not included in the classifications mentioned earlier but that is highlighted in five of the studies is reflection – providing opportunities for learners to reflect on the reasons for choices and think about the knowledge obtained from the game experience (Garris, Ahlers, and Driskell 2002). Yusoff et al. (2009) recommended that this be incorporated within the ‘game world’ and gave as an example providing feedback on mistakes and offering ‘corrective suggestions’.

In the study by Johnson and Mayer (2010) the most effective of several methods tested to encourage reflection was requiring learners to justify their choices by choosing from a list of options. This method proved to be more effective than requiring learners to type in their reasons, which disrupted the flow of the game. Providing opportunities for learners to discuss and share with others can also encourage reflection (Kiili 2007), as can providing time between activities and using questions as prompts (Yalabik, Howard, and Roden 2012).

Aspects that can have a negative effect on reflection include activities that are not at the correct level of challenge for the learner, as this can result in anxiety or boredom, both of which appeared to have an adverse effect (Liu, Cheng, and Huang 2011); using time limits for activities (Kiili 2007); and having time ‘lockouts’ after incorrect answers (Sindre, Natvig, and Jahre 2009).

Considering how to incorporate opportunities for reflection can include thinking about how and when feedback is provided and whether or not challenge levels are appropriate. It may also link with interaction methods, as encouraging discussion with peers can be a useful way for learners to reflect and share ideas.

Suggestions for implementing game-based learning and gamification features into e-learning activities

On reviewing the analysis of the case studies by category a number of approaches for the successful incorporation of game-based learning and gamification features can be suggested.

Challenge

The importance of having an appropriate challenge level for the audience was emphasised in a number of the studies. This can be achieved through having varying levels of difficulty; these could be increased throughout the game to allow learners to build on existing knowledge, or learners could be allowed to choose their own level. In general challenges should be engaging and interesting, should stretch the learner but be achievable.

Competition

Evidence in this area is less clear-cut, with advantages and disadvantages of different approaches being covered across the studies. However, in general it is useful to have a mechanism that will provide some sort of recognition of individual achievement and effort, and learners can be encouraged to practise and improve by being able to redo activities to better previous scores and/or performance. Opportunities to compete against others can be motivating for some but may not suit everyone; a compromise may to be provide a leaderboard (or similar) with the option to opt out of having scores published to allow a more flexible arrangement to meet individual needs. Competition can also be against the computer or game, and this could be included through activities that allow learners to apply knowledge by transforming the game environment or the use of scenarios that allow them to see the results of their choices or actions.

Feedback

This is a vital component of learning experiences and was highlighted in many of the case studies. It should be clear and easy to understand and, in order to improve its efficiency as a learning tool, provide reasons to explain choices rather than just highlighting right and wrong answers. It is worth considering when to provide feedback; it could be in real-time or at the end of the activity, depending on the format or level of difficulty. In addition to summative feedback, formative feedback and hints can allow learners to refine choices, delivering feedback through interaction with the game environment, or characters can provide a more seamless experience; for more complex learning situations, feedback could be provided by a facilitator or tutor, which would allow for more individualised responses. As well as feedback on specific activities, learners should be able to track their overall progress.

Interaction

The ability to interact with others can encourage discussions and sharing through peer learning and allow for expert intervention. For learning delivered to a class or team within a specific setting, these opportunities may be easy to identify; however in other situations this may be more difficult to achieve. Even though the arguments about competition still apply, although perhaps to a lesser degree, team playing and group competition were demonstrated in a number of the studies as having the potential to increase motivation and collaboration. The opportunity to engage with others can be provided through a medium such as a chat facility or discussion board; however thought needs to be given to how these will be moderated and managed. In addition to interaction with others, the use of role play and interaction with game characters can also be used as a way to provide information in a contextualised and engaging way. Explaining a topic to others requires learners to process their knowledge in a different way, and providing an opportunity within the game to ‘teach’ others (who could be real people or game characters) can help to reinforce longer-term learning.

Representation

Within e-learning activities there may be limits on how complex or realistic the game environment can be. However, the evidence from some of the case studies does show that this is not necessarily detrimental to learning; it is more important that scenarios, situations and the result of learner actions be believable. Providing information within an appropriate environment (even if simply rendered) can help to contextualise information. Where the opportunity exists, the use of visuals and graphics can be helpful in representing concepts, and providing information in multimedia formats can increase engagement; however this should be balanced against not overloading the learner’s cognitive processing.

Rules and/or goals

In order to be a meaningful and useful experience, clear goals and rules are required. Some of the studies show that focus on the game elements at the expense of the learning outcomes can have a negative impact, so it is useful to present the game as a learning experience and align goals and tasks within the game to the desired learning outcomes. To ensure learner engagement, instructions and rules should be clear and easy to follow, but care should be taken not to overload learners with unnecessary information.

Reflection

Providing learners with the opportunity to reflect on their learning and the new knowledge obtained (and not just react to the situations within the game) can encourage deeper and longer-term learning. This could be incorporated by asking learners to provide reasons for their choices during game interactions (this needs to be carefully designed to ensure that it does not disrupt the game flow); it is also useful to ensure learners have sufficient time for reflection between activities and to avoid time limits and lockout features. Discussion with others is a useful tool that encourages reflection and sharing, and this can be facilitated through a number of methods such as discussion boards.

The preceding recommendations have been rationalised (where overlap occurred, points were combined and clarified) and are outlined in the table in Appendix B. In order to simplify practical implementation, these are shown in three sections: points to consider during the overall structure design, factors that could be considered in the design of individual activities and optional points that may be useful in some activities or situations.

Conclusion

In conclusion the research appears to show that there are a number of gamification approaches that may be effective when designing e-learning activities for adult learners. The translation of these into an effective learning resource, however, relies on the ‘key role of design beyond medium’ (Clark, Tanner-Smith, and Killingsworth 2015, p.116), on careful consideration of the audience’s needs and on how to translate the learning outcomes to provide a meaningful learning experience rather than just on technical implementation.

Gamification is a method of presenting learning in an active way, as it is based on learning by doing (Garris, Ahlers, and Driskell 2002); however, it is often best utilised in conjunction with other methods (Sitzmann 2011), and a decision about the use of games should be made only after ‘detailed analysis of learning requirements and trade-offs among alternate instructional approaches’ (Hays 2005, p.53).

The list of features developed from this research (as outlined in Appendix B) may be useful as a starting point for others who are considering including gamification or game-based learning features when designing e-learning activities or other learning experiences.

Acknowledgements

This paper is based on a literature review and research work carried out as part of the final project of an MSc in ELearning Technology. The author thanks Trevor Barker and Andrew Pyper at Hertfordshire University for their support and encouragement.

References

Note: References marked with a star ´ are the case studies

Adams, D. M., et al., (2012) ‘Narrative games for learning: testing the discovery and narrative hypotheses’, Journal of Educational Psychology, vol. 104, no. 1, pp. 235–249. https://doi.org/10.1037/a0025595.

Anderson, J. & Barnett, M. (2011) ‘Using video games to support pre-service elementary teachers learning of basic physics principles’, Journal of Science Education and Technology, vol. 20, no. 4, pp. 347–362. https://doi.org/10.1007/s10956-010-9257-0. ´

Antonaci, A., et al., (2014) ‘A gamified collaborative course in entrepreneurship: focus on objectives and tools’, Computers in Human Behavior, vol. 51 (Part B), pp. 1276–1283. https://doi.org/10.1016/j.chb.2014.11.082. ´

Barab, S. A., et al., (2009) ‘Transformational play as a curricular scaffold: using videogames to support science education’, Journal of Science Education and Technology, vol. 18, no. 4, pp. 305–320. https://doi.org/10.1007/s10956-009-9171-5. ´

Beckem, J. M. & Watkins, M. (2012) ‘Bringing life to learning: immersive experiential learning simulations for online and blended courses’, Journal of Asynchronous Learning Network, vol. 16, no. 5, pp. 61–71. Available at: https://eric.ed.gov/?id=EJ1000091. ´

Bedwell, W., et al., (2012) ‘Toward a taxonomy linking game attributes to learning: an empirical study’, Simulation & Gaming, vol. 43, no. 6, pp. 729–760. https://doi.org/10.1177/1046878112439444.

Beetham, H. (2013) ‘Designing for learning in an uncertain future’, in Rethinking Pedagogy for a Digital Age: designing for 21st Century Learning, eds H. Beetham, & R. Sharpe, Routledge, New York, pp. 258–281.

Berns, A., Gonzalez-Pardo, A. & Camacho, D. (2013) ‘Game-like language learning in 3-D virtual environments’, Computers and Education, vol. 60, no. 1, pp. 210–220. https://doi.org/10.1016/j.compedu.2012.07.001. ´

Boeker, M., et al., (2013) ‘Game-based e-learning is more effective than a conventional instructional method: a randomized controlled trial with third-year medical students’, PLoS One, vol. 8, no. 12, pp. e82328. https://doi.org/10.1371/journal.pone.0082328. ´

Carenys, J., Moya, S. & Perramon, J. (2017) ‘Is it worth it to consider videogames in accounting education? A comparison of a simulation and a videogame in attributes, motivation and learning outcomes’, Revista de Contabilidad -- Spanish Accounting Review, vol. 20, no. 2, pp. 118–130. https://doi.org/10.1016/j.rcsar.2016.07.003. ´

Charles, M. T., Bustard, D., & Black, M. (2009) ‘Game inspired tool support for e-learning processes’, Electronic Journal of E-Learning [Online], vol. 7, no. 2, pp. 101–110. Available at: http://www.ejel.org/issue/download.html?idArticle=144 ´

Clark, D. B., Tanner-Smith, E. E. & Killingsworth, S. S. (2015) ‘Digital games, design, and learning: a systematic review and meta-analysis’, Review of Educational Research, vol. 86, no. 1, pp. 79–122. https://doi.org/10.3102/0034654315582065.

Coller, B. D. & Scott, M. J. (2009) ‘Effectiveness of using a video game to teach a course in mechanical engineering’, Computers & Education, vol. 53, no. 3, pp. 900–912. https://doi.org/10.1016/j.compedu.2009.05.012. ´

Davidovitch, L., Parush, A., & Shtub, A. (2008) ‘Simulation-based learning: the learning-forgetting-relearning process and impact of learning history’, Computers & Education, vol. 50, no. 3, pp. 866–880. https://doi.org/10.1016/j.compedu.2006.09.003. ´

de Freitas, S. & Maharg, P. (2011) ‘Digital games and learning: modelling learning experiences in the digital age’, in Digital Games and Learning, eds S. de Freitas & P. Maharg, Continuum, London, pp. 17–41.

Domínguez, A., et al., (2013) ‘Gamifying learning experiences: practical implications and outcomes’, Computers and Education, vol. 63 (April 2013), pp. 380–392. https://doi.org/10.1016/j.compedu.2012.12.020. ´

Eagle, M. & Barnes, T. (2009) ‘Experimental evaluation of an educational game for improved learning in introductory computing’, SIGCSE Bulletin Inroads, vol. 41, no. 1, pp. 321–325. https://doi.org/10.1145/1539024.1508980. ´

Ebner, M. & Holzinger, A. (2007) ‘Successful implementation of user-centered game-based learning in higher education: an example from civil engineering’, Computers & Education, vol. 49, no. 3, pp. 873–890. https://doi.org/10.1016/j.compedu.2005.11.026. ´

Echeverría, A., et al., (2012) ‘The atomic intrinsic integration approach: a structured methodology for the design of games for the conceptual understanding of physics’, Computers and Education, vol. 59, no. 2, pp. 806–816. https://doi.org/10.1016/j.compedu.2012.03.025.

Erhel, S. & Jamet, E. (2013) ‘Digital game-based learning: impact of instructions and feedback on motivation and learning effectiveness’, Computers and Education, vol. 67, pp. 156–167. https://doi.org/10.1016/j.compedu.2013.02.019. ´

Garris, R., Ahlers, R. & Driskell, J. E. (2002) ‘Games, motivation and learning: a research and practice model’, Simulation & Gaming, vol. 33, no. 4, pp. 441–467. https://doi.org/10.1177/1046878102238607.

Gold, S. (2016) ‘Design and effectiveness of a self-study pedagogical approach to using a simulation game in the classroom’, Developments in Business Simulation and Experiential Learning [Online], vol. 43, no. 1. Available at: https://journals.tdl.org/absel/index.php/absel/article/viewFile/3026/2974 ´

Guillén-Nieto, V. & Aleson-Carbonell, M. (2012) ‘Serious games and learning effectiveness: the case of It’s a Deal’, Computers and Education, vol. 58, no. 1, pp. 435–448. https://doi.org/10.1016/j.compedu.2011.07.015. ´

Hainey, T., et al., (2011) ‘Evaluation of a game to teach requirements collection and analysis in software engineering at tertiary education level’, Computers & Education, vol. 56, no. 1, pp. 21–35. https://doi.org/10.1016/j.compedu.2010.09.008. ´

Hakulinen, L., Auvinen, T. & Korhonen, A. (2013) ‘Empirical study on the effect of achievement badges in trakla2 online learning environment’, in Proceedings of Learning and Teaching in Computing and Engineering (LaTiCE) Conference 21–24 March 2013, IEEE, Macau. https://doi.org/10.1109/LaTiCE.2013.34. ´

Halpern, D. F., et al., (2012) ‘Operation ARA: a computerized learning game that teaches critical thinking and scientific reasoning’, Thinking Skills and Creativity, vol. 7, no. 2, pp. 93–100. https://doi.org/10.1016/j.tsc.2012.03.006. ´

Hamalainen, R., Oksanen, K., & Hakkinen, P. (2008) ‘Designing and analyzing collaboration in a scripted game for vocational education’, Computers in Human Behavior, vol. 24, no. 6, pp. 2496–2506. https://doi.org/10.1016/j.chb.2008.03.010. ´

Hannig, A., et al., (2012) ‘EMedOffice: a web-based collaborative serious game for teaching optimal design of a medical practice’, BMC Medical Education, vol. 12, no. 1, pp. 104. https://doi.org/10.1186/1472-6920-12-104. ´

Hays, R. T. (2005) The effectiveness of Instructional games: a literature review and discussion. (Technical Report 2005-004). Orlando, FL: Naval Air Warfare Center. Available at: www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA441935

Hess, T. & Gunter, G. (2013) ‘Serious game-based and nongame-based online courses: learning experiences and outcomes: serious game-based and nongame-based online courses’, British Journal of Educational Technology, vol. 44, no. 3, pp. 372–385. https://doi.org/10.1111/bjet.12024. ´

Huang, W. (2011) ‘Evaluating learners’ motivational and cognitive processing in an online game-based learning environment’, Computers in Human Behavior, vol. 27, no. 2, pp. 694–704. https://doi.org/10.1016/j.chb.2010.07.021. ´

Johnson, C. I. & Mayer, R. E. (2010) ‘Applying the self-explanation principle to multimedia learning in a computer-based game-like environment’, Computers in Human Behavior, vol. 26, no. 6, pp. 1246–1252. https://doi.org/10.1016/j.chb.2010.03.025. ´

Kanthan, R. & Senger, J. L. (2011) ‘The impact of specially designed digital games-based learning in undergraduate pathology and medical education’, Archives of Pathology & Laboratory Medicine [Online], vol. 135(January 2011), pp. 135–142. Available at: http://www.archivesofpathology.org/doi/pdf/10.1043/2009-0698-OAR1.1 ´

Kiili, K. (2007) ‘Foundation for problem-based gaming’, British Journal of Educational Technology, vol. 38, no. 3, pp. 394–404. https://doi.org/10.1111/j.1467-8535.2007.00704.x. ´

Kikot, T., et al., (2014) ‘Why use-centered game-based learning in higher education? The case of Cesim Simbrand’, Journal of Spatial and Organizational Dynamics, vol. II, no. 3, pp. 229–241. Available at: http://repositorio.ual.pt/handle/11144/465 ´

Kim, B., Park, H. & Baek, Y. (2009) ‘Not just fun, but serious strategies: using meta-cognitive strategies in game-based learning’, Computers & Education, vol. 52, no. 4, pp. 800–810. https://doi.org/10.1016/j.compedu.2008.12.004.

Knight, J. F., et al., (2010) ‘Serious gaming technology in major incident triage training: a pragmatic controlled trial’, Resuscitation, vol. 81, no. 9, pp. 1175–1179. https://doi.org/10.1016/j.resuscitation.2010.03.042. ´

Lancaster, R. J. (2014) ‘Serious game simulation as a teaching strategy in pharmacology’, Clinical Simulation in Nursing, vol. 10, no. 3, pp. e129–e137. https://doi.org/10.1016/j.ecns.2013.10.005. ´

Li, Z., Cheng, Y. & Liu, C. (2013) ‘A constructionism framework for designing game-like learning systems: its effect on different learners’, British Journal of Educational Technology, vol. 44, no. 2, pp. 208–224. https://doi.org/10.1111/j.1467-8535.2012.01305.x. ´

Liu, C., Cheng, Y. & Huang, C. (2011) ‘The effect of simulation games on the learning of computational problem solving’, Computers & Education, vol. 57, no. 3, pp. 1907–1918. https://doi.org/10.1016/j.compedu.2011.04.002. ´

Malone, T. W. (1980) ‘What makes things fun to learn? Heuristics for designing instructional computer games’, In Proceedings of the 3rd ACM SIGSMALL Symposium and the First SIGPC Symposium on Small Systems, 18–19 September, 1980, ACM, Palo Alto, CA, USA. https://doi.org/10.1145/800088.802839.

Malone, T. W., & Lepper, M. R. (1987) ‘Making learning fun: a taxonomy of intrinsic motivations for learning’, in Aptitude, Learning and Instruction, Vol. 3: Conative and Affective Process Analyses, eds R. E. Snow & M. J. Farr, Laurence Erlbaum Associates, Hillsdale, NJ, pp. 223–253.

Mayer, R. E. (2009) Multimedia learning, 2nd edn, Cambridge University Press, New York.

Misfeldt M. (2015) ‘Scenario based education as a framework for understanding students engagement and learning in a project management simulation game’, The Electronic Journal of e-Learning [Online], vol. 13, no. 3, pp. 181–191. Available at: http://www.ejel.org/issue/download.html?idArticle=416 ´

Moreno, J. (2012) ‘Digital competition game to improve programming skills’, Educational Technology and Society, vol. 15, no. 3, pp. 288–297. Available at: https://www.researchgate.net/publication/268294104_Digital_Competition_Game_to_Improve_Programming_Skills ´

Moreno-Ger, P., Burgos, D. & Torrente, J. (2009) ‘Digital games in e-learning environments; current uses and emerging trends’, Simulation & Gaming, vol. 40, no. 5, pp. 669–687. https://doi.org/10.1177/1046878109340294.

Moshirnia, A., & Israel, M. (2010) ‘The educational efficacy of distinct information delivery systems in modified video games’, Journal of Interactive Learning Research, vol. 21, no. 3, pp. 383–340. Available at: https://www.researchgate.net/publication/277819894_The_Educational_Efficacy_of_Distinct_Information_Delivery_Systems_in_Modified_Video_Games ´

Nickel, A. & Barnes, T. (2010) ‘Games for CS education: computer-supported collaborative learning and multiplayer games’, in Proceedings of the Fifth International Conference on the Foundations of Digital Games, 19–21 June 2010, Monterey, CA: ACM. https://doi.org/10.1145/1822348.1822391.

NMC Horizon Project. (2012) NMC Horizon Report: 2012 Higher education edition [Online] Available at: http://www.nmc.org/pdf/2012-horizon-report-HE.pdf

Ranchhod, A., et al., (2014) ‘Evaluating the educational effectiveness of simulation games: A value generation model’, Information Sciences, vol. 264, no. 2014, pp. 75–90. https://doi.org/10.1016/j.ins.2013.09.008. ´

Rieber, L. P. & Noah, D. (2008) ‘Games, simulations, and visual metaphors in education: antagonism between enjoyment and learning’, Educational Media International, vol. 45, no. 2, pp. 77–92. https://doi.org/10.1080/09523980802107096.

Ritterfeld, U., et al., (2009) ‘Multimodality and interactivity: connecting properties of serious games with educational outcomes’, CyberPsychology & Behavior, vol. 12, no. 6, pp. 691–697. https://doi.org/10.1089/cpb.2009.0099. ´

Sindre, G., Natvig, L. & Jahre, M. (2009) ‘Experimental validation of the learning effect for a pedagogical game on computer fundamentals’, IEEE Transactions on Education, vol. 52, no. 1, pp. 10–18. https://doi.org/10.1109/TE.2007.914944. ´

Sitzmann, T. (2011) ‘A meta-analytic examination of the instructional effectiveness of computer-based simulation games’, Personnel Psychology, vol. 64, no. 2, pp. 489–528. https://doi.org/10.1111/j.1744-6570.2011.01190.x.

Sward, K. A., et al., (2008) ‘Use of a web-based game to teach pediatric content to medical students’, Ambulatory Pediatrics, vol. 8, no. 6, pp. 354–359. https://doi.org/10.1016/j.ambp.2008.07.007. ´

Tsekleves, E., Cosmas, J. & Aggoun, A. (2016) ‘Benefits, barriers and guideline recommendations for the implementation of serious games in education for stakeholders and policymakers’, British Journal of Educational Technology, vol. 47, no. 1, pp. 164–183. https://doi.org/10.1111/bjet.12223.

Tüysüz, C. (2009) ‘Effect of the computer based game on pre-service teachers’ achievement, attitudes, metacognition and motivation in chemistry’, Scientific Research and Essays, vol. 4, no. 8, pp. 780–790. Available at: https://www.researchgate.net/publication/268205182_Effect_of_the_computer_based_game_on_pre-service_teachers’_achievement_attitudes_metacognition_and_motivation_in_chemistry ´

Wang, H., & Sun, C. T. (2011) ‘Game reward systems: gaming experiences and social meanings’, in Proceedings of the DiGRA 2011 Conference: Think design play, eds C. Marinka, K. Helen & W. Annika, DiGRA. Available at: http://www.digra.org/wp-content/uploads/digital-library/11310.20247.pdf

Wilson, K. A., et al., (2009) ‘Relationships between game attributes and learning outcomes: review and research proposals’, Simulation & Gaming, vol. 40, no. 2, pp. 217–266. https://doi.org/10.1177/1046878108321866.

Woo, J. (2014) ‘Digital game-based learning supports student motivation, cognitive success, and performance outcomes’, Journal of Educational Technology & Society, vol. 17, no. 3, pp. 291–307. Available at: https://www.researchgate.net/publication/286063834_Digital_Game-Based_Learning_Supports_Student_Motivation_Cognitive_Success_and_Performance_Outcomes ´

Yalabik, B., Howard, M. & Roden, S. (2012) ‘The innovation game: lessons in strategy and managing operations’, International Journal of Operations & Production Management, vol. 32(12), pp. 1441–1459. https://doi.org/10.1108/01443571211284188. ´

Yusoff, A., et al., (2009) ‘A conceptual framework for serious games’, in Ninth IEEE International Conference on Advanced Learning Technologies, 15–17 July 2009, ICALT, Riga. https://doi.org/10.1109/ICALT.2009.19.

Appendix A: Categories used to analyse case studies

Appendix B: Suggested features derived from analysis of case studies