Pragmatic meta-analytic studies: learning the lessons from naturalistic evaluations of multiple cases
This paper explores the concept of pragmatic meta-analytic studies in eLearning. Much educational technology literature focuses on developers and teachers describing and reflecting on their experiences. Few connections are made between these experiential ‘stories’. The data set is fragmented and offers few generalisable lessons. The field needs guidelines about what can be learnt from such single-case reports. The pragmatic meta-analytic studies described in this paper have two common aspects: (1) the cases are related in some way, and (2) the data are authentic, that is, the evaluations have followed a naturalistic approach. We suggest that examining a number of such cases is best done by a mixed-methods approach with an emphasis on qualitative strategies. In the paper, we overview 63 eLearning cases. Three main meta-analytic strategies were used: (1) meta-analysis of the perception of usefulness across all cases, (2) metaanalysis of recorded benefits and challenges across all cases, and (3) meta-analysis of smaller groups of cases where the learning design and/or use of technology are similar. This study indicated that in Hong Kong the basic and non-interactive eLearning strategies are often valued by students, while their perceptions of interactive strategies that are potentially more beneficial fluctuate. One possible explanation relates to the level of risk that teachers and students are willing to take in venturing into more innovative teaching and learning strategies.
Keywords: evaluation; meta-analysis; eLearning cases; risk-taking
Authors contributing to Research in Learning Technology retain the copyright of their article and at the same time agree to publish their articles under the terms of the Creative Commons CC-BY 4.0 License (http://creativecommons.org/licenses/by/4.0/) allowing third parties to copy and redistribute the material in any medium or format, and to remix, transform, and build upon the material, for any purpose, even commercially, under the condition that appropriate credit is given, that a link to the license is provided, and that you indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.