Pragmatic meta-analytic studies: learning the lessons from naturalistic evaluations of multiple cases

Abstract

This paper explores the concept of pragmatic meta-analytic studies in eLearning. Much educational technology literature focuses on developers and teachers describing and reflecting on their experiences. Few connections are made between these experiential ‘stories’. The data set is fragmented and offers few generalisable lessons. The field needs guidelines about what can be learnt from such single-case reports. The pragmatic meta-analytic studies described in this paper have two common aspects: (1) the cases are related in some way, and (2) the data are authentic, that is, the evaluations have followed a naturalistic approach. We suggest that examining a number of such cases is best done by a mixed-methods approach with an emphasis on qualitative strategies. In the paper, we overview 63 eLearning cases. Three main meta-analytic strategies were used: (1) meta-analysis of the perception of usefulness across all cases, (2) metaanalysis of recorded benefits and challenges across all cases, and (3) meta-analysis of smaller groups of cases where the learning design and/or use of technology are similar. This study indicated that in Hong Kong the basic and non-interactive eLearning strategies are often valued by students, while their perceptions of interactive strategies that are potentially more beneficial fluctuate. One possible explanation relates to the level of risk that teachers and students are willing to take in venturing into more innovative teaching and learning strategies.

Keywords: evaluation; meta-analysis; eLearning cases; risk-taking

DOI: 10.1080/09687760802315879