The E-Design Assessment Tool: an evidence-informed approach towards a consistent terminology for quantifying online distance learning activities
Abstract
Online distance learning (ODL) continues to expand rapidly, despite persistent concerns that student experience is poorer and retention lower than for face-to-face courses. Various factors affect ODL quality, but the impact of recommended learning activities, such as student interaction activities and those involving feedback, have proven difficult to assess because of challenges in definition and measurement. Although learning design frameworks and learning analytics have been used to evaluate learning designs, their use is hampered by this lack of an agreed terminology. This study addresses these challenges by initially identifying key ODL activities that are associated with higher quality learning designs. The learning activity terminology was tested using independent raters, who categorised the learning activities in four ODL courses as ‘interaction’, ‘feedback’ or ‘other’, with inter-rater reliability near or above recommended levels. Whilst challenges remain for consistent categorisation, the analysis suggests that increased clarity in the learning activity will aid categorisation. As a result of this analysis, the E-Design Assessment Tool (eDAT) has been developed to incorporate this key terminology and enable improved quantification of learning designs. This can be used with learning analytics, particularly retention and attainment data, thus providing an effective feedback loop on the learning design.
Downloads
References
Agostinho, S. (2011) ‘The use of a visual learning design representation to support the design process of teaching in higher education’, Australasian Journal of Educational Technology, vol. 27, no. 6, pp. 961–978. http://doi.org/10.14742/ajet.923
Agostinho, S., et al., (2002) ‘A tool to evaluate the potential for an ICT-based learning design to foster “high-quality learning”’, Winds of Change in the Sea of Learning. Proceedings of the 19th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, Auckland, pp. 29–38, [online] Available at: http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1128&context=edupapers
Akin, L. & Neal, D. (2007) ‘CREST+ model: writing effective online discussion questions’, Journal of Online Learning and Teaching, vol. 3, no. 2, pp. 191–202, [online] Available at: http://jolt.merlot.org/vol3no2/akin.htm
Allen, I. E. & Seaman, J. (2017) Distance Learning Compass: Distance Education Enrollment Report, [online] Available at: https://onlinelearningsurvey.com/reports/digtiallearningcompassenrollment2017.pdf
Anderson, L. W. & Krathwohl, D. R. (2001) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives, Longman: New York.
Bakharia, A., et al., (2016) ‘A conceptual framework linking learning design with learning analytics’, in Proceedings of the 6th International Conference on Learning Analytics & Knowledge, Edinburgh, pp. 329–338.
Bawa, P. (2016) ‘Retention in online courses: exploring issues and solutions-A literature review’, SAGE Open, vol. 6, no. 1, pp. 1–11. http://doi.org/10.1177/2158244015621777
Bonk, C. J. & Khoo, E. (2014) Adding Some TEC-Variety: 100+ Activities for Motivating and Retaining Learners Online, Open World Books, Bloomington, IN.
Charlton, P., Magoulas, G. & Laurillard, D. (2012) ‘Enabling creative learning design through semantic technologies’, Technology, Pedagogy and Education, vol. 21, no. 2, pp. 231–253. http://doi.org/10.1080/1475939X.2012.698165
Choi, H., et al., (2013) ‘The extent of and reasons for non re-enrollment: a case of Korea National Open University’, International Review of Research in Open & Distance Learning, vol. 14, no. 4, pp. 19–35. http://dx.doi.org/10.19173/irrodl.v14i4.1314
Conole, G. (2013) Designing for Learning in an Open World, Springer Science & Business Media, Milton Keynes.
Conole, G. (2014) ‘The 7Cs of learning design – A new approach to rethinking design practice’, in Proceedings of the 9th International Conference on Networked Learning, pp. 502–509, [online] Available at: http://www.lancaster.ac.uk/fss/organisations/netlc/past/nlc2014/abstracts/pdf/conole.pdf
Cross, S., et al., (2012) Challenge and Change in Curriculum Design Process, Communities, Visualisation and Practice, [online] Available at: http://www.open.ac.uk/blogs/OULDI/wp-content/uploads/2010/11/OULDI_Final_Report_Final.pdf
Croxton, R. A. (2014) ‘The role of interactivity in student satisfaction and persistence in online learning’, Journal of Online Learning and Teaching, vol. 10, no. 2, pp. 314–325, [online] Available from: http://jolt.merlot.org/vol10no2/croxton_0614.pdf
Currier, S., et al., (2006) ‘Vocabularies for describing pedagogical approach in e-learning: a scoping study’, in Proceedings of the International Conference on Dublin Core and Metadata Applications: Metadata for Knowledge and Learning, Colima, Oct 2006.
Dalziel, J. R., et al., (2013) The Larnaca Declaration on Learning Design, [online] Available at: https://larnacadeclaration.wordpress.com/
Dalziel, J. R., et al., (2016) ‘Learning design: where do we go from here?’, in Learning Design: Conceptualizing a Framework for Teaching and Learning Online, ed J. R. Dalziel, Routledge, Abingdon.
Dubuclet, K. S., Lou, Y. & MacGregor, K. (2015) ‘Design and cognitive level of student dialogue in secondary school online courses’, American Journal of Distance Education, vol. 29, no. 4, pp. 283–296. http://doi.org/10.1080/08923647.2015.1085722
Ekwunife-Orakwue, K. C. V. & Teng, T. L. (2014) ‘The impact of transactional distance dialogic interactions on student learning outcomes in online and blended environments’, Computers and Education, vol. 78, pp. 414–427. http://doi.org/10.1016/j.compedu.2014.06.011
Evans, M. J. & Moore, J. S. (2013) ‘Peer tutoring with the aid of the Internet’, British Journal of Educational Technology, vol. 44, no. 1, pp. 144–155. http://doi.org/10.1111/j.1467-8535.2011.01280.x
Fasse, R., Humbert, J. & Rappold, R. (2009) ‘Rochester Institute of Technology: analyzing student success’, Journal of Asynchronous Learning Networks, vol. 13, no. 3, pp. 37–48. Available from: https://files.eric.ed.gov/fulltext/EJ862354.pdf
Fill, K. & Conole, G. (2005) ‘A learning design toolkit to create pedagogically effective learning activities’, Journal of Interactive Media in Education, vol. 1, no. 9, pp. 1–16. http://doi.org/10.5334/2005-8
Fisher, M. & Baird, D. E. (2005) ‘Online learning design that fosters student support, self-regulation, and retention’, Campus-Wide Information Systems, vol. 22, no. 2, pp. 88–107. http://doi.org/10.1108/10650740510587100
Garrett, R. (2015) Up, Down, Flat: Distance Learning Data Collection and Enrollment Patterns in Australia, UK, and USA, [online] Available at: http://www.obhe.ac.uk/documents/download?id=1023
Garrison, D. R. (2011) Community of Inquiry Model, [online] Available at: https://coi.athabascau.ca/coi-model/an-interactive-coi-model/
Geri, N. (2012) ‘The resonance factor: probing the impact of video on student retention in distance learning’, Interdisciplinary Journal of E-Learning and Learning Objects, vol. 8, [online] Available at: https://www.learntechlib.org/p/44757/
Godwin, S. J., Thorpe, M. & Richardson, J. T. E. (2008) ‘The impact of computer-mediated interaction on distance learning’, British Journal of Educational Technology, vol. 39, no. 1, pp. 52–70. http://doi.org/10.1111/j.1467-8535.2007.00727.x
Gorsky, P. & Blau, I. (2009) ‘Online teaching effectiveness: a tale of two instructors’, International Review of Research in Open & Distance Learning, vol. 10, no. 3, pp. 1–28. http://dx.doi.org/10.19173/irrodl.v10i3.712
Hattie, J. (2003) ‘Teachers make a difference: what is the research evidence?’, in Proceedings of the Australian Council for Educational Research Annual Conference: Building Teacher Quality, Melbourne, Oct 2003.
Hatzipanagos, S. & Warburton, S. (2009) ‘Feedback as dialogue: exploring the links between formative assessment and social software in distance learning’, Learning, Media and Technology, vol. 34, no. 1, pp. 45–59. http://doi.org/10.1080/17439880902759919
Hawkins, A., et al., (2013) ‘Academic performance, course completion rates, and student perception of the quality and frequency of interaction in a virtual high school’, Distance Education, vol. 34, no. 1, pp. 64–83. http://doi.org/10.1080/01587919.2013.770430
Haywood, J. (2016) ‘Learning from MOOCs: lessons for the future’, in From Books to MOOCs? Emerging Models of Learning and Teaching in Higher Education, eds E. De Corte, L. Engwall & U. Teichler, Portland Press, London, pp. 69–80.
Hillman, D., Willis, D. & Gunawardena, C. (1994) ‘Learner-interface interaction in distance education: an extension of contemporary models and strategies for practitioners’, American Journal of Distance Education, vol. 8, no. 2, pp. 30–42. http://doi.org/10.1080/08923649409526853
Jaggars, S. S. & Xu, D. (2016) ‘How do online course design features influence student performance?’, Computers & Education, vol. 95, pp. 270–284. http://doi.org/10.1016/j.compedu.2016.01.014
Krippendorff, K. (2004) ‘Reliability in content analysis: some common misconceptions and recommendations’, Human Communication Research, vol. 30, no. 3, pp. 411–433. http://doi.org/10.1111/j.1468-2958.2004.tb00738.x
Krippendorff, K. (2011) Computing Krippendorff’s Alpha-Reliability, Annenberg School for Communication Departmental Papers, Philadelphia.
Krippendorff, K. (2013) Content Analysis: An Introduction to its Methodology, 3rd edn., Sage, London.
Laurillard, D. (2002) Rethinking University Teaching: A Conversational Framework for the Effective Use of Educational Technology, 2nd edn., Routledge, London.
Laurillard, D. (2012) Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology [Vitalsource e-book], Routledge, Abingdon.
Lenert, K. A. & Janes, D. P. (2017) ‘The incorporation of quality attributes into online course design in higher education’, International Journal of E-Learning & Distance Education, vol. 33, no. 1, pp. 1–14, [online] Available at: http://www.ijede.ca/index.php/jde/article/view/987/1658
Lin, J.-W., et al., (2014) ‘Development and evaluation of across-unit diagnostic feedback mechanism for online learning’, Journal of Educational Technology & Society, vol. 17, no. 3, pp. 138–153, [online] Available at: https://www.jstor.org/stable/jeductechsoci.17.3.138
Liu, S., Gomez, J. & Yen, C.-J. (2009) ‘Community college online course retention and final grade: predictability of social presence’, Journal of Interactive Online Learning, vol. 8, no. 2, pp. 165–182. Available from https://pdfs.semanticscholar.org/3bea/7b0a25381625b933f0d91f6e3a5286ff9ac2.pdf
London Knowledge Lab. (2016) Learning Designer, UCL Institute of Education, [online] Available at: https://www.ucl.ac.uk/learning-designer/index.php
Moore, M. G. (1989) ‘Three types of interaction’, American Journal of Distance Education, vol. 3, no. 2, pp. 1–7. http://doi.org/10.1080/08923648909526659
Moore, M. G. & Kearsley, G. (2011) Distance Education: A Systems View of Online Learning, 3rd edn., Wadsworth, Belmont.
Mor, Y., Ferguson, R. & Wasson, B. (2015) ‘Editorial: learning design, teacher inquiry into student learning and learning analytics: a call for action’, British Journal of Educational Technology, vol. 46, no. 2, pp. 221–229. http://doi.org/10.1111/bjet.12273
Neuendorf, K. A. (2002) The Content Analysis Guidebook, Sage, London.
Nicol, D. J. & Macfarlane-Dick, D. (2006) ‘Formative assessment and self-regulated learning: a model and seven principles of good feedback practice’, Studies in Higher Education, vol. 31, no. 2, pp. 199–218. http://doi.org/10.1080/03075070600572090
Persico, D., et al., (2013) ‘Learning design Rashomon I – Supporting the design of one lesson through different approaches’, Research in Learning Technology, vol. 21. http://doi.org/10.3402/rlt.v21i0.20224
Reeves, T. (1996) Evaluating What Really Matters in Computer-Based Education, [online] Available at: http://eduworks.com/Documents/Workshops/EdMedia1998/docs/reeves.html
Rienties, B. & Toetenel, L. (2016) ‘The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules’, Computers in Human Behavior, vol. 60, pp. 333–341. http://doi.org/10.1016/j.chb.2016.02.074
Rienties, B., Toetenel, L. & Bryan, A. (2015) ‘“Scaling up” learning design: impact of learning design activities on LMS behavior and performance’, in Proceedings of the 5th International Conference on Learning Analytics And Knowledge, Poughkeepsie, March 2015.
Salmon, G. (2004) E-Moderating: The Key to Online Teaching and Learning, Routledge, London.
Sancho-Vinuesa, T., Escudero-Viladoms, N. & Masià, R. (2013) ‘Continuous activity with immediate feedback: a good strategy to guarantee student engagement with the course’, Open Learning: The Journal of Open, Distance and e-Learning, vol. 28, no. 1, pp. 51–66. http://doi.org/10.1080/02680513.2013.776479
Shelton, B. E., Hung, J.-L. & Lowenthal, P. R. (2017) ‘Predicting student success by modeling student interaction in asynchronous online courses’, Distance Education, vol. 38, no. 1, pp. 59–69. http://doi.org/10.1080/01587919.2017.1299562
Simpson, O. (2010) ‘22% – can we do better?’ – The CWP Retention Literature Review, Centre for Widening Participation, The Open University, Milton Keynes.
Simpson, O. (2013) ‘Student retention in distance education: are we failing our students?’, Open Learning: The Journal of Open, Distance and e-Learning, vol. 28, no. 2, pp. 105–119. http://doi.org/10.1080/02680513.2013.847363
So, H.-J. (2009) ‘When groups decide to use asynchronous online discussions: collaborative learning and social presence under a voluntary participation structure’, Journal of Computer Assisted Learning, vol. 25, no. 2, pp. 143–160. http://doi.org/10.1111/j.1365-2729.2008.00293.x
Stephenson, J. & Coomey, M. (2001) ‘Online learning: It is all about dialogue, involvement, support and control – According to the research’, in Teaching and Learning Online: New Pedagogies for New Technologies (Creating Success), ed J. Stephenson, Kogan Page, London. pp. 37–52.
Stott, P. (2016) ‘The perils of a lack of student engagement: reflections of a “lonely, brave, and rather exposed” online instructor’, British Journal of Educational Technology, vol. 47, no. 1, pp. 51–64. http://doi.org/10.1111/bjet.12215
Sutton, R. (2014) ‘Unlearning the past: new foundations for online student retention’, Journal of Educators Online, vol. 11, no. 3, pp. 1–30. Available from https://www.thejeo.com/archive/archive/2014_113/suttonpdf
Swan, K., et al., (2015) ‘Metaphors for learning and the pedagogies of MOOCs’, in Meeting of the American Educational Research Association, Chicago, IL, April 2015.
Tenório, T., et al., (2016) ‘Does peer assessment in on-line learning environments work? A systematic review of the literature’, Computers in Human Behavior, vol. 64, pp. 94–107. http://doi.org/10.1016/j.chb.2016.06.020
Thorpe, M. (2008) ‘Effective online interaction: mapping course design to bridge from research to practice’, Australasian Journal of Educational Technology, vol. 24, no. 1, pp. 57–72. http://doi.org/10.14742/ajet.1230
Trigwell, K., Prosser, M. & Ginns, P. (2005) ‘Phenomenographic pedagogy and a revised “Approaches to teaching inventory”’, Higher Education Research & Development, vol. 24, no. 4, pp. 349–360. http://doi.org/10.1080/07294360500284730
University of Ulster. (2008) Hybrid Learning Model, [online] Available at: http://addl.ulster.ac.uk/odl/hybridlearningmodel
Vygotsky, L. S. & Cole, M. (1978) Mind in Society: the Development of Higher Psychological Processes, Harvard University Press. Cambridge, MA.
Walmsley, H. (2017) Best Practice Models for e-Design, [online] Stoke-on-Trent Available at: http://blogs.staffs.ac.uk/bestpracticemodels/
Wanstreet, C. E. (2006) ‘Interaction in online learning environments: a review of the literature’, The Quarterly Review of Distance Education, vol. 7, no. 4, pp. 399–411.
Xiao, J. (2017) ‘Learner-content interaction in distance education: the weakest link in interaction research’, Distance Education, vol. 38, no. 1, pp. 123–135. http://doi.org/10.1080/01587919.2017.1298982
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors contributing to Research in Learning Technology retain the copyright of their article and at the same time agree to publish their articles under the terms of the Creative Commons CC-BY 4.0 License (http://creativecommons.org/licenses/by/4.0/) allowing third parties to copy and redistribute the material in any medium or format, and to remix, transform, and build upon the material, for any purpose, even commercially, under the condition that appropriate credit is given, that a link to the license is provided, and that you indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
.png){kind=spacer}
