The E-Design Assessment Tool: an evidence-informed approach towards a consistent terminology for quantifying online distance learning activities

  • Helen Walmsley-Smith Academic Development Unit, Staffordshire University
  • Lynn Machin School of Life Sciences and Education, Staffordshire University
  • Geoff Walton Department of Languages, Information & Communications, MMU
Keywords: learning, activity, technology enhanced learning, terminology, online learning, learning design

Abstract

Online distance learning (ODL) continues to expand rapidly, despite persistent concerns that student experience is poorer and retention lower than for face-to-face courses. Various factors affect ODL quality, but the impact of recommended learning activities, such as student interaction activities and those involving feedback, have proven difficult to assess because of challenges in definition and measurement. Although learning design frameworks and learning analytics have been used to evaluate learning designs, their use is hampered by this lack of an agreed terminology. This study addresses these challenges by initially identifying key ODL activities that are associated with higher quality learning designs. The learning activity terminology was tested using independent raters, who categorised the learning activities in four ODL courses as ‘interaction’, ‘feedback’ or ‘other’, with inter-rater reliability near or above recommended levels. Whilst challenges remain for consistent categorisation, the analysis suggests that increased clarity in the learning activity will aid categorisation. As a result of this analysis, the E-Design Assessment Tool (eDAT) has been developed to incorporate this key terminology and enable improved quantification of learning designs. This can be used with learning analytics, particularly retention and attainment data, thus providing an effective feedback loop on the learning design.

Downloads

Download data is not yet available.

References

Agostinho, S. (2011) ‘The use of a visual learning design representation to support the design process of teaching in higher education’, Australasian Journal of Educational Technology, vol. 27, no. 6, pp. 961–978. http://doi.org/10.14742/ajet.923

Agostinho, S., et al., (2002) ‘A tool to evaluate the potential for an ICT-based learning design to foster “high-quality learning”’, Winds of Change in the Sea of Learning. Proceedings of the 19th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, Auckland, pp. 29–38, [online] Available at: http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1128&context=edupapers

Akin, L. & Neal, D. (2007) ‘CREST+ model: writing effective online discussion questions’, Journal of Online Learning and Teaching, vol. 3, no. 2, pp. 191–202, [online] Available at: http://jolt.merlot.org/vol3no2/akin.htm

Allen, I. E. & Seaman, J. (2017) Distance Learning Compass: Distance Education Enrollment Report, [online] Available at: https://onlinelearningsurvey.com/reports/digtiallearningcompassenrollment2017.pdf

Anderson, L. W. & Krathwohl, D. R. (2001) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives, Longman: New York.

Bakharia, A., et al., (2016) ‘A conceptual framework linking learning design with learning analytics’, in Proceedings of the 6th International Conference on Learning Analytics & Knowledge, Edinburgh, pp. 329–338.

Bawa, P. (2016) ‘Retention in online courses: exploring issues and solutions-A literature review’, SAGE Open, vol. 6, no. 1, pp. 1–11. http://doi.org/10.1177/2158244015621777

Bonk, C. J. & Khoo, E. (2014) Adding Some TEC-Variety: 100+ Activities for Motivating and Retaining Learners Online, Open World Books, Bloomington, IN.

Charlton, P., Magoulas, G. & Laurillard, D. (2012) ‘Enabling creative learning design through semantic technologies’, Technology, Pedagogy and Education, vol. 21, no. 2, pp. 231–253. http://doi.org/10.1080/1475939X.2012.698165

Choi, H., et al., (2013) ‘The extent of and reasons for non re-enrollment: a case of Korea National Open University’, International Review of Research in Open & Distance Learning, vol. 14, no. 4, pp. 19–35. http://dx.doi.org/10.19173/irrodl.v14i4.1314

Conole, G. (2013) Designing for Learning in an Open World, Springer Science & Business Media, Milton Keynes.

Conole, G. (2014) ‘The 7Cs of learning design – A new approach to rethinking design practice’, in Proceedings of the 9th International Conference on Networked Learning, pp. 502–509, [online] Available at: http://www.lancaster.ac.uk/fss/organisations/netlc/past/nlc2014/abstracts/pdf/conole.pdf

Cross, S., et al., (2012) Challenge and Change in Curriculum Design Process, Communities, Visualisation and Practice, [online] Available at: http://www.open.ac.uk/blogs/OULDI/wp-content/uploads/2010/11/OULDI_Final_Report_Final.pdf

Croxton, R. A. (2014) ‘The role of interactivity in student satisfaction and persistence in online learning’, Journal of Online Learning and Teaching, vol. 10, no. 2, pp. 314–325, [online] Available from: http://jolt.merlot.org/vol10no2/croxton_0614.pdf

Currier, S., et al., (2006) ‘Vocabularies for describing pedagogical approach in e-learning: a scoping study’, in Proceedings of the International Conference on Dublin Core and Metadata Applications: Metadata for Knowledge and Learning, Colima, Oct 2006.

Dalziel, J. R., et al., (2013) The Larnaca Declaration on Learning Design, [online] Available at: https://larnacadeclaration.wordpress.com/

Dalziel, J. R., et al., (2016) ‘Learning design: where do we go from here?’, in Learning Design: Conceptualizing a Framework for Teaching and Learning Online, ed J. R. Dalziel, Routledge, Abingdon.

Dubuclet, K. S., Lou, Y. & MacGregor, K. (2015) ‘Design and cognitive level of student dialogue in secondary school online courses’, American Journal of Distance Education, vol. 29, no. 4, pp. 283–296. http://doi.org/10.1080/08923647.2015.1085722

Ekwunife-Orakwue, K. C. V. & Teng, T. L. (2014) ‘The impact of transactional distance dialogic interactions on student learning outcomes in online and blended environments’, Computers and Education, vol. 78, pp. 414–427. http://doi.org/10.1016/j.compedu.2014.06.011

Evans, M. J. & Moore, J. S. (2013) ‘Peer tutoring with the aid of the Internet’, British Journal of Educational Technology, vol. 44, no. 1, pp. 144–155. http://doi.org/10.1111/j.1467-8535.2011.01280.x

Fasse, R., Humbert, J. & Rappold, R. (2009) ‘Rochester Institute of Technology: analyzing student success’, Journal of Asynchronous Learning Networks, vol. 13, no. 3, pp. 37–48. Available from: https://files.eric.ed.gov/fulltext/EJ862354.pdf

Fill, K. & Conole, G. (2005) ‘A learning design toolkit to create pedagogically effective learning activities’, Journal of Interactive Media in Education, vol. 1, no. 9, pp. 1–16. http://doi.org/10.5334/2005-8

Fisher, M. & Baird, D. E. (2005) ‘Online learning design that fosters student support, self-regulation, and retention’, Campus-Wide Information Systems, vol. 22, no. 2, pp. 88–107. http://doi.org/10.1108/10650740510587100

Garrett, R. (2015) Up, Down, Flat: Distance Learning Data Collection and Enrollment Patterns in Australia, UK, and USA, [online] Available at: http://www.obhe.ac.uk/documents/download?id=1023

Garrison, D. R. (2011) Community of Inquiry Model, [online] Available at: https://coi.athabascau.ca/coi-model/an-interactive-coi-model/

Geri, N. (2012) ‘The resonance factor: probing the impact of video on student retention in distance learning’, Interdisciplinary Journal of E-Learning and Learning Objects, vol. 8, [online] Available at: https://www.learntechlib.org/p/44757/

Godwin, S. J., Thorpe, M. & Richardson, J. T. E. (2008) ‘The impact of computer-mediated interaction on distance learning’, British Journal of Educational Technology, vol. 39, no. 1, pp. 52–70. http://doi.org/10.1111/j.1467-8535.2007.00727.x

Gorsky, P. & Blau, I. (2009) ‘Online teaching effectiveness: a tale of two instructors’, International Review of Research in Open & Distance Learning, vol. 10, no. 3, pp. 1–28. http://dx.doi.org/10.19173/irrodl.v10i3.712

Hattie, J. (2003) ‘Teachers make a difference: what is the research evidence?’, in Proceedings of the Australian Council for Educational Research Annual Conference: Building Teacher Quality, Melbourne, Oct 2003.

Hatzipanagos, S. & Warburton, S. (2009) ‘Feedback as dialogue: exploring the links between formative assessment and social software in distance learning’, Learning, Media and Technology, vol. 34, no. 1, pp. 45–59. http://doi.org/10.1080/17439880902759919

Hawkins, A., et al., (2013) ‘Academic performance, course completion rates, and student perception of the quality and frequency of interaction in a virtual high school’, Distance Education, vol. 34, no. 1, pp. 64–83. http://doi.org/10.1080/01587919.2013.770430

Haywood, J. (2016) ‘Learning from MOOCs: lessons for the future’, in From Books to MOOCs? Emerging Models of Learning and Teaching in Higher Education, eds E. De Corte, L. Engwall & U. Teichler, Portland Press, London, pp. 69–80.

Hillman, D., Willis, D. & Gunawardena, C. (1994) ‘Learner-interface interaction in distance education: an extension of contemporary models and strategies for practitioners’, American Journal of Distance Education, vol. 8, no. 2, pp. 30–42. http://doi.org/10.1080/08923649409526853

Jaggars, S. S. & Xu, D. (2016) ‘How do online course design features influence student performance?’, Computers & Education, vol. 95, pp. 270–284. http://doi.org/10.1016/j.compedu.2016.01.014

Krippendorff, K. (2004) ‘Reliability in content analysis: some common misconceptions and recommendations’, Human Communication Research, vol. 30, no. 3, pp. 411–433. http://doi.org/10.1111/j.1468-2958.2004.tb00738.x

Krippendorff, K. (2011) Computing Krippendorff’s Alpha-Reliability, Annenberg School for Communication Departmental Papers, Philadelphia.

Krippendorff, K. (2013) Content Analysis: An Introduction to its Methodology, 3rd edn., Sage, London.

Laurillard, D. (2002) Rethinking University Teaching: A Conversational Framework for the Effective Use of Educational Technology, 2nd edn., Routledge, London.

Laurillard, D. (2012) Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology [Vitalsource e-book], Routledge, Abingdon.

Lenert, K. A. & Janes, D. P. (2017) ‘The incorporation of quality attributes into online course design in higher education’, International Journal of E-Learning & Distance Education, vol. 33, no. 1, pp. 1–14, [online] Available at: http://www.ijede.ca/index.php/jde/article/view/987/1658

Lin, J.-W., et al., (2014) ‘Development and evaluation of across-unit diagnostic feedback mechanism for online learning’, Journal of Educational Technology & Society, vol. 17, no. 3, pp. 138–153, [online] Available at: https://www.jstor.org/stable/jeductechsoci.17.3.138

Liu, S., Gomez, J. & Yen, C.-J. (2009) ‘Community college online course retention and final grade: predictability of social presence’, Journal of Interactive Online Learning, vol. 8, no. 2, pp. 165–182. Available from https://pdfs.semanticscholar.org/3bea/7b0a25381625b933f0d91f6e3a5286ff9ac2.pdf

London Knowledge Lab. (2016) Learning Designer, UCL Institute of Education, [online] Available at: https://www.ucl.ac.uk/learning-designer/index.php

Moore, M. G. (1989) ‘Three types of interaction’, American Journal of Distance Education, vol. 3, no. 2, pp. 1–7. http://doi.org/10.1080/08923648909526659

Moore, M. G. & Kearsley, G. (2011) Distance Education: A Systems View of Online Learning, 3rd edn., Wadsworth, Belmont.

Mor, Y., Ferguson, R. & Wasson, B. (2015) ‘Editorial: learning design, teacher inquiry into student learning and learning analytics: a call for action’, British Journal of Educational Technology, vol. 46, no. 2, pp. 221–229. http://doi.org/10.1111/bjet.12273

Neuendorf, K. A. (2002) The Content Analysis Guidebook, Sage, London.

Nicol, D. J. & Macfarlane-Dick, D. (2006) ‘Formative assessment and self-regulated learning: a model and seven principles of good feedback practice’, Studies in Higher Education, vol. 31, no. 2, pp. 199–218. http://doi.org/10.1080/03075070600572090

Persico, D., et al., (2013) ‘Learning design Rashomon I – Supporting the design of one lesson through different approaches’, Research in Learning Technology, vol. 21. http://doi.org/10.3402/rlt.v21i0.20224

Reeves, T. (1996) Evaluating What Really Matters in Computer-Based Education, [online] Available at: http://eduworks.com/Documents/Workshops/EdMedia1998/docs/reeves.html

Rienties, B. & Toetenel, L. (2016) ‘The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules’, Computers in Human Behavior, vol. 60, pp. 333–341. http://doi.org/10.1016/j.chb.2016.02.074

Rienties, B., Toetenel, L. & Bryan, A. (2015) ‘“Scaling up” learning design: impact of learning design activities on LMS behavior and performance’, in Proceedings of the 5th International Conference on Learning Analytics And Knowledge, Poughkeepsie, March 2015.

Salmon, G. (2004) E-Moderating: The Key to Online Teaching and Learning, Routledge, London.

Sancho-Vinuesa, T., Escudero-Viladoms, N. & Masià, R. (2013) ‘Continuous activity with immediate feedback: a good strategy to guarantee student engagement with the course’, Open Learning: The Journal of Open, Distance and e-Learning, vol. 28, no. 1, pp. 51–66. http://doi.org/10.1080/02680513.2013.776479

Shelton, B. E., Hung, J.-L. & Lowenthal, P. R. (2017) ‘Predicting student success by modeling student interaction in asynchronous online courses’, Distance Education, vol. 38, no. 1, pp. 59–69. http://doi.org/10.1080/01587919.2017.1299562

Simpson, O. (2010) ‘22% – can we do better?’ – The CWP Retention Literature Review, Centre for Widening Participation, The Open University, Milton Keynes.

Simpson, O. (2013) ‘Student retention in distance education: are we failing our students?’, Open Learning: The Journal of Open, Distance and e-Learning, vol. 28, no. 2, pp. 105–119. http://doi.org/10.1080/02680513.2013.847363

So, H.-J. (2009) ‘When groups decide to use asynchronous online discussions: collaborative learning and social presence under a voluntary participation structure’, Journal of Computer Assisted Learning, vol. 25, no. 2, pp. 143–160. http://doi.org/10.1111/j.1365-2729.2008.00293.x

Stephenson, J. & Coomey, M. (2001) ‘Online learning: It is all about dialogue, involvement, support and control – According to the research’, in Teaching and Learning Online: New Pedagogies for New Technologies (Creating Success), ed J. Stephenson, Kogan Page, London. pp. 37–52.

Stott, P. (2016) ‘The perils of a lack of student engagement: reflections of a “lonely, brave, and rather exposed” online instructor’, British Journal of Educational Technology, vol. 47, no. 1, pp. 51–64. http://doi.org/10.1111/bjet.12215

Sutton, R. (2014) ‘Unlearning the past: new foundations for online student retention’, Journal of Educators Online, vol. 11, no. 3, pp. 1–30. Available from https://www.thejeo.com/archive/archive/2014_113/suttonpdf

Swan, K., et al., (2015) ‘Metaphors for learning and the pedagogies of MOOCs’, in Meeting of the American Educational Research Association, Chicago, IL, April 2015.

Tenório, T., et al., (2016) ‘Does peer assessment in on-line learning environments work? A systematic review of the literature’, Computers in Human Behavior, vol. 64, pp. 94–107. http://doi.org/10.1016/j.chb.2016.06.020

Thorpe, M. (2008) ‘Effective online interaction: mapping course design to bridge from research to practice’, Australasian Journal of Educational Technology, vol. 24, no. 1, pp. 57–72. http://doi.org/10.14742/ajet.1230

Trigwell, K., Prosser, M. & Ginns, P. (2005) ‘Phenomenographic pedagogy and a revised “Approaches to teaching inventory”’, Higher Education Research & Development, vol. 24, no. 4, pp. 349–360. http://doi.org/10.1080/07294360500284730

University of Ulster. (2008) Hybrid Learning Model, [online] Available at: http://addl.ulster.ac.uk/odl/hybridlearningmodel

Vygotsky, L. S. & Cole, M. (1978) Mind in Society: the Development of Higher Psychological Processes, Harvard University Press. Cambridge, MA.

Walmsley, H. (2017) Best Practice Models for e-Design, [online] Stoke-on-Trent Available at: http://blogs.staffs.ac.uk/bestpracticemodels/

Wanstreet, C. E. (2006) ‘Interaction in online learning environments: a review of the literature’, The Quarterly Review of Distance Education, vol. 7, no. 4, pp. 399–411.

Xiao, J. (2017) ‘Learner-content interaction in distance education: the weakest link in interaction research’, Distance Education, vol. 38, no. 1, pp. 123–135. http://doi.org/10.1080/01587919.2017.1298982

Published
2019-02-08
How to Cite
Walmsley-Smith, H., Machin, L., & Walton, G. (2019). The E-Design Assessment Tool: an evidence-informed approach towards a consistent terminology for quantifying online distance learning activities. Research in Learning Technology, 27. https://doi.org/10.25304/rlt.v27.2106
Section
Original Research Articles