The E-Design Assessment Tool: an evidence-informed approach towards a consistent terminology for quantifying online distance learning activities

  • Helen Walmsley-Smith Academic Development Unit, Staffordshire University
  • Lynn Machin School of Life Sciences and Education, Staffordshire University
  • Geoff Walton Department of Languages, Information & Communications, MMU
Keywords: learning, activity, technology enhanced learning, terminology, online learning, learning design


Online distance learning (ODL) continues to expand rapidly, despite persistent concerns that student experience is poorer and retention lower than for face-to-face courses. Various factors affect ODL quality, but the impact of recommended learning activities, such as student interaction activities and those involving feedback, have proven difficult to assess because of challenges in definition and measurement. Although learning design frameworks and learning analytics have been used to evaluate learning designs, their use is hampered by this lack of an agreed terminology. This study addresses these challenges by initially identifying key ODL activities that are associated with higher quality learning designs. The learning activity terminology was tested using independent raters, who categorised the learning activities in four ODL courses as ‘interaction’, ‘feedback’ or ‘other’, with inter-rater reliability near or above recommended levels. Whilst challenges remain for consistent categorisation, the analysis suggests that increased clarity in the learning activity will aid categorisation. As a result of this analysis, the E-Design Assessment Tool (eDAT) has been developed to incorporate this key terminology and enable improved quantification of learning designs. This can be used with learning analytics, particularly retention and attainment data, thus providing an effective feedback loop on the learning design.


Download data is not yet available.


Agostinho, S. (2011) ‘The use of a visual learning design representation to support the design process of teaching in higher education’, Australasian Journal of Educational Technology, vol. 27, no. 6, pp. 961–978.

Agostinho, S., et al., (2002) ‘A tool to evaluate the potential for an ICT-based learning design to foster “high-quality learning”’, Winds of Change in the Sea of Learning. Proceedings of the 19th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, Auckland, pp. 29–38, [online] Available at:

Akin, L. & Neal, D. (2007) ‘CREST+ model: writing effective online discussion questions’, Journal of Online Learning and Teaching, vol. 3, no. 2, pp. 191–202, [online] Available at:

Allen, I. E. & Seaman, J. (2017) Distance Learning Compass: Distance Education Enrollment Report, [online] Available at:

Anderson, L. W. & Krathwohl, D. R. (2001) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives, Longman: New York.

Bakharia, A., et al., (2016) ‘A conceptual framework linking learning design with learning analytics’, in Proceedings of the 6th International Conference on Learning Analytics & Knowledge, Edinburgh, pp. 329–338.

Bawa, P. (2016) ‘Retention in online courses: exploring issues and solutions-A literature review’, SAGE Open, vol. 6, no. 1, pp. 1–11.

Bonk, C. J. & Khoo, E. (2014) Adding Some TEC-Variety: 100+ Activities for Motivating and Retaining Learners Online, Open World Books, Bloomington, IN.

Charlton, P., Magoulas, G. & Laurillard, D. (2012) ‘Enabling creative learning design through semantic technologies’, Technology, Pedagogy and Education, vol. 21, no. 2, pp. 231–253.

Choi, H., et al., (2013) ‘The extent of and reasons for non re-enrollment: a case of Korea National Open University’, International Review of Research in Open & Distance Learning, vol. 14, no. 4, pp. 19–35.

Conole, G. (2013) Designing for Learning in an Open World, Springer Science & Business Media, Milton Keynes.

Conole, G. (2014) ‘The 7Cs of learning design – A new approach to rethinking design practice’, in Proceedings of the 9th International Conference on Networked Learning, pp. 502–509, [online] Available at:

Cross, S., et al., (2012) Challenge and Change in Curriculum Design Process, Communities, Visualisation and Practice, [online] Available at:

Croxton, R. A. (2014) ‘The role of interactivity in student satisfaction and persistence in online learning’, Journal of Online Learning and Teaching, vol. 10, no. 2, pp. 314–325, [online] Available from:

Currier, S., et al., (2006) ‘Vocabularies for describing pedagogical approach in e-learning: a scoping study’, in Proceedings of the International Conference on Dublin Core and Metadata Applications: Metadata for Knowledge and Learning, Colima, Oct 2006.

Dalziel, J. R., et al., (2013) The Larnaca Declaration on Learning Design, [online] Available at:

Dalziel, J. R., et al., (2016) ‘Learning design: where do we go from here?’, in Learning Design: Conceptualizing a Framework for Teaching and Learning Online, ed J. R. Dalziel, Routledge, Abingdon.

Dubuclet, K. S., Lou, Y. & MacGregor, K. (2015) ‘Design and cognitive level of student dialogue in secondary school online courses’, American Journal of Distance Education, vol. 29, no. 4, pp. 283–296.

Ekwunife-Orakwue, K. C. V. & Teng, T. L. (2014) ‘The impact of transactional distance dialogic interactions on student learning outcomes in online and blended environments’, Computers and Education, vol. 78, pp. 414–427.

Evans, M. J. & Moore, J. S. (2013) ‘Peer tutoring with the aid of the Internet’, British Journal of Educational Technology, vol. 44, no. 1, pp. 144–155.

Fasse, R., Humbert, J. & Rappold, R. (2009) ‘Rochester Institute of Technology: analyzing student success’, Journal of Asynchronous Learning Networks, vol. 13, no. 3, pp. 37–48. Available from:

Fill, K. & Conole, G. (2005) ‘A learning design toolkit to create pedagogically effective learning activities’, Journal of Interactive Media in Education, vol. 1, no. 9, pp. 1–16.

Fisher, M. & Baird, D. E. (2005) ‘Online learning design that fosters student support, self-regulation, and retention’, Campus-Wide Information Systems, vol. 22, no. 2, pp. 88–107.

Garrett, R. (2015) Up, Down, Flat: Distance Learning Data Collection and Enrollment Patterns in Australia, UK, and USA, [online] Available at:

Garrison, D. R. (2011) Community of Inquiry Model, [online] Available at:

Geri, N. (2012) ‘The resonance factor: probing the impact of video on student retention in distance learning’, Interdisciplinary Journal of E-Learning and Learning Objects, vol. 8, [online] Available at:

Godwin, S. J., Thorpe, M. & Richardson, J. T. E. (2008) ‘The impact of computer-mediated interaction on distance learning’, British Journal of Educational Technology, vol. 39, no. 1, pp. 52–70.

Gorsky, P. & Blau, I. (2009) ‘Online teaching effectiveness: a tale of two instructors’, International Review of Research in Open & Distance Learning, vol. 10, no. 3, pp. 1–28.

Hattie, J. (2003) ‘Teachers make a difference: what is the research evidence?’, in Proceedings of the Australian Council for Educational Research Annual Conference: Building Teacher Quality, Melbourne, Oct 2003.

Hatzipanagos, S. & Warburton, S. (2009) ‘Feedback as dialogue: exploring the links between formative assessment and social software in distance learning’, Learning, Media and Technology, vol. 34, no. 1, pp. 45–59.

Hawkins, A., et al., (2013) ‘Academic performance, course completion rates, and student perception of the quality and frequency of interaction in a virtual high school’, Distance Education, vol. 34, no. 1, pp. 64–83.

Haywood, J. (2016) ‘Learning from MOOCs: lessons for the future’, in From Books to MOOCs? Emerging Models of Learning and Teaching in Higher Education, eds E. De Corte, L. Engwall & U. Teichler, Portland Press, London, pp. 69–80.

Hillman, D., Willis, D. & Gunawardena, C. (1994) ‘Learner-interface interaction in distance education: an extension of contemporary models and strategies for practitioners’, American Journal of Distance Education, vol. 8, no. 2, pp. 30–42.

Jaggars, S. S. & Xu, D. (2016) ‘How do online course design features influence student performance?’, Computers & Education, vol. 95, pp. 270–284.

Krippendorff, K. (2004) ‘Reliability in content analysis: some common misconceptions and recommendations’, Human Communication Research, vol. 30, no. 3, pp. 411–433.

Krippendorff, K. (2011) Computing Krippendorff’s Alpha-Reliability, Annenberg School for Communication Departmental Papers, Philadelphia.

Krippendorff, K. (2013) Content Analysis: An Introduction to its Methodology, 3rd edn., Sage, London.

Laurillard, D. (2002) Rethinking University Teaching: A Conversational Framework for the Effective Use of Educational Technology, 2nd edn., Routledge, London.

Laurillard, D. (2012) Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology [Vitalsource e-book], Routledge, Abingdon.

Lenert, K. A. & Janes, D. P. (2017) ‘The incorporation of quality attributes into online course design in higher education’, International Journal of E-Learning & Distance Education, vol. 33, no. 1, pp. 1–14, [online] Available at:

Lin, J.-W., et al., (2014) ‘Development and evaluation of across-unit diagnostic feedback mechanism for online learning’, Journal of Educational Technology & Society, vol. 17, no. 3, pp. 138–153, [online] Available at:

Liu, S., Gomez, J. & Yen, C.-J. (2009) ‘Community college online course retention and final grade: predictability of social presence’, Journal of Interactive Online Learning, vol. 8, no. 2, pp. 165–182. Available from

London Knowledge Lab. (2016) Learning Designer, UCL Institute of Education, [online] Available at:

Moore, M. G. (1989) ‘Three types of interaction’, American Journal of Distance Education, vol. 3, no. 2, pp. 1–7.

Moore, M. G. & Kearsley, G. (2011) Distance Education: A Systems View of Online Learning, 3rd edn., Wadsworth, Belmont.

Mor, Y., Ferguson, R. & Wasson, B. (2015) ‘Editorial: learning design, teacher inquiry into student learning and learning analytics: a call for action’, British Journal of Educational Technology, vol. 46, no. 2, pp. 221–229.

Neuendorf, K. A. (2002) The Content Analysis Guidebook, Sage, London.

Nicol, D. J. & Macfarlane-Dick, D. (2006) ‘Formative assessment and self-regulated learning: a model and seven principles of good feedback practice’, Studies in Higher Education, vol. 31, no. 2, pp. 199–218.

Persico, D., et al., (2013) ‘Learning design Rashomon I – Supporting the design of one lesson through different approaches’, Research in Learning Technology, vol. 21.

Reeves, T. (1996) Evaluating What Really Matters in Computer-Based Education, [online] Available at:

Rienties, B. & Toetenel, L. (2016) ‘The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules’, Computers in Human Behavior, vol. 60, pp. 333–341.

Rienties, B., Toetenel, L. & Bryan, A. (2015) ‘“Scaling up” learning design: impact of learning design activities on LMS behavior and performance’, in Proceedings of the 5th International Conference on Learning Analytics And Knowledge, Poughkeepsie, March 2015.

Salmon, G. (2004) E-Moderating: The Key to Online Teaching and Learning, Routledge, London.

Sancho-Vinuesa, T., Escudero-Viladoms, N. & Masià, R. (2013) ‘Continuous activity with immediate feedback: a good strategy to guarantee student engagement with the course’, Open Learning: The Journal of Open, Distance and e-Learning, vol. 28, no. 1, pp. 51–66.

Shelton, B. E., Hung, J.-L. & Lowenthal, P. R. (2017) ‘Predicting student success by modeling student interaction in asynchronous online courses’, Distance Education, vol. 38, no. 1, pp. 59–69.

Simpson, O. (2010) ‘22% – can we do better?’ – The CWP Retention Literature Review, Centre for Widening Participation, The Open University, Milton Keynes.

Simpson, O. (2013) ‘Student retention in distance education: are we failing our students?’, Open Learning: The Journal of Open, Distance and e-Learning, vol. 28, no. 2, pp. 105–119.

So, H.-J. (2009) ‘When groups decide to use asynchronous online discussions: collaborative learning and social presence under a voluntary participation structure’, Journal of Computer Assisted Learning, vol. 25, no. 2, pp. 143–160.

Stephenson, J. & Coomey, M. (2001) ‘Online learning: It is all about dialogue, involvement, support and control – According to the research’, in Teaching and Learning Online: New Pedagogies for New Technologies (Creating Success), ed J. Stephenson, Kogan Page, London. pp. 37–52.

Stott, P. (2016) ‘The perils of a lack of student engagement: reflections of a “lonely, brave, and rather exposed” online instructor’, British Journal of Educational Technology, vol. 47, no. 1, pp. 51–64.

Sutton, R. (2014) ‘Unlearning the past: new foundations for online student retention’, Journal of Educators Online, vol. 11, no. 3, pp. 1–30. Available from

Swan, K., et al., (2015) ‘Metaphors for learning and the pedagogies of MOOCs’, in Meeting of the American Educational Research Association, Chicago, IL, April 2015.

Tenório, T., et al., (2016) ‘Does peer assessment in on-line learning environments work? A systematic review of the literature’, Computers in Human Behavior, vol. 64, pp. 94–107.

Thorpe, M. (2008) ‘Effective online interaction: mapping course design to bridge from research to practice’, Australasian Journal of Educational Technology, vol. 24, no. 1, pp. 57–72.

Trigwell, K., Prosser, M. & Ginns, P. (2005) ‘Phenomenographic pedagogy and a revised “Approaches to teaching inventory”’, Higher Education Research & Development, vol. 24, no. 4, pp. 349–360.

University of Ulster. (2008) Hybrid Learning Model, [online] Available at:

Vygotsky, L. S. & Cole, M. (1978) Mind in Society: the Development of Higher Psychological Processes, Harvard University Press. Cambridge, MA.

Walmsley, H. (2017) Best Practice Models for e-Design, [online] Stoke-on-Trent Available at:

Wanstreet, C. E. (2006) ‘Interaction in online learning environments: a review of the literature’, The Quarterly Review of Distance Education, vol. 7, no. 4, pp. 399–411.

Xiao, J. (2017) ‘Learner-content interaction in distance education: the weakest link in interaction research’, Distance Education, vol. 38, no. 1, pp. 123–135.

How to Cite
Walmsley-Smith, H., Machin, L., & Walton, G. (2019). The E-Design Assessment Tool: an evidence-informed approach towards a consistent terminology for quantifying online distance learning activities. Research in Learning Technology, 27.
Original Research Articles