As Information and Communication Technologies (ICT) permeate every aspect of education, supporting the preparation of technology-enhanced teaching and learning activities – the process of learning design – is becoming increasingly important. A growing number of tools have been developed to assist teachers in planning learning outcomes, activities, assessment and other aspects of learning, both inside and outside the classroom.

Learning design authoring tools generally have the common goal of facilitating the sharing, adaptation and reuse of teachers’ pedagogical ideas, and they are also useful as tools for reflection on practice (Laurillard 2012). However, as Conole (2010) notes, different tools may employ various representations and operate at different levels of granularity, from simply capturing the essence of a design to aiding in its semi-automated enactment with students. For example, LAMS (Learning Activity Management System: Dalziel 2003) facilitates the micro-level planning and automation of the learning activities, whereas the Learning Designer (Laurillard et al. 2013) facilitates design work at the macro level, aiding the design of whole learning sessions and modules, and supporting more strategic pedagogical thinking. These different foci of support require different visual representations to abstract the individual elements of a design such as learning activities and resources, so that they can be easily defined and manipulated.

As the number of learning design authoring tools continues to grow, it becomes increasingly difficult to keep track of them or to engage with each one in depth. Few researchers or practitioners have the time to try multiple tools or to compare how different tools model a particular learning scenario in order to see which one best suits their needs. This, therefore, is the motivation for this article: to capture a single learning design using several authoring tools, in order to reflect on the different perspectives they provide and to learn about the design process itself from the various methods that the tools represent. Our aim is not to compare the authoring tools to see which one reflects the original conceptual design best. Indeed, such a comparison would be inappropriate as the research teams involved in the exercise did not have access to the designers’ original conceptions, instead they only had access to the design artefacts generated by them.

Inspired by Akira Kurosawa's film “Rashomon” (1950), in which an incident is portrayed through four alternative points of view, we reveal a design narrative from the perspectives of five different learning design tools. Persico et al. (2013) perform a similar exercise using different learning design conceptual frameworks, and related exercises previously performed in the learning design community include Botturi et al. (2007) and Vignollet et al. (2008).

In the following sections, we describe the rationale and method followed in this comparative study, before outlining the processes entailed in modelling an inquiry-based learning scenario using the different learning design tools. Finally, we discuss some of the main lessons learned from these complementary views of the common design narrative and report briefly the reactions of teachers who have engaged with the tools.

The common learning scenario that we modelled was the “healthy eating” lesson from the Personal Inquiry project (Anastopoulou et al. 2012), which guided students aged 11–14 in an inquiry-based investigation into their eating habits, the food that they ate and its nutritional value.1 This learning scenario was chosen as it explored a different set of pedagogical concerns from previous exercises: the studies by Botturi et al. (2007) and Vignollet et al. (2008) were based on project-based and game-based learning situations, respectively.

The learning scenario was modelled using five of the learning design tools that were presented at the “Art and Science of Learning Design” workshop in 2011. They are all publicly available and well known within the learning design community of practice for the different levels of authoring support that they provide. We selected the tools that reflect the most recent evolution of research in the field, which has moved away from a concern with data modelling issues and now focuses on providing greater usability and conceptual support to a wider range of practitioners. The tools used were:

The Learning Designer: a tool for teachers to plan and reflect on their pedagogy at a more strategic level;

The Learning Designer3 (Laurillard et al. 2013) is a community knowledge-building tool that supports teachers in creating, modifying, finding, sharing and reusing learning designs. Its aim is to support the teaching community in collaborating on how to design and express its pedagogical ideas, especially in relation to the use of technology-enhanced learning to improve learners’ experience.

The software environment represents learning designs using formal learning concepts (e.g. Bloom's taxonomy of learning outcomes and a typology of forms of learning), underpinned by semantic technologies. This kind of computer-interpretable representation enables the tool to provide visual analytics of the learning designs (for example, in the form of pie charts representing the character of the designed learning experience) and intelligent recommendations using community knowledge. The community knowledge is expressed in terms of pedagogical patterns for different learning outcomes, a wiki of advice and guidance on ways to introduce technology-enhanced learning and alternative design ideas generated by context-aware artificial intelligence techniques. Finally, learning designs can be exported as document templates or as files that can be shared with, and imported and adapted by, colleagues in their version of the tool. The files can even be shared with students for their comment. A storage site is available for this purpose.

To model the “healthy eating” lesson in the Learning Designer, we took the lesson descriptions and transformed them into properties and “teaching–learning activities” (TLAs) in a block of sessions.

Although a detailed description of the modelling process exceeds the scope of this article, certain aspects of the use of the Learning Designer are worth highlighting. Figure 1 shows the Properties page, which describes the overall features of the design.

The detailed pedagogical features of the learning design are expressed in the Timeline screen, where the teacher-designer4 can drag and drop pre-defined TLAs, and customise them in terms of name, activity notes, group size, duration, and so on (Figure 2). TLAs are sized on the basis of their duration, and they contain an editable analysis of the likely learning experience, in terms of acquisition, inquiry, discussion, practice and/or production. The software offers suggestions for alternatives on the basis of the properties of the currently highlighted TLA (Figure 2, bottom left).

At any stage, the teacher-designer can check the Analysis page of the design (shown in Figure 3) to see the overall proportions of types of learning afforded by the design so far. This page also displays the likely time that the teacher will need for preparation and presentation.

When modelling the scenario with the Learning Designer, we found several gaps in the information in the scenario description, in particular the learning time supported, learning outcomes, student feedback, group size and duration of activities. The modelling of resources linked to activities had to be edited, since currently the tool only offers three resource types (local files, URLs and notes). Some of the activities were not described in sufficient detail to clarify what exactly learners were doing, and so they could not be categorised by the Learning Designer's taxonomy of activity types. Other activities which were grouped together as one activity in the scenario are actually several different kinds of learning activities in the Learning Designer.

The Open Graphical Learning Modeller5 (OpenGLM: Derntl, Neumann, & Oberhuemer 2011) is a learning design authoring toolkit that supports the creation of IMS Learning Design (IMS-LD) units of learning at levels A and B (IMS Global Consortium 2003). Its main goal is to provide comprehensive and intuitive IMS-LD support for teacher-designers by interpreting a graphical representation of a learning design and converting it to the required XML format as specified in the IMS-LD standard. These IMS-LD units of learning can then be run on IMS-LD-enabled learning management systems (LMS).

We modelled the “healthy eating” lesson using OpenGLM by defining the teaching and learning activities and the roles that enact those activities. These activities can in turn be connected to each other, forming a sequence. Learning materials and other resources (e.g. a discussion forum) can then be linked to the activities, so that they are available to the actors who perform them.

OpenGLM does not impose a prescribed sequence of steps for modelling a learning scenario. We started by using the lesson description to prepare a simple list of tentative activity labels and roles performing those activities (e.g. “Watch introductory short video – students” and “Split up the class – teacher”). However, this activity list was not directly translated to IMS-LD activities in OpenGLM. Instead, some teacher and student activities were merged. Teacher activities that were actually part of a student activity, such as giving instructions about the activity, were incorporated into the description of the student activity. Similarly, the presentation of content or the sharing of artefacts was integrated into the subsequent learning activity: for example, the distribution of pictures for discussion was represented as a resource in the discussion activity.

The graphical representation of the resulting activity sequence is shown in Figure 4. There are three roles – student, group and teacher – and their activities are represented using different colours along vertical “swim lanes” (the visual arrangement has no effect on the underlying IMS-LD executable package). Learning activities are represented with a solid outline, and support activities (e.g. teacher activities) have a dashed outline. Resources and tools are represented as icons within each activity box, which also includes activity descriptions, add-ons and, for teacher activities, the roles to be supported.

The resulting OpenGLM model can be shared with other designers using OpenGLM for viewing, editing and exporting to IMS-LD. The exported IMS-LD package is a valuable artefact that enables the underlying learning design to be reused through being shared with other IMS-LD authoring tools and deployed in IMS-LD runtime environments (Derntl, Neumann, & Oberhuemer 2011). However, the fact that the “healthy eating” lesson was to take place primarily in a face-to-face setting led to certain peculiarities during modelling, as IMS-LD was conceived primarily for the deployment of computer-managed learning opportunities (Derntl et al. 2012). In general, it is difficult to model non-software resources and services in IMS-LD, such as the “device” described in the lesson plan (which can be represented by an environment of services and learning objects) and other physical artefacts (e.g. pictures on a whiteboard, which have been modelled as a digital picture pool). Face-to-face activities such as a classroom discussion are equally difficult to model accurately in IMS-LD. Indeed, the “healthy eating” lesson was not described in a way that suggested computer-managed activities. Therefore, we tried to imagine the activities taking place in a virtual environment. This required thinking about virtual equivalents for the physical objects used in the scenario. For example, where shared pictures were to be annotated and discussed, we imagined an online picture pool (e.g. Flickr) as the virtual equivalent of the pictures on a whiteboard. Of course, this modified the original scenario and added interpretation bias to the formal representation of the scenario in OpenGLM.

CADMOS6 (CoursewAre Development Methodology for Open instructional Systems) is a graphical editor aimed at teachers with basic computer skills and knowledge of learning standards (Katsamani & Retalis 2011). Its main purpose is to make it easy for teachers who are not experts in learning design to create and share learning designs and to enact them in a real environment. CADMOS proposes a “separation of concerns” approach for the learning design process, which stems from the principles of Web engineering (Papasalouros, Retalis, & Papaspyrou 2004; Rossi et al. 2008). In accordance with this approach, the designer creates two different models:

A conceptual model, which describes the learning activities in which the different roles are involved, and the corresponding learning resources and services;

In CADMOS, the teacher can define simple and composite activities; however, in modelling the “healthy eating” lesson, we only defined simple activities. The conceptual model of the lesson, which is shown in Figure 5, has 14 activities: for example, “Each student watches a video and asks questions”, “The teacher presents different inquiry methods” and “Each group chooses a question and sends it to an expert”. We defined metadata for each activity, including title, description, goal, the roles performing it and the activity type (e.g. informative, communicative or assessment). At any time, the teacher-designer can view statistics of the different kinds of activities that have been used so far. We also associated resources with each activity (see Figure 5) and specified metadata for each one, including title, author, description and resource file.

The flow model of the lesson, which we created next, is shown in Figure 6. In this model, activities are set in vertical “swim lanes” in chronological order and have been divided into seven phases (i.e. sets of activities), as suggested by the original description of the lesson.

The CADMOS tool was developed primarily for designing computer-supported lessons. Thus, although we were able to represent the lesson using CADMOS, we faced a number of constraints, generally relating to the description of face-to-face situations. Certain resources (e.g. whiteboard, email and face-to-face discussion) are currently omitted from CADMOS’ list of resource types. The length of each phase was modelled using comments, as CADMOS only permits users to specify timings at the activity level. There were also certain issues in presentation: even though CADMOS allows unlimited space for modelling a design, the length and complexity of the “healthy eating” design meant that the resulting model was not very readable.

Web Collage7 (Villasclaras-Fernández et al. 2011) is an authoring tool developed to support teachers who are not experts in learning design in creating computer-supported collaborative learning (CSCL) scenarios. Its main conceptual and visual metaphor is design patterns: general solutions to recurrent design problems (Alexander, Ishikawa, & Silverstein 1977).

Web Collage integrates a set of Collaborative Learning Flow Patterns (CLFPs: Hernández-Leo et al. 2006), which describe validated collaborative pedagogy techniques such as “Jigsaw” and “Pyramid”. These CLFPs serve as ideas for suitable collaborative learning structures, and Web Collage also uses them to generate the computer-interpretable components of the design that are needed to implement the collaborative techniques. The current version of Web Collage also includes assessment design patterns in order to encourage the teacher-designer to think about assessment and to design appropriate plans (Villasclaras-Fernández et al. 2011). If a teacher-designer wishes to split a class into groups, Web Collage can retrieve a list of the students from an LMS such as Moodle, which the teacher-designer can then use to allocate individual students to their groups. When the design work is complete, Web Collage, working alongside with GLUE!-PS (Prieto et al. 2011), can implement the learning design directly in the LMS, automatically creating the required activities, groups, resources and so forth.

Our model of the “healthy eating” lesson in Web Collage is shown in Figure 7. The left side shows the “learning flow” modelling of the learning and support activities. Some of the activities (in particular, individual tasks) did not follow any of Web Collage's CLFPs, and were modelled using generic “phases”; however, others could be represented using CLFPs. For example, the activities “Propose ideas for a healthy diet” and “Compile a list of food items” were modelled as “Brainstorming” CLFPs. The group activity of selecting a number of photos and the whole-class discussion which followed it comprised an example of a Pyramid pattern. The graphical representations of the Pyramid and Brainstorming CLFPs are intended to help make the teacher-designer's intentions easily understood, including the type of collaboration and outcomes expected for those phases. The left side of Figure 7 shows only the coarse-grained phases; specific learning and support activities can be described inside each phase (see, for example, Figure 7A). Within each pattern or phase, the tool can also model the number of groups and the students’ flow across the different groups (Figure 7B), as well as the assessment plan, which here includes a quiz with subsequent feedback from the teacher (Figure 7A).

However, Web Collage's representation of the lesson also has limitations. There is no way to specify the duration of an activity, and there is only limited provision for modelling resources. For example, although the teacher-designer can define the URL of a certain Web-based resource, or specify the use of an interactive whiteboard, they cannot describe how the resource will be used, or how to access the whiteboard system. Another limitation is that, in activities other than assessment, Web Collage models the kind of task to be performed using natural language only in the description of each activity. Also, as Figure 7 shows, the model of the scenario can quickly become complex. Web Collage currently models activity sequencing, group creation and assessment. However, the scenario used in this article also included other aspects such as timings and the sharing of resources. Whether these aspects need to be modelled depends on how the scenario will be enacted: for example, to what extent the LMS will be used to manage the activities. Indeed, it is possible that these other aspects, which are interrelated, might be more easily configured using design patterns.

ScenEdit8 (Emin, Pernin, & Aguirre 2010) is a Web-based authoring tool developed to structure and support teachers and trainers in the design and reuse of blended pedagogical scenarios in formal learning. It combines two approaches to the design process; namely:

Organise the scenario by eliciting formally the intentions of the teacher-designer and representing explicit learning strategies;

ScenEdit supports the visual description of learning scenarios according to these three dimensions of the ISiS model, including the locations where the activities take place, and the tools and resources to be used by the actors. This visual representation can be exported as an XML or PDF file and shared with other teachers. ScenEdit also provides templates or patterns of different kinds (i.e. intentions, strategies or interactional situations), which have been derived from examples of good practice uncovered in the literature or in communities of practice.

The “healthy eating” lesson, as modelled in ScenEdit, is shown in Figure 8 and 9. As can be seen in Figure 8, the different types of components in an ISiS scenario are represented by symbols: triangles denote steps, squares denote intentions, circles denote strategies and clapperboards denote interactional situations. The main aim of the lesson was for pupils to develop skills in undertaking inquiry-based learning tasks. The teaching method comprised a sequencing strategy in eight phases, modelled on the activities in the nQuire tool (e.g. “Collect my evidence”). Each of these eight activities was modelled through one or more interactional situations (e.g. “Collect data from the food diary meal” and “Organise data collected”).

Figure 9 shows the ScenEdit screen on which the interactional situations are described. In terms of tools, each student needs a laptop, a camera and an Internet connection in order to access the nQuire project. In terms of resources, the nQuire website contains the data, either collected by students or provided by the teacher (diagrams, notes, description of objectives, links and so forth). Five roles were identified (pupil, group, class, teacher and nutritional expert), as well as a number of locations (including home and restaurant).

One of the main difficulties in modelling the lesson was that the scenario allowed activities to be carried out in any order, something that could not be modelled easily with ScenEdit. Using interactional situation design components, ScenEdit was able to model a sketch of the activities, specifying the kinds of resource, tool, location and role, together with the URLs linking to online materials such as the documents describing the activity, the tasks to be performed by the students and the resources they could access. However, this description, which is stored in an XML file, needs conversion tools in order to be implemented in, for example, an LMS.

From the five design representations described above, we can see how different tools emphasise different aspects of a learning design, ranging from the abstract (e.g. the “learning experience” types of the Learning Designer or CADMOS) to the specific (e.g. the URLs of resources that can be defined in the IMS-LD compliant tools). We can also note how the use of different tools – and the abstractions and processes that they represent – to specify a learning design in computer-interpretable form can shape the resulting learning design in very marked ways. For example, learning designs specified using the Learning Designer contain more detail about the pedagogical composition of lessons or their timing, while designs created with CADMOS, Web Collage and OpenGLM provide more detail about the digital resources to be used. The same is true of other tools and formalisations, such as the textual and abstract descriptions of the “healthy eating” scenario used as the common base for this study. Thus, the transformations described in this paper are merely examples of the kinds of transformations which a teacher's pedagogical ideas may undergo when they are implemented in digital form (for a further discussion on this topic, see Muñoz-Cristóbal et al. 2012).

Throughout the preceding five sections of this article, we have noted the difficulties encountered when modelling the “healthy eating” scenario in the authoring tools. These difficulties, which we have synthesised in Table 1, were normally associated with face-to-face activities (e.g. classroom discussions) and physical resources (e.g. interactive whiteboards), and were more prominent in tools compliant with the IMS-LD standard (i.e. OpenGLM, CADMOS and Web Collage). When using these tools to model the “healthy eating” scenario, the research teams often had to circumvent their limitations by modelling activity and resource types in slightly counter-intuitive ways, in contrast to the tools that rely more on natural language descriptions and teaching–learning activity types derived from teaching practice (e.g. the Learning Designer). This apparent shortcoming stems from the fact that the IMS-LD standard was devised mostly with online education in mind and, more specifically, with a view to the automation of activities in IMS-LD compliant enactment environments.

Another issue that is closely related to the previous one – and, indeed, could be seen as its converse – is the ease with which a learning design can be implemented in a digital environment such as an LMS. As Table 1 shows, the tightly modelled elements of IMS-LD compliant tools are at an advantage in this respect, since the learning designs expressed with those tools can be implemented and managed with relative ease by execution environments that also comply with that standard (e.g. Coppercore and.LRN).9 In addition, CADMOS’ learning designs can also be exported to LMS such as Moodle (Boloudakis et al. 2012). On the contrary, designs produced with tools that take less account of the execution environment (such as the Learning Designer and ScenEdit) might be harder to implement for teachers without advanced ICT skills. This is because these tools require an ad-hoc mapping of their XML output either to the aforementioned IMS-LD standard10 or to the functionalities available within an LMS (as suggested by Bower et al. 2011). Interestingly, proposals have recently been made for the semi-automatic implementation of learning designs using applications that are agnostic with respect to the authoring tools used to create the designs (e.g. GLUE!-PS: Prieto et al. 2011). The deployment process could therefore be semi-automated even for non-IMS LD tools. Of course, purely face-to-face elements that do not use digital tools or resources would remain outside this “modelling for deployment” and would be implemented by teachers manually or through socially mediated operations.

We should also note that the five tools presented in this article are intended for different target audiences and stakeholders. For example, OpenGLM is more oriented towards specialists in learning design, while the others are aimed at teachers with varying degrees of technical and learning design knowledge (teacher-designers), as shown in Table 1. Learning design authoring tools typically have at least two different audiences: those who create the learning designs using the tool and those who enact (or realise) the learning designs with learners. Depending on the educational context, these two kinds of users might be the same or different individuals: some institutions may have specialist learning designers who produce, share and reuse the designs, which are then enacted by teachers, while in other institutions, both the production and enactment roles may be fulfilled by teacher-designers. Indeed, additional audiences may have a stake in these tools, such as managers and those responsible for the institutional teaching and learning strategy (Derntl et al. 2011, Masterman & Manton 2011).

Similarly, some of the tools are intended to support particular pedagogical approaches and/or educational contexts (see Table 1). For example, OpenGLM and CADMOS lend themselves specifically to the design of online courses in an IMS-LD environment, whereas the patterns in Web Collage and ScenEdit are intended to support collaborative learning. Other tools may support designing for multiple educational contexts, as is the case with the Learning Designer, which was developed in order to stimulate teachers to make greater use of technology-enhanced learning, but can also be used to design face-to-face learning with little or no use of ICT.

The research team responsible for developing each tool has explored its usability and usefulness to the learning design community in a number of studies. The Learning Designer was evaluated iteratively in workshops and individual guided walkthroughs with lecturers in UK universities, who appreciated the way in which the tool simultaneously supported and challenged their current practice, particularly through the analysis of students’ learning experience (Masterman & Craft this issue). The usability of OpenGLM and its usefulness in creating representations of the design of courses were evaluated with a sample of 21 instructors, with favourable responses (Neumann & Oberhuemer 2009). CADMOS has been evaluated by non-expert teachers in an MSc programme (Katsamani & Retalis 2012), and it was found that the tool had a friendly interface and was simple to use. They also valued the guidance offered during the learning design process and the implementation of learning designs in Moodle. The functionality for modelling collaborative scenarios in Web Collage has been validated both by non-expert teachers and by CSCL experts, who created designs for their own practice; however, further work needed is to address the challenges posed by complex designs (Villasclaras-Fernández et al. in press). The usability of ScenEdit has been evaluated with groups of secondary school teachers in France (Emin et al. 2009).

To summarise, different tools and processes may be a better fit to different teachers, depending on their pedagogical aims, institutional and technological contexts, and so forth. The example “healthy eating” lesson, with its emphasis on inquiry-based learning that involved substantial face-to-face work in the classroom and at home, was not equally suited to every tool in our selection. This fact in itself provides an additional insight: face-to-face activities have not been adequately considered in research into the modelling of learning designs, and so we recommend this as an avenue for future work in the field.

We have thus reached the end of this second “Rashomon” exercise, in which we have compared different learning design authoring tools. Our purpose has not been to declare a winner; rather, it has been to highlight the different ways in which the tools conceptualise and represent learning, and to provide readers with a means to decide which one might best suit their needs. As Sobreira and Tchounikine (2012) comment, “languages and editors are trade-offs, design decisions relating to matters of concern, targeted users and/or expected utilisations”. We hope that we have indeed provided informative insights into these trade-offs.

By looking from different points of view, and through the lenses of the different tools, we have also highlighted a number of outstanding issues that are relevant to the wider learning design community: namely, the difficulties of using computers to model learning and teaching that happens independently of technology; the difficulties of designing learning at different scales of time and complexity; and the various stakeholders who use, and who may be influenced by, the activity of learning design and its mediating tools. Even if a universal solution to these matters may be unattainable, we should continue to seek viable smaller-scale solutions that address individual contexts. Indeed, this is reflected in the continuing increase in authoring tools available to teachers: descriptions and resources for more tools can be found on the LD-Grid website,11 in addition to the ones analysed in this article.