Learning Design research has resulted in the development of a number of software tools to support teachers in designing learning experiences for their students. These tools include Phoebe (Masterman and Manton 2011), the London Pedagogy Planner (San Diego et al. 2007), Compendium LD (Conole 2013) and the Learning Designer, which is the subject of this article.

Design support tools are intended to help teachers through the series of decisions involved in bringing together into a learning design the aims, learning outcomes, teaching approach, method of assessment and the activities that learners will carry out in a particular sequence of learning, together with the resources needed and the constraints on the learning situation such as the learning environment and learner characteristics. The learning in question may occupy a single session (for example, tutorial, lecture, seminar or practical class), or it may extend across a module (i.e. a series of sessions related through an overarching topic of study and learning outcomes) or an entire programme.

Design can be characterised as an individual or collective cognitive activity, which is normally externalised through a series of intermediate representations, each of which can be characterised by its underlying structure, or form. The purpose of these representations is to facilitate the designer's thinking and, in collaborative design activities, to share the emerging learning design with others in a process that culminates in a final assemblage of artefacts: traditionally, a tabular representation of the learning activities and the resources that will be required during the learning session.

Several representations, with differing forms, have been proposed by Learning Design research for supporting the process and product of the design activity. They include flowchart-style visualisations of learning activities (LAMS: Dalziel 2003), a columnar arrangement of resources, tasks and supports (Agostinho 2006), and representations of the learning design in concept-map form (Conole 2013). However, what is missing from their descriptions is a principled account of the rationale behind the chosen form of representation and its associated notation: that is, how they are intended to facilitate the cognitive tasks involved in planning students’ learning experiences. This is not to suggest that these representations are not effective in terms of their usability and value to lecturers. Rather, we propose that adopting an explicitly theory-informed approach to the design and evaluation of such representations is advantageous in two ways. Firstly, it should optimise the user's task and, thereby, maximise the usability and usefulness of the representations. Secondly, it should contribute to our understanding of users’ differing – and sometimes contradictory – reactions to the representations, as revealed by evaluation data. This should equip us to address any shortcomings in the representations more effectively.

This paper puts forward one possible theoretical basis for designing and evaluating such representations: the framework of epistemic efficacy proposed by Peterson (1996) in his editorial introduction to a collection of papers authored largely by researchers from the cognitive science field. The framework draws on a number of schemes for classifying representations according to the factors that account for their effectiveness in supporting cognitive tasks, and combines these factors under a unified set of “general headings” that nonetheless acknowledge the “web of relations and trade-offs” entailed. It begins by outlining the five dimensions of fit that constitute epistemic efficacy, illustrating each one with examples drawn from the initial research and prototyping phases of the Learning Designer project. It moves on to show how these dimensions are manifested in representations within the Learning Designer tool that enable teachers to model their pedagogy in a constructionist microworld. It then uses epistemic efficacy as a framework for analysing evaluation data collected by the project, discusses its contribution to answering the questions addressed in the evaluation of the Learning Designer and appraises its overall value to the design community.

If representations are to be offered to teachers specifically to facilitate the pedagogic design process, then the developers of supportive digital tools should have a theory-informed understanding of what forms of representation are more, or less, conducive to this purpose. Previous research by the first author has drawn from the cognitive science literature in order to analyse lecturers’ relationships to existing forms of representation (Masterman 2009) and to design alternative forms where the existing ones fall short (for example, representations to support students’ reasoning about problems of historical causation: Masterman 2004). The particular theoretical framework adopted in those studies and, hence, in the present paper, is epistemic efficacy, operationalised through its five dimensions of fit (Peterson 1996): ontology, task, process, user and circumstance.

In outlining the framework here, we draw from data collected from lecturers during the requirements-gathering elicitation phase of the Learning Designer project, when we interviewed lecturers about their current design practice. A fuller account of this initial phase and a summary of the findings from the interviews can be found in Laurillard et al. (2013) and Masterman (2013). However, many of the quotations from the data are published here for the first time.

Having outlined the five dimensions of fit with illustrations from teachers’ design practice, we now turn to their application in the design and evaluation of representations of pedagogy in the Learning Designer tool. In this section of the article, we describe its intended purpose and the representations that were designed in order to enable teachers to model their pedagogy.

The Learning Designer tool has been developed to help teachers to develop their practice within a knowledge-building community of educators. In so doing, it goes beyond merely supporting tasks in the design process such as specifying learning outcomes, durations, learning activities and resources, and so forth. Rather, it provides an environment, or microworld, in which teachers can manipulate representations containing these elements in order to explore, reflect on and adapt their ideas until they have created a learning design which meets their objectives, and which they can share with others (Laurillard et al. 2013).

The computational model underlying the microworld is an ontology of Learning Design, which was developed by the project team through a lengthy and detailed collaborative process of abstraction from a wide range of artefacts. These included lesson plans and module design documents collected from interviewees and from the six institutions involved in the project. The ontology itself was implemented in the software using a standard methodology, described by Charlton, Magoulas and Laurillard (2012).

“Pedagogy” is broadly interpreted here as “learning in the context of teaching, and teaching that has learning as its goal” (Beetham and Sharpe 2007, p. 2). The modelling functionality in the Learning Designer enables teachers to model the kind of learning that their students might experience when the learning design is realised in a learning session, and to explore how changes in the type of learning activity, its duration and/or the use of digital technology could affect that learning experience. This entailed extending the Learning Designer's ontology with two additional sets of concepts, which are defined as properties (attributes) of each type of learning activity:

The relative proportions of the different types of cognitive activity in which learners engage when performing the learning activity: acquisition (i.e. reading, listening or watching), inquiry, discussion, practice and production;

The concepts were derived from Laurillard's (2002) Conversational Framework, and the relative proportions of cognitive activity were based on data collected over a number of years at one of the partner institutions in the project. However, users can modify these default values according to their context. It is important to emphasise that the Learning Designer only provides an indication of the learning experience that might ensue from these changes since learning design is, ultimately, design for learning (Beetham and Sharpe 2007).

To support the new modelling task and expose the additional elements from the ontology, we selected four forms of representation with which users would be broadly familiar in their professional and everyday lives: timeline, bar chart, table and pie-chart. These are deployed in two linked “views”: Timeline and Analysis.

The Timeline view (Figure 2) is where the teacher assembles the teaching and learning activities (TLAs) and determines their duration, group size (in collaborative learning) and the resources required. The timeline addresses the graphical constraining issues of the tabular representation, in that the relative durations of the different learning activities are now clear. For example, in comparison with Figure 1, the teacher's initial briefing can now clearly be seen to be one-third the length of data collection. However, although the timeline exploits the familiar linear conceptualisation of time, it serves a slightly different purpose in that it shows the accumulation of learning time, not elapsed time. This means that all of the learning activities must appear on the timeline contiguously with each other, even if they are to be carried out on, say, different days. In this way, the Learning Designer functions as a design tool, rather than as a planning or scheduling tool – i.e. the purpose that might more usually be associated with a timeline. Our selection of this form of representation, therefore, carried potential implications for process-fit.

The cognitive activities are represented in this view in two ways. The bar chart within each learning activity on the timeline shows how that activity is comprised of the different types of cognitive activity. For example, students merely listen and watch (“acquisition”) during the teacher's initial briefing, but collecting data in collaboration with other students is a more varied exercise that includes a substantial element of active learning through inquiry. However, the bar chart is actually a re-representation of numeric information, which is shown in tabular form in the “Properties” pane on the right: namely, the proportions for the learning activity, which is currently highlighted on the timeline. For example, data collection, an activity of the type “TEL resource-based group activity,” is estimated to involve 45% acquisition, 25% discussion and 30% inquiry. By seeing the proportions expressed quantitatively, the teacher can decide whether they are appropriate to the way in which she normally designs such activities, and modify them. The changes will be reflected in the bar chart accordingly.

Although the teacher can make an overall inference regarding the proportion of students’ learning taken up in different cognitive activities from the bar charts in the timeline, this requires some cognitive effort. Moreover, the timeline does not give any visual indication of the learning experience. For a cumulative representation of these aspects of learning, the teacher can switch to the Learning Designer's Analysis view, shown in Figure 3.

The cognitive activities have been aggregated into a single pie-chart for the whole lesson, using the same colour coding as on the timeline. The different types of learning experience supported by the learning activities are represented as a stacked bar. Unsurprisingly, the dominance of discussion has resulted in an overwhelmingly social learning experience, but there is no individualisation.

The modelling functionality can be seen at work by comparing Figures 3 and 4. Figure 4 assumes that the group activities to plan, collect and analyse the data have been replaced by, respectively, a teacher-led discussion, and data collection and analysis conducted by students working individually. Moreover, the final group discussion has been replaced by an individual essay, which is marked by the teacher.

The cognitive activity of discussion has been much reduced, and there is a greater element of practice and production (writing the essay). The learning experience has also changed, with “social” learning making way for “one-size-fits-all” and “individualised” learning (the latter is accounted for by the feedback which the student will receive on their essay). Neither of these two models is right or wrong: what the Learning Designer has done is to help the teacher to see in advance how variations in learning activities can alter the students’ experience. They can thus fine-tune their design on the timeline, checking the outcomes in Analysis view until it approximates to the kind of learning that they want their students to have.

The principal dimensions which have been considered in this section are task-fit, process-fit, ontology-fit and, to a lesser extent, user-fit. This is not to say that circumstance-fit (usability) was not considered in the design of these representations; however, apart from adding learning activities from the palette to the timeline and editing the values in the table of learning activity properties, users do not directly manipulate the representations themselves. Moreover, circumstance-fit was compromised to some extent by the incorporation of ready-made open source code for the timeline for the sake of efficiency. This limited the extent to which the timeline could be modified, both to meet our initial user requirements and to address subsequent suggestions by evaluation participants.

The Learning Designer was evaluated with experienced and early-career lecturers in several UK universities as part of an iterative cycle of design, development, evaluation and redesign. To concentrate on specific aspects of the tool, we conducted guided walkthroughs with individual lecturers; to try it out with particular demographic groups, we ran workshops. We collected qualitative data regarding the modelling functionality and its representations in four discrete events as follows:

November 2010–January 2011: walkthroughs with 10 experienced lecturers;

The walkthroughs were audio-recorded. Qualitative data were gathered in the workshops through free-text responses to reflective questionnaires and through audio-recorded plenary discussions facilitated by the researchers.

In relation to the modelling functionality and representations, we wanted to find out:

Do lecturers understand the concepts and terminology?

In this section, we analyse data from these evaluations in terms of the five dimensions of fit.

To answer our first evaluation question, “Do lecturers understand the concepts and terminology?”, we can draw on data collected in relation to domain-fit. In the main, lecturers understood the Learning Designer's terminology. However, it was not wholly surprising that they contested the categories of cognitive activity, since not only do alternative typologies exist (such as Conole's pedagogy profile: Conole 2013), but the disagreement also reflected a broader issue in our research. Although we were able to identify a core set of commonly used concepts when developing the ontology, we found that they can be named differently; for example, a “module” can also be known as a “unit” or “course.” Moreover, concepts can be operationalised in different ways: for example, the duration of a learning session can be measured in minutes or in hours (and fractions thereof). These differences emphasise the need for users to customise the ontology to their needs.

Data in the categories of process-fit and user-fit help address question 2, “Do the representations make sense to lecturers?” We noted that lecturers in our evaluations were able to adapt to the unexpected (to them) cognitive operations supported by the timeline; however, they had the benefit of a researcher to point out this novel use to them. The question therefore remains as to how quickly a lecturer coming to the tool without such support would become aware of it. This situation is also reflected in the comments of lecturers under the heading “circumstance-fit” in the previous section, and suggests that the Learning Designer is best introduced to lecturers in the context of staff development initiatives.

Another question raised in relation to process-fit was the quantification of the different proportions of cognitive activities comprising a learning activity, which for at least one lecturer came as an unnatural activity. This suggests that a semi-quantitative approach to adjusting the proportions might be desirable, possibly by concealing the numeric data beneath “sliders” in the user interface. In manipulating these, the lecturer would merely need to think in terms of, for example, “mostly discussion,” “very little acquisition” or “roughly equal proportions of inquiry and practice.” These approximations would then be translated into numeric values by the tool, but the user need never be exposed to them.

User-fit is closely associated with process-fit, since the ease with which users read off the information in a representation depends on how comfortable they are with different representational forms. However, our criterion for classifying data in this category depends on participants’ level of self-awareness, and so we have to take their contributions at face value. An appreciation of user-fit can help to identify those issues that can be reasonably reliably attributed to individual differences: in other words, it means we must acknowledge that some lecturers will not be won over to the tool.

Question 3, “Do lecturers find the ability to model these aspects of their pedagogy useful?”, invited an analysis of the data categorised under task-fit. Lecturers’ mixed opinions regarding the introduction of a new task (modelling their pedagogy) suggests that introducing an additional cognitive burden is acceptable provided that the tool offers them something in return. In the case of the Learning Designer, it offers insights into students’ possible learning experience that enable a lecturer to review their design and to make adjustments that have a positive impact on students’ satisfaction as well as their performance. This notion of repayment for additional effort applies also to the intended role of the Learning Designer in supporting knowledge-building among the teaching community. As Falconer and Littlejohn (2009) note, a teacher contemplating reusing a particular learning design needs to have sufficient information about the pedagogic intention underlying that design in order to make an informed decision about its usefulness. This requires the person who is sharing the design to make explicit some aspects of their pedagogy that are normally tacit, and so they may be more motivated to do so if they too can benefit.

In this paper, we have sought to demonstrate how epistemic efficacy can be applied both as a prescriptive framework for designing representations and as an analytical framework for evaluating those representations. To illustrate our case we have used the “modelling” feature of the Learning Designer. In appraising the extent to which the goals in designing the representations have been met, we have also been able to understand the reasons why certain users have found them wanting and, hence, how we might address those shortcomings.

We should emphasise that the framework is not intended to act as a guide to deploying existing representations in particular design tasks, and so it stands in contrast to the typology of representations offered by Conole (2013).

An additional strength of epistemic efficacy – and, hence, an argument for using it for the purposes described here – is that each dimension is reflected in respected empirical research into the cognitive aspects of representations (some of which was cited in our initial outline earlier in the article). This means that we can bring together, into a coherent and cohesive whole, findings from different pieces of research, each of which on its own can only offer partial insights into the properties and affordances of representations. Of course, one must take care that these different research findings are compatible with each other and avoid a “pick-and-mix” approach that disregards inconvenient contradictions. We trust that future applications of the framework will reinforce its value to the design community.