Guidelines for supporting re-use of existing digital learning materials and methods in higher education

10.1080/09687760903033074 ALT-J, Research in Learning Technology 0968 7769 (print)/1741-1629 (onli e) Original Article 2 09 ssoci tion for Learning Techno ogy 7 000July 2009 D JudithSch one boom j.i. choon b m@uv .nl The literature on the re-use of learning materials has largely focused on the development of materials. This paper explores how re-use can be stimulated after learning materials have been developed and made available. We searched for and developed guidelines that support staff and/or management most frequently adopt in cases of (un)successful re-use of existing digital teaching and learning materials and methods by teachers in higher education. In a grounded theory approach, we collected existing guidelines from 11 literature studies, and developed new guidelines from 19 (mainly Dutch) case studies. Through constant comparison, we developed ‘low-level’ guidelines, which were classified into high-level guidelines. Five high-level guidelines turned out to be robust when subjected to repeated comparison with the case studies. Further, this paper elaborates existing guidelines by showing specific elaborations of them in our case studies.


Introduction
There is now an extensive literature on the re-use of learning objects, i.e. resources "usually digital and web-based, that can be used and re-used to support learning" (Wiley 2000).Up till now, the literature has focused on the development of materials, with significant attention being paid to the design of learning objects to optimise their re-use.This has led to a debate on the extent to which learning objects should contain such diverse elements as context (Quinton 2007), pedagogy (McCormick 2003) and learning objectives (Boyle 2003).Learning materials have been classified into various hierarchies (Cisco Systems 2003;Wagner 2002;Wiley 2000), and learning object design trade-offs have been formulated, e.g. between size and re-usability (South and Monson 2000).Attention has also been paid to encouraging re-use by making learning materials available through repositories, using metadata (Sonntag 2007).This has led to the formulation of principles for good metadata (NISO Press 2004); the development of metadata standards such as Learning Object Metadata (LOM) (Institute of Electrical and Electronics Engineers 2002); a discussion on the use of tagging versus metadata (Smith 2008); and the formulation of a trade-off between learning material size and costs of metadata (Wiley 2002).
However, much less attention has been paid to the question of how re-use can be stimulated after development.In this situation, one or more collections of existing learning materials are available, and the question is how can teachers be stimulated to re-use these existing materials?This question has become increasingly relevant in recent years, as an abundance of learning materials has become available through the Internet and through institutional and shared repositories.In this paper we will focus on the re-use of existing learning materials in Dutch higher education.
Re-using existing learning materials in higher education is different from developing re-usable learning materials, in several respects.Firstly, it is educational support staff and managers rather than developers who support re-use in this situation.Secondly, support staff have fewer possibilities than developers to create an optimal situation for re-use.They will be confronted with a range of existing materials, varying in terms of fit to teachers' needs, quality of the metadata and appropriateness of their size.Thirdly, while developers can stimulate re-use by developing high-quality learning materials, support staff focus must be on the teachers, coaching them towards successful re-use in their teaching.
Although this post hoc stimulation of re-use has received less attention than the development of learning materials, several guidelines for it can be found in the literature.An example is peer review (Campbell 2003;Lim 2007;Metros, Bennett, and Diaz 2001), which was shown to resolve teachers' anxiety about the quality of the learning materials (Littlejohn 2003a).
The aim of our study is to bring together the most important guidelines from the literature and develop additional guidelines from existing case studies.We develop a theory for supporting the re-use of existing learning materials, which is both 'grounded' in the data (Charmaz 2006) and 'humble' in the sense of Cobb et al. (2003, 10), the theory being "concerned with domain-specific [support] processes, … accountable to the activity of design", and "provid[ing] detailed guidance in organising [support]".Although these guidelines may be matched onto stages of Rogers' (2003) 'grand theory' of the innovation process, this is not the focus of our paper.Most of the research for our study was conducted in a six-month study for SURF, the Dutch foundation for ICT in Higher Education (Schoonenboom et al. 2007).
Our main research question is: What guidelines do support staff and/or management most frequently follow in successful and unsuccessful cases of re-use of existing digital teaching and learning materials and methods by teachers in higher education?How can these guidelines be classified?
In this research question, following a guideline can be done either explicitly or implicitly.Some guidelines were mentioned by participants as guidelines they had followed, others were derived from information on cases of re-use, and participants may not have been aware of them.We looked at successful and unsuccessful cases of re-use of existing learning materials, as we would expect that useful guidelines should contribute to the success of re-use (and ignoring them may lead to unsuccessful reuse).Our research aims to discover the guidelines most frequently followed.The more often a guideline is encountered in successful cases, the more likely it is that its success is not a coincidence, and the more useful it will be to educational support staff.Guidelines are classified not only to arrive at a more manageable presentation, but also to arrive at more general guidelines to be followed, and to inform practice in several ways.
Our criteria for including learning materials and methods is whether they can be used by individual teachers in their own teaching.Examples include materials such as study tasks, courses, pictures or theses, or methods such as working in small virtual groups, concept mapping or the use of webcams.Outside the scope of this research lies, for example, the institution-wide deployment of competence-based learning, or the development of a master degree at a distance.

Design
Guidelines were derived through two methods.Firstly, we searched for guidelines in a sample of relevant literature.These guidelines were not treated as 'proven important guidelines', but as input for the formulation of guidelines.Secondly, we derived guidelines from a sample of case studies of successful and unsuccessful re-use of existing learning materials.We then applied a grounded theory approach (Charmaz 2006), in which we derived general guidelines from the literature and case studies samples together, using constant comparison.Finally, the resulting guidelines were compared with the recent literature.

Sampling from practice
We followed a multi-stage purposeful sampling strategy (Onwuegbuzie and Leech 2007).Our first sample contained case studies of re-use of existing digital learning materials and methods in Dutch higher education.Following the general principles of Robert Stake (1995Stake ( , 2006)), we selected those cases from which we expected to learn most.We used the following criteria for selecting case studies: (1) we knew the cases; (2) they were recent; (3) descriptions with some detail were available; (4) we knew something interesting was done on re-use of existing materials; (5) something had been learned and/or was assumed by the participants on success or fail factors with reuse; and (6) which together led to a variety of guidelines.On the basis of the first five criteria, we selected a first set of case studies.
Since it turned out that the guidelines derived from these case studies were too limited in scope, we then used theoretical sampling (Glaser and Strauss 1967) to increase the variety of the guidelines, to ensure they covered the whole innovation process and to find more evidence and counterevidence for our guidelines-in-development.We added a second sample of case studies from an existing collection, which had been brought together in our earlier study Learning objects in practice (Schoonenboom et al. 2006).In this study, carried out between 2004 and 2006, we identified and contributed to solutions for important problems for those working with learning objects in Dutch higher education.In this study, a set of six Dutch case studies had been collected, which met the following criteria: (1) they were related to digital learning materials; (2) they had successfully applied for funding with the Dutch national SURF organisation or the Dutch Digital University; (3) they had been finished successfully; and (4) the results were used in practice.This second sample had in the original study already been supplemented by two other case studies of successful practice and two running experiments, in order to increase diversity and to increase the validity of the discovered guidelines.
The literature that we used as data for developing guidelines was also taken from our study Learning objects in practice (Schoonenboom et al. 2006).At the start of this project in 2004, four learning objects experts, each with four or more years of experience (both at a national and an international level) conducted a literature review in the problems of working with digital learning materials.
A description of the 19 case studies and 11 literature studies used in the analyses can be found in Schoonenboom et al. (2007).

Procedure
The procedure consisted of four steps.Figure 1 shows the results of each step, which in the text are indicated in bold.Step 1. Identification of 'actions' in case studies Detailed descriptions of the case studies were either collected or prepared by the researchers.Each description included a description of the success or lack of success of the practice.Within each example of practice, 'actions' were identified, which seemed to have yielded or contributed to its success or lack of success.

The practice:
The department of Psychology of the Erasmus University in Rotterdam, The Netherlands, has set up a repository which contains all relevant educational materials of the Psychology program (articles, chapters of books, lectures, animations, video clips, presentations).Materials are collected by a small team.

Description of success or lack of success:
In the first half year, 4500 resources were collected; in a few years, the collection had grown to include 7500 resources, and the project leader claims that users are enthusiastic.

Short descriptions of actions:
(1) The repository is organised such as to support the departmentÕs central pedagogy of problem-based learning.
(2) Students can get a view on materials that might be of interest for their specific learning goals from different perspectives, such as per problem, course, or program.(3) Students can add ratings to materials, write notes or a review and make these visible to others.
Low-level guidelines: Ad 1.Within courses, materials and methods should support a more comprehensive pedagogical approach or educational ambition (high-level guideline: 1).Ad 2. To the extent that teachers and students themselves have to search or browse for learning materials, one search or browse attempt should yield enough, but not too much materials which are sensibly ordered (high-level guideline: 2).Ad 3. Enable the addition of annotations by users, which can give teachers and students an impression of the quality of the materials and methods in practice (high-level guideline: 4).Step 2. Formulation of low-level guidelines For each action, a short description was created.The literature sample was scanned for existing guidelines, and short descriptions of these guidelines were prepared.From the short descriptions of both the existing guidelines in the literature and the practice actions, low-level guidelines were derived.The derivation of the low-level guidelines was an iterative process, using repeated comparison (Charmaz 2006;Glaser and Strauss 1967).First, a set of guidelines was formulated, and as many short descriptions as possible were grouped under these guidelines.Reflection on the remaining descriptions led to the formulation of new guidelines and so on, until each short description was grouped under one low-level guideline.Then, the literature and the detailed descriptions of the case studies were analysed again, looking for additional instances of actions that would fit under the low-level guidelines.
Step 3. Formulation of high-level guidelines Finally, the low-level guidelines were categorised into high-level guidelines, such as 'Provide easy access to the materials and methods' in Figure 2. The same procedure of repeated comparison was used as in step 2.
Step 4. Testing the validity of the guidelines The resulting high-level guidelines were tested for robustness.We performed a splithalf test, in which we randomly divided the literature and practice cases into two.The split-half test showed that guidelines derived from 10 or more cases were robust in that they could be derived from either half of the cases.These five guidelines were kept, and the other non-robust guidelines were removed.Finally, we checked a recent three-volume work on learning objects for remarks on these guidelines (Harman and Koohang 2007;Koohang andHarman 2007a, 2007b).

Validity
Throughout our study, we had to deal with a variety of validity threats.Consequently, we used a variety of strategies to deal with these threats.
• Case studies were divided among the researchers, and each researcher analysed those practices with which he or she was most familiar (Stake 1995(Stake , 2006)).Most of the case studies we knew very well and over a long period, thereby reducing the risk of misinterpretations (Becker 1970).• We used multiple data sources; we used multiple cases, which were multiple in kind, including both descriptions of case studies and literature studies, and by multiple authors.This reduced the risk of biases, resulting from studying one specific case using a specific type of resource by one specific researcher.• Descriptions of case studies had been validated by participants.This applied to all samples.• To ensure that the guidelines would be robust enough to capture a variety of situations, we monitored the guidelines at any point in our research project, to see if they provided enough variation.When it turned out that the guidelines of our first sample did not explicitly refer to digital learning materials, we included a second sample of case studies and literature studies.
• The descriptions and analyses of the six most demanding, initially selected case studies were either done by two researchers, or by one researcher and checked by another.• Within each individual practice, we devoted one question in our analysis template to a systematic search for additional evidence and for rival explanations of the presumed relation between action and success (or failure).Evidence often included signs of growing and sustained use.Sometimes, the presumed relation was backed by evidence from the literature.• With each guideline that we had derived from an individual practice, we checked its occurrence in all other case studies.Furthermore, we checked whether guidelines found in one case study were contradicted in other case studies, but we found no examples of this.• Reformulations in the guidelines were thoroughly discussed between researchers.

Outcomes
Figure 2 shows the five high-level guidelines that we derived.After each guideline, the number of cases on which it is based is indicated in brackets.
Figure 2. The five high-level guidelines for supporting the re-use of existing digital learning materials and methods; numbers indicate the number of the 30 case studies and literature studies on which each item is based.
Guideline 1 is mostly relevant to managers who at the beginning of the process are considering re-use of existing learning materials.Guidelines 2-5 become relevant once the decision for re-use has been taken and is mostly relevant to support staff.
The sections below each present one high-level guideline, with an explanation of how it was derived from its underlying low-level guidelines.A document is available in Dutch that shows for all guidelines the whole derivation process from detailed descriptions to actions to low-level guidelines to high-level guidelines (Schoonenboom et al. 2007).

Develop an appropriate vision on the incipient re-use of the materials and methods within the programme, department or institution
From the beginning an appropriate vision of re-use should be developed by the teacher(s), department or institution involved.Our study showed that the following elements of this vision, which had been identified in the literature, occur frequently in the case studies: (1) Develop an appropriate vision on the incipient re-use of the materials within the programme, department or institution (N=14).
(3) Provide materials that fit well the teachersÕ own needs and way of teaching (N=14).
(4) Provide materials of good quality, and enable teachers and students to judge the quality of materials and methods (N=10).
(5) Enable necessary adaptations of the materials and methods to the context (N=18).(1) The materials should have a demonstrable added value for teaching (Rogers 2003), which is visible to the teachers and managers involved.Examples include materials that can be taken into account in students' assessment and materials on which use feedback can be provided by either teacher or students.
(2) Re-use of materials should be considered a gradual process of small changes.
Teachers are more inclined to apply small innovations than to change the whole curriculum (Rogers 2003).
(3) Re-use of materials should be viewed as a way of improving education, not as a way of reducing costs.In fact, costs may go up (Pennings et al. 2005).Our case studies show that institutions should consider costs for updating the materials (Littlejohn, Jung, and Broumley 2003).Also, materials or methods that might be provided for free during the course of a project might require a license after the project has finished.
New in our study is that it shows examples in practice of visions that may stimulate re-use.We found that this vision might be a big or a small idea.In one practice, a repository was established in support of the 'big' idea of problem-based learning.Existing learning materials were collected that together support the whole cycle of problem-based learning in the department involved.In another practice, the 'small' idea of visual learning was prominent, and learning materials and methods were collected that could contribute to this idea.Although earlier literature has identified ideas such as problem-based learning as a basis for developing new learning materials (many examples in Koohang and Harman 2007a;Wiley 2003), we found that this is also relevant in working with existing learning materials.

Provide easy access to the materials and methods
If it is too difficult for teachers to get access to the digital learning materials and methods, they will not use them.This point has received much attention in the literature on learning object development.As a first solution to this problem, the importance of good quality metadata has been emphasised, so that users find relevant, and neither too few nor too many learning materials (Koper 2003;Sonntag 2007).A second solution is the use of 'tailored wrappers', information on the local context of use, which is added to existing learning materials (Quinton 2007).
These two solutions show up in our case studies, but with two elaborations.First, two case studies show that it can be useful not to provide access to all learning materials of an available repository, but instead to make a selection of materials that are deemed relevant to the teachers involved.Second, one practice made use of tailored wrappers which contained: information on which colleagues had already used these materials; information on where one could get help; own metadata with familiar local terms which was added on top of the existing metadata.

Provide materials and methods that are a good fit to teachers' own needs and way of teaching
If materials and methods are a good fit to teachers' own needs and ways of teaching, then it will be easy for teachers to use them (Rogers 2003).On the other hand, learning materials cannot easily be used if, for example, the cultural gap with the teacher's own practice is too wide (Littlejohn 2003b), or each piece is too large, containing too much material that is not useful (Downes 2002).
In line with the literature, our case studies show that if the conditions of use of the target institution are too different from those where the materials were developed, then re-use will not be possible.A good fit also implies that teachers should be able to use the materials selectively, only the elements they deem to be useful.Student needs be considered too; for example, materials that fit the students' future professional practice can be particularly useful.Materials that are a poor fit include those that are too complex for the level at which they are intended to be used, or those that clash significantly with aspects of the pedagogy in use, or if they are built upon an 'alien' pedagogy.

Provide materials of good quality, and enable teachers and students to judge the quality of materials
Learning materials should be of good quality.At the level of the repository, too much variation between learning materials is considered a problem (Downes 2002).Teachers and students should be enabled to judge their quality (Littlejohn 2003a); peer review is seen as a very useful method (Metros, Bennett, and Diaz 2001).
In line with the literature, we found that the collection of materials in one repository should be consistent, and with respect to the domain involved the materials should be as complete as possible, and of equal quality.For students, it is important that multimedia and interactive materials are provided (Ravenscroft, Boyle, and Jones 2004).Regarding judgement of quality by teachers and students, we found in the cases that we studied the use of peer review, in which peer teachers tested the usability of materials before use in the institution.In one practice, peer review was seen as especially important for materials on the web because of the unclear status of these materials.The second form of judgement in our case studies was rating and annotating by users to inform their peers on how to use the learning material involved (Sonntag 2007).
In addition to what is known from the literature, we found in one example of practice a clear limit to differences in the size of learning materials.Here, teachers did not appreciate that the results of one search could include materials as diverse as study tasks and pictures.In examining peer review, we found that teachers rely partly on whether they know the person who developed the materials and the status that this person has.

Enable necessary adaptations of the materials and methods to the context
Teachers should be able to use the materials and methods in their teaching.An important element of this is that they should be able to adapt the materials and methods to their own teaching context.Several factors might hinder use.As a necessary condition, learning materials should 'run' in the institution's virtual learning environment.Also, copyright problems concerning the re-use of the materials should be solved (Campbell 2003).Teachers must be provided with the necessary tools and support.Lack of userfriendly tools with which teachers can re-use learning objects is considered a problem (McNaught 2003).Also, time is an important constraining factor (Littlejohn, Jung, and Broumley 2003), and enough time should be provided for teachers to evaluate and test materials.
In line with these findings in the literature, we found in our research that the userfriendliness of the materials and the occurrence of technical problems were also constraining factors.If adaptation takes too much time, teachers will not use the materials.We also found the need for specific forms of support.Support should be provided to users at several levels of expertise, for both students, managers, beginning teachers and experienced teachers.Explaining the pedagogical possibilities of the materials helps, as this may not be clear from the materials themselves.Collective support should be complemented by individual support.
In addition to what was found in the literature, our case studies showed that use of the materials is stimulated by removing two important practical hindrances to re-use by enabling teachers and students to work with the materials from home and by making the necessary software freely available.One additional form of support in the case studies was to provide support to beginners by letting them join more experienced colleagues.

Conclusions
This paper has contributed to the discussion on re-use of learning materials in several ways.It has brought together existing answers and developed new answers to questions that had perhaps been solved for the development of learning materials from scratch, but not for situations in which teachers and their supporters are working with existing learning materials.Our case studies have shown that also in this situation, an overarching educational idea or vision is required.Furthermore, although the existing materials may not all be optimal, access can be optimised by making a relevant selection, by adding own metadata on top of existing metadata and by bringing down the range of different levels of aggregation of learning materials that teachers can find in one search.
Our study has grounded existing guidelines and provided an overview by showing which of these are important in practice, in that they occur often in our cases.It has specified existing guidelines, in that it has shown some specific elaborations of these guidelines in practice.
Finally, our paper has demonstrated a method for deriving guidelines from case studies, testing their robustness and combining these with findings in the literature.

Figure 1 .
Figure 1.Summary of one practice, showing the elements of the derivation of guidelines: description of success, short descriptions of actions, low-level guidelines and high-level guidelines; numbers of high-level guidelines refer to their number in Figure 2.

Figure 1 .
Figure 1.Summary of one practice, showing the elements of the derivation of guidelines: description of success, short descriptions of actions, low-level guidelines and high-level guidelines; numbers of high-level guidelines refer to their number in Figure 2.

Figure 2 .
Figure2.The five high-level guidelines for supporting the re-use of existing digital learning materials and methods; numbers indicate the number of the 30 case studies and literature studies on which each item is based.