The technological affordances of computers and the Internet have made the production of learning resources easier, leading to a proliferation of resources that may be shared between institutions and individuals, therefore enabling the possibility of open educational resources (OER). According to proponents of OER such as McGreal, Kinuthia, and Marshall (2013), ‘the free and open sharing of educational resources is essential to the building of ubiquitous learning networks and reducing the knowledge divide that separates and partitions societies’ (p. xviii).

Indeed, prominent educators, institutions and international organisations have identified the need to ‘open up’ educational resources as a crucial next step in the evolution of education. This position is summarised in point A of the Paris OER Declaration, which recommends that States

Promote and use OER to widen access to education at all levels, both formal and non-formal, in a perspective of lifelong learning, thus contributing to social inclusion, gender equity and special needs education. Improve both cost-efficiency and quality of teaching and learning outcomes through greater use of OER. (UNESCO 2012, p. 1)

For Hylén (2006), OER is a means to promote the dissemination of knowledge more widely, increasing the pace of development and, additionally, increasing the quality of education and reducing social inequalities. According to Butcher (2010), the sharing of quality open resources unleashes the power of resource-based learning and, for Tosato and Bodi (2011), reusing and remixing teaching materials produced by others can unleash the creativity of educators, encouraging them to rethink the ways in which they deliver their teaching and encouraging collaboration in the scholarly community around resource improvement and adaptation.

Ferguson and Shum (2012), however, sound a note of caution, contending that while OER can ‘greatly improve the quality of material available online to learners … this wealth of resources can leave learners adrift in an ocean of information, struggling to solve ill-structured problems, with little clear idea of how to approach them, or how to recognise when they have made progress’ (p. 316).

Thus, discussions around OER (and the infrastructure which supports their storage and use) must take into account educational practice. According to Alevizou (2012), this is reflected in the fact that,

while the origin of the OER movement is located on the emphasis of entitlement (of access to, and adaptation of, free pedagogical material), the new wave of policy and advocacy initiatives focus on transparency enabled by the adoption of open educational practices. (p. 3)

On the ground, opening up academic practice is a work in progress that requires a combination of solutions, including cultural change as well as better technologies. The sharing and ‘reuse’ of published research output (assuming referencing conventions and any copyright restrictions are observed) is considered to be normal and desirable – in other words, it is hoped that other scholars will read and cite one's papers, but one's teaching materials seem to be subject to a different set of criteria (Davis et al. 2010; Tosato and Bodi 2011). For Alevizou (2012), there is a ‘lack of professional incentives and a culture (or expertise) in sharing and remixing openly’. Beyond these questions of culture and motivation, it is also the case that the reuse of teaching and learning resources is not straightforward when academics are unsure of the aim for which and context in which the resource was produced (Campbell et al. 2013; Koppi, Bogle, and Bogle 2005). Additionally, according to Petrides and Nguyen (2008), ‘users may lack the confidence, capacity, or willingness to contribute changes to OER’.

Assuming that adopting Open Educational Practices (OEP) represents a significant and desirable cultural shift for many academics, there is at the same time a fundamental requirement for appropriate tools to enable and support such practices. For the OECD (2007), ‘the rapidly growing number of learning materials and repositories makes the issue of how to find the resources that are most relevant and of best quality a pressing one’ (p. 100). Indeed, the success of the OER movement hinges on both the ability and possibility of educators to effectively create, share, discover and reuse quality resources. Repositories have come to be regarded as a key link in this chain (Currier et al. 2004; Downes 2003; McGreal 2011), as evidenced by the development of a wide array of OER repository initiatives worldwide in the decade or so since the term OER began to gain currency (Marcus-Quinn and Diggins 2013).

Repositories of OER (ROER) have been defined by McGreal (2011) as

digital databases that house learning content, applications and tools such as texts, papers, videos, audio recordings, multimedia applications and social networking tools. Through OER repositories, [resources] are rendered accessible to learners and instructors on the World Wide Web. (p. 1)

However, repositories matching this definition are quite a diverse and non-standard group (Armellini and Nie 2013; Geser 2007; McGreal 2011; Nikoi and Armellini 2012; Tuomi 2013). It seems therefore to be an opportune moment to build on the considerable work that the OER community has put into the development and use of such repositories, and consider how they might best be implemented or enhanced going forward.

In order to identify the most relevant literature for this study, 122 peer-reviewed journal articles, conference papers and books were reviewed and analysed. A subset which specifically discusses repositories was identified. Within this subset we identified mentions of good practices and elements lacking in the design of ROER. The literature reviewed was obtained from relevant articles in the field of OER, learning objects and repositories, from both Open Access and paywalled journals. The timeframe of the literature review is from 1996 to 2014 and the articles reviewed were in English, German and Spanish.

The objective of this study is therefore to contextualise and to describe a method for evaluation of ROER. Through this literature review we will define key themes in the ethos of ROER and distil a set of quality indicators (QI) for repositories, in order to ultimately derive an evaluation model.

The first World Congress on OER was held at UNESCO in Paris in June 2012, and this generated the first World Declaration on OER. The declaration is based on Article 26.1 of the Declaration of Human Rights (1949), which states that:

Everyone has the right to education. Education shall be free, at least in the elementary and fundamental stages. Elementary education shall be compulsory. Technical and professional education shall be made generally available and higher education shall be equally accessible to all on the basis of merit.

The concept of OER was first introduced in 2002 at a UNESCO conference in which participants identified a need for ‘provision of OER, enabled by information technology and communication, for consultation, use and adaptation of these by a user community for non-commercial purposes’ (UNESCO 2002). This was echoed by the OECD (2007) who also called for OER, defined as ‘digitised materials offered freely and openly for educators, students and self-learners to use and reuse for teaching, learning and research’ (p. 11). More recently UNESCO (2011) have provided a more nuanced description, understanding OER as

any educational resources (including curriculum maps, course materials, textbooks, streaming videos, multimedia applications, podcasts, and any other materials that have been designed for use in teaching and learning) that are openly available for use by educators and students, without an accompanying need to pay royalties or licence fees. (p. 5)

The philosophy of the OER movement has consisted from the outset of two fundamental concepts: free and open access to knowledge, and the capacity to adapt and reuse to create new materials (Abeywardena, Tham, and Raviraja 2012). For Smith and Casserly (2006),

At the heart of the movement toward Open Educational Resources is the simple and powerful idea that the world's knowledge is a public good and that technology in general and the Worldwide Web in particular provide an extraordinary opportunity for everyone to share, use, and reuse it. (p. 8)

The OER movement aims to facilitate the democratisation of knowledge by increasing the circulation and reuse of free, quality content in a variety of formats, without economic or legal barriers (Andrade et al. 2011; Downes 2007; Gourley and Lane 2009; McAndrew et al. 2012).

The most defining, common feature across the multitude of resources described by the umbrella term OER is therefore openness, which crucially indicates ‘openly licensed’ rather than simply ‘freely available’. As the Paris OER Declaration indicates, OER must

reside in the public domain or have been released under an open license, such as creative commons licence (CCL), that permits no-cost access, use, adaptation and redistribution by others with no or limited restrictions. Open licensing is built within the existing framework of intellectual property rights as defined by relevant international conventions and respects the authorship of the work. (UNESCO 2012)

The OECD (2007) have also cited open licensing as a legal driver of OER, with specific reference to ‘new legal means to create and distribute open tools and content through licensing schemes such as Creative Commons’ (p. 59). It seems that although OER could potentially be released under some other type of open license, in practice it is Creative Commons that is discussed in the literature. Released in 2002, the Creative Commons licensing (CCL) scheme aims to facilitate the transfer, sharing, use and adaptation of academic and creative content, while protecting the intellectual property rights of the authors. CCL is often regarded as at the heart of OER openness as it facilitates the adaptation and modification of the resources (Bissell 2009; Downes 2007; Jacobi and van der Woert 2013; Rolfe 2012; Willems and Bossu 2012).

The case for ROER is reflected in point I of the Paris OER declaration, which calls on states to

Facilitate finding, retrieving and sharing of OER. Encourage the development of user-friendly tools to locate and retrieve OER that are specific and relevant to particular needs. Adopt appropriate open standards to ensure interoperability and to facilitate the use of OER in diverse media. (p. 2)

The value of ROER is thus in relation to the need to open and democratise access to teaching resources, promoting the growth and dissemination of knowledge as a free resource, and removing the traditional copyright barrier, thus aiding in the transformation of the teacher's role from a content producer to ‘user-producer’ (OECD 2007; UNESCO 2011).

McGreal (2011) argues that ROER can be important resources for students and academics, dependent on the quality of the resources stored in the repositories; however, according to Butcher, Kanwar, and Uvalić-Trumbić (2011), in the OER environment, resource quality will ‘be assisted by the development of such repositories, which will provide at least first levels of quality assurance’ (p. 2). Housing resources within ROER is advantageous because the centralisation of resources facilitates retrieval, and widens access; furthermore, if a social layer is included, it encourages collaboration as users feel part of a community of practice (Browne et al. 2010; Jacobi and van der Woert 2012; Petrides and Nguyen 2008).

Drawing from our analysis of the literature, we would argue that the ethos underlying the creation of ROER can be said to comprise four key themes, which we refer to as Search, Share, Reuse, and Collaborate. The purpose of ROER is to support educators in searching for content, sharing their own resources, reusing and evaluating materials, and adapting materials made by or in collaboration with other members of the community.

The four themes can be understood in greater detail as follows:

Search: As Google tends to be the first reference point for many people, it can be considered a ‘living index and repository for enormous content’ (Atkins, Brown, and Hammond 2007). Although the internet has among its archives billions of documents and multimedia materials that can be found by using search engines, it is a more complex task to ensure that the materials and documentation discovered in such searches are appropriate to a specific educational field and context. For Wang and Hwang (2004), it is difficult for educators to build and maintain personal collections and is ‘very time consuming to locate and retrieve distributed learning materials’. For Rolfe (2012), searching for OER in repositories facilitates the non-commercial reuse of content with minimal restrictions.

Assuming these four themes together make up the ethos of OER repositories, it is also worth distinguishing between the technical and social characteristics of a repository, and how these are transformed by openness. For Tuomi (2006),

Whereas openness in the social domain is fundamentally motivated by the expected social benefits and by ethical considerations related to human freedoms, openness in the technical domain, in contrast, is motivated by technical interoperability and functionality. (p. 9)

Based upon our review of the literature, there are certain indicators of quality in the design and implementation of ROER, which, taken together, constitute a range of social and technical characteristics, and provide a support structure for the four themes. The 10 most recurrent QI identified in the literature are as follows:

Featured resources: Featuring (highlighting) certain resources held in a repository can benefit educators by exposing them to additional materials that can be either interesting, original or novel, or by showcasing resources indicated as high quality by peers (Hylén 2006; Pegler 2012; Petrides and Nguyen 2008).

Thus, any development of ROER must consider three main areas, which are the social and technical characteristics of the ROER, the specific QI, and the extent to which the repository supports the themes, as summarised in Table 1.

This literature review represents a high-level synthesis and analysis of the research related to OER and ROER to date, with the aim of understanding whether or to what extent this body of work is in agreement, and on which points, if any, we find voices diverging. We have understood OER as, first of all, a movement grounded in a particular philosophy, and secondly, as a term representing a collection of resources which have been ‘declared open’ via the application of the OER movement's specified practices and technologies. Happily for the OER movement, there are significant areas of commonality among the diverse group of perspectives represented here.

Notwithstanding the problematic nature of the term ‘open’ when used in wider contexts, there does seem to be a shared understanding of an underlying ethos of openness where it comes to OER. It is within the discussions of the varied interweaving routes to achieving the aims of the OER movement where different priorities and concerns emerge. The body of literature reviewed can thus be said to support our four themes, and the principle that repository design and implementation should reflect their importance. But authors have advanced a range of perspectives in relation to the more practical questions of how best to enable participation in, and support sustainability of, ROER projects.

ROER developers have aimed to address the need for what UNESCO (2012) calls ‘appropriate user-friendly tools’ for the exchange of OER by implementing a variety of socially- and technically- oriented features which assist in opening resources and practices. Ten such features have been conceptualised in this study as QI for ROER; the presence of an indicator shows that the repository has been designed to support a particular aspect of open practice.

The four themes and 10 QI can form the basis for a framework which is suitable for evaluating an individual ROER initiative, or conducting a macro-level analysis of a group of ROER. Evaluation of each repository should begin by checking for the presence or absence of the QI. Through an understanding of which QI are present in the repository, it is possible to determine what functionality is in place to support the four themes. As seen in Table 1, some QI are relevant to a single theme, while others act to support more than one theme. An additional consideration is that some QI are in a sense, ‘in competition’ with each other. For example, both keywords and standardised metadata are designed to facilitate users in searching and retrieving relevant content. Similarly, both peer review and user evaluation tools represent methods of assessing the quality of resources. Repository managers may view some QI as providing functionality, which overlaps with, or adds little to, what is present in the existing toolset, or view some QI, no matter how desirable, as too resource-intensive to set up or maintain.

For an organisation which is planning to either enhance an existing repository, or develop a new one, we suggest an additional level of analysis. We call this level ‘three-dimensional analysis’ because we consider that the desirability of implementing a feature must be considered in relation to the three dimensions priority , achievability and sustainability (see Figure 1).

Priority reflects the perceived importance and urgency of adding a particular feature, on the basis of the value added for the user community.

Each feature should be rated low or high for each dimension. A feature must be rated highly across the three dimensions in order to fall into the zone of quality enhancement (Figure 1). By conducting this level of analysis, it should become clear which features should be implemented.

From our point of view as learning technologists, embracing OEP represents a significant cultural shift for academics, as well as a technical toolset to be mastered. It is therefore our hope that current and future repository managers considering the next iterations of ROER platforms will be aided by our evaluation model in assessing which QI are going to best fit the needs of the communities they aim to serve. In this way, repositories can enhance the level of support they provide to academics who are attempting to adopt new, open ways of working.