The distributed development of quality courses for a virtual university

Although virtual universities are widely touted as a way of competing globally in a postcompulsory educational market, systems for implementing them are currently underdeveloped The central thesis in this paper is that in order to ensure the quality of the development and delivery of course materials for a virtual university, a collaborative and iterative approach to authoring is required. In this paper, the development of such a process will be discussed The paper is based on experience with a project whose aims included the provision of Masters-level courses in supply chain management for learners in full-time employment in small and medium-sized enterprises (SMEs). The materials developed through the project were required to be academically rigorous, vocationally relevant, and situated in the context of the learners. The project relied on distributed development, with authors based at institutions across the UK and in Europe. This paper focuses on the creation and evolution of the development processes adopted by the project, illustrating these with examples of good and bad practice. Based on these, tensions between quality and resourcing are identified, and implications will be drawn for other teams working on the development of online courses.


Introduction
The perceived need for global competition in higher education has led to the proliferation of projects and initiatives focused on the development of online education (see, for example, Smith and Oliver, 2000).Much resource and effort is being invested in this area, but there has been relatively little research into effective development and quality assurance models that can support the creation of online courses.Still less has been carried out that considers the particular developmental challenges posed by distributed teams of authors.
In this paper, these issues will be discussed in the context of the Training for Innovation in Supply Chain Management (TISCAM) project.After describing the project, the development process will be outlined.Examples of materials produced will be considered to illustrate the strengths and weaknesses of the process, and revisions to the model will be discussed.The paper will conclude by highlighting a series of challenges for development work of this type.

Background
The project that provides the context for this work was an ADAPT-funded initiative that sought to offer Masters-level courses in supply chain management to learners in the workplace.The rationale for this was to provide learners with access to high-quality materials and leading academic experts, irrespective of their geographical location.
This model involves the central co-ordination of the authoring and delivery of courses, as illustrated in Figure 1.

Regional delivery centre
Learner IT" Learner Within this structure, the core team is sub-divided into groups with distinct areas of expertise.The groups relevant to the development process are illustrated in Figure 2. As these figures illustrate, development was organized around an elaborate division of labour that was intended to cover each step in the educational process from course conception through to assessment and evaluation.Initially, twelve modules were developed, either by individual authors or teams led by a nominated co-ordinator who liaised with the development teams; Each module constitutes 100 hours of study, and is broken down into 10 units of approximately 10 hours of study time each.The notional study time includes working through the materials, carrying out and completing activities and assessments, building up a portfolio, engaging in online discussion groups and reading case studies and related materials.
Learners are supported by a range of different specialists, each with their own expertise, as illustrated in Figure 3.It is important to note that, with this model, it is possible to have materials written at one institution, supported online by someone at a second institution and delivered to learners registered at a third.Clearly, the complexity of the development and delivery processes raises many issues for developing, delivering and quality-assuring the learning process.

Models of quality assurance
The notion of quality in higher education is a contested one; at least five interpretations have been identified in the research literature (Green, 1994): • quality as excellence; • conformance to specifications or standards; • fitness for purpose; • effectiveness in achieving institutional goals; • meeting customers' needs.
As Green notes, these interpretations are neither exclusive nor exhaustive.In practice, several of these interpretations may be held alongside each other, and the contradictions between different interpretations may only become apparent as the work progresses.The original model for this project was based on the notion of excellence -it focused on the provision of cutting-edge knowledge, resources and materials, and on access to leading experts in the field.
O'Shea, Bearman and Downes (1996) have argued that quality is particularly important and unusually difficult for open and distance teaching.Because such approaches fall well outside traditional models of education, 'the biggest obstacle that assessors have to overcome in assessing an open and distance teaching institution is knowing and recognising the teaching to be assessed ' (ibid.: 198).Consequently, they have distinguished between the quality of academic content (including learner support), the quality of tuition and the learning experience and finally the quality of assessment.This paper will focus on the first of these three, although reference to the other two areas will also be made.
Development models for distance learning Peters (1998) has described distance education as the most industrialized form of learning and teaching.Specifically, he contrasts the 'craft' model of lecturers designing.and developing traditional courses themselves with the division of labour common amongst distance initiatives, in which authoring, design, editing, development (particularly technical development) and teaching may all be undertaken by distinct individuals.
This industrialized model was adopted for this project, as was reflected in its internal structure.Separate contracts were issued for authors and for tutors, even where these went to the same individual; decisions about the courses to develop and the authors to contract were taken by the central management team.In addition, central teams were established to work with authors providing pedagogic and technical.support,covering the design and development aspects of the project respectively.

The project's quality system
Given the conceptual and organizational complexity of the project, it was inevitable that the quality systems would also be complex.In keeping with the project's strategy of organizing itself around educational processes, quality systems were devised for each of the main areas of work.The necessity of this became apparent when the division of labour, described in Figures 1 and 2, led to confusion rather than clarity for many project partners -particularly those who held contracts for more than one area of work.
Guidelines were drawn up specifying the responsibilities of authors, tutors, in-company mentors, regional facilitators and even of the learners.As well as providing an agreed description of each role, these specifications allowed a minimal level of quality to be assured (in terms of conformance to standards).A selection of these will be outlined below, with reference made to the definitions of quality that most closely describe their purpose, before summarizing the process with an overview of the quality system for authoring materials.

Support and guidance for authors
In the initial stages of development, a series of face-to-face meetings was held for members of the development teams.These consisted of preliminary briefing meetings and workshops on writing for online delivery.The latter drew on Laurillard's conversational framework (1993) and incorporated the use of the Media Adviser software (Oliver and Conole, 2000) to improve the pedagogic quality of module plans.Written guidelines for The context of authoring forTISCAM : The submission of materials ' -....:: The unit template sheet -a planning document for authors to specify the unft aims, 1 objectives, .learning outcomes and assessment methods, and a map of the content against the learning. .outcomes and methods of assessment.
authors were produced arid circulated early oh in the project.The main focus of these guidelines was to outline the elements the materials should contain, and give guidance on the organization arid, submission of drafts (see Table I).In this way, they' provided threshold standards against which submissions could be judged.

P e e r r e v i e w '••
Iri addition to this process of review by specialists, a peer review element was incorporated into the quality system.This involved providing access to draft materials, allowing authors to Work through each others' materials arid discuss these online.By exposing materials to inspection by other experts hr the field, this process ensured the accuracy and currency of information and cases.It also provided a mechariisrn for sharing examples of best practice, promoting quality as excellence and extending the model of conformance to standards represented by the authoring guidelines....... <.o -. •.• : .-•. .; r Academic review ; •''•'.•' ' ' ' ' i: '' '''"'',' ' ' " :: '; : " ' ' "' " "'""' : " : ' Ensuring the academic quality of the courses was a primary cbricern for the project; One elerrierit of the authoring process involved mapping out the cbriterit of units in relation to other materials' under development.Approved develdprrients were subsequently checked by an academic board with responsibility for issues such as the consistency of format and of level across the materials.Once' again, this is best characterized as a process of conformance to standards.. ; .: .• • r , , , .; ., Delivering institutions were also asked to seek internal validation of the courses they were offering.However, giventhat materials were being developed throughout the project, and that validation pfbcesses often required six months or more, the project partners understood that this stage of the quality process might not be completed during the tw6-year funded period of the project.' A model of the authoring process .
•, .•:•, .:,: .: ; .; The division of labour within TISCAM, combined with the production of standards and the use of either specialist or peer review at each stage of development, made it possible to specify a clear, iterative system for ensuring the quality of materials.This is illustrated in Figure4.L ; .
•::!. ; :v •nu-; ::-:.; : >. ,--; i,;i - It was clear from the outset that this model could have significant implications for.the authors'^ workloads as a consequence of a series of requests for revision.However, the expectation was that authors would be able to create materials that, once signed off after pedagogic review by the learner support team, would quickly progress through the other phases of development.

Applying theTISCAM development model
Whilst every effort was taken to ensure that all modules exceeded agreed minimum levels of quality, many of the initial drafts required considerable revision.Authors generally grasped the guidelines that were provided, but -for reasons not always apparent to the.reviewersneglected some aspects of this advice.Moreover, they demonstrated a wide diversity of styles from one module to the next, and in some cases were even inconsistent in their approach between units within the same module.To illustrate the variety of materials that were submitted, and the issues that arose from these, three examples will briefly be outlined.

Example A
In this first example, the materials reflected a 'textbook' approach to teaching.The content was academically sound and was well written, but as learning material for online delivery it was poor.It was heavily text-based, requiring considerable on-screen reading, and there were few opportunities for learner interaction or engagement with the materials.There were only three activities, each of which required learners to read additional material from a recommended textbook and then produce written reports.No feedback or model answers were provided for these activities, and no self-assessment activities were included for learners to check their own understanding of the materials.. Furthermore, the guidelines for learner support had been ignored -there was an over-reliance on.tutorsupport and feedback, and no use of the in-company mentor for discussion and work-based application.
This case exemplifies an approach to writing that views learning as transmission of information.In the first batch of submitted materials, this approach was all too common; however, as authors developed a more sophisticated understanding of online learning, they were able to develop more engaging, activity-and discussion-based materials instead.
The feedback from the pedagogic review provided a number of suggestions on how this unit could be developed for more effective online delivery.The amount of text-based content was substantially reduced, and was divided into a larger number of shorter sections that would be more manageable for learners to work through.Activities were redesigned to minimize the amount of additional reading and report-writing, greater variety was introduced to increase learner motivation, and self-assessment activities with online feedback were also added.The author revised the material, taking account of many of these suggestions, which resulted in the production of a much more interesting and effective online learning unit.

Example B
In this example the materials were well written and structured, with learner reviews at the beginning and end of the unit, and clear introductions to the content being covered.In marked contrast to the original material submitted in Example A, Example B contained regular and interesting activities, with a good balance of self-assessment and work-based examples.However, there was little opportunity to share views or experiences with fellow learners, and model answers and written feedback was scarce.Although this outline reflected constructivist values and approaches to learning, the lack of guidance meant that it was almost too open, providing no frame of reference for learners prior to the summative assessment.
Examples of good practice from another module were made available to the author, resulting in the revised materials resembling a 'guided discovery' approach to learning.Model answers and tutors' comments were incorporated into the design to help students gain a sense of their ability and progression.

Example C
In this final example, the author submitted a draft of one of his units at a very early stage in the development process.This draft adopted an action-based learning approach, in which learning was developed as activities were carried out and case studies were read and analysed.This draft influenced the technical team in its framework for the materials delivery system.However, in order to ensure that the system was navigable and that learners could manage their work, the final system imposed a specific structure on materials.Because the author had already drafted several units by this stage, he was required to rework a lot of his material.As a result, his units became more rigid and less interactive.
This particular example is important in that it highlights a tension between pedagogic ideals and the pragmatics of the research project.Whilst there was little wrong with the original materials, the necessity of providing learners with consistency from unit to unit meant that the design of materials produced by authors converged.In many, cases, this improved the pedagogic approach that had been taken, but in this case, it set necessary limits that restricted what could be achieved.

Discussion
The pedagogic review process proved crucial in reinforcing the strengths within the materials, and identifying and providing suggestions to overcome weaknesses.Strengths became incorporated into good practice guidelines, and were shared amongst authors to help develop the quality of their materials and their authoring skills.This demonstrated the importance of the team-based approach in adding value to the development process and in promoting shared expertise and experiences.

Common issues
A range of common issues emerged from the materials submitted (Table 2).Most of the authors were subject experts; they were not necessarily writers or experienced distance or online courseware designers.Consequently, many required considerable help and guidance from the development teams on their early drafts.One particular problem that arose from authors' differing approaches was that there were many inconsistencies from one module to another, which could cause confusion and a lack of cohesion for learners.
The need to organize material into manageable amounts for short online study sessions.
The adoption of a clear and informal writing style, and provision of adequate guidance for learners to understand the material and what was required of them.For example, the inclusion of the learning objectives, unit introductions, and providing navigational cues to guide learners to relevant material, such as an activity or a case study.The lack of ongoing activities for learner engagement, learning application and reinforcement Interestingly, many authors found it easier to design work-based activities for learning application than they did self-assessment activities.
Activities often lacked feedback for learners, such as examples or model answers.
The need for authors to be aware of the potential, and also the limitations, of the online delivery medium.For example, thinking about how large and complex graphics can be presented on the limited size of the computer screen; the effects of including several animated or interactive graphics within one page on download times.Raising awareness of the uses and abuses of online peer discussion.Authors were encouraged to make occasional suggestions for online discussion to help avoid learner isolation, but as learners did not study in cohorts, involvement in discussions should not prohibit their progression through, and completion of, units.Encouragement to some authors to include related case study materials to enrich their units.It became apparent from early drafts that authors were not adequately considering an assessment methodology for their module as a whole, and suggesting unrealistic levels of support from tutors and in-company mentors on a unit-by-unit basis.As many authors submitted units for review out of sequence, it was difficult for reviewers to get a clear picture of the module assessment framework and support requirements, or indeed if authors were even considering these.
Authors who were open to develop their skills and expertise through their involvement with the project took account of the feedback given during the review process, and learnt from the emerging good practice of other authors.The quality and effectiveness of their materials improved dramatically over the development period, as illustrated by Examples A and B, above.
Importantly, the editorial role within the development process had been underestimated.It transpired that some more experienced authors, who published regularly, assumed that an editorial process was in place and submitted drafts without thoroughly checking them.This led to additional work for the pedagogic reviewers, who took on some of the work of an editorial team in addition to their other duties.Moreover, some authors were simply less able writers than others, and the pedagogic reviewers identified a lot of problems that could have been addressed by a competent editor.Later in the project, the technical team recruited editorial staff to perform this function. •'.."'..
In general, there was an initial underestimation of the timescale of the development and review process.In fact, this turned:out to require far longer than the project actually permitted.This was caused by the learning curve that authors faced, and the need for more revisions to drafts being required than had originally been anticipated.

Revisions to theTISCAM model
In light of the examples above, the development model that emerged during TISCAM differed in several important respects from that shown in Figure 3. Firstly, the expertise of the learner support team providing the pedagogic review also extended to technical matters, and as noted, editorial concerns were also addressed during this step.This allowed one review to cover several aspects of the development process, shortening the total time required.
The academic review, meanwhile, concentrated on the process of co-ordinating submitted material across the project in terms of the topics covered and the potential construction of pathways through the material.Making this distinction between the strategic (academic) and tactical (pedagogic and technical) elements of the review process facilitated the development work considerably, allowing the academic review and the pedagogic review to take place in parallel.
Finally, the idea of an online peer review suffered from a lack of dedicated time.Authors were simply under too much pressure to produce their own materials to be able to spend long periods critiquing those produced by others.Instead, regular authors' meetings were scheduled.By booking time aside, it became possible to showcase particular sets of material that illustrated good, bad or controversial features.This provided a 'low-tech' forum for learning and discussion to support the development process, but importantly, it depended on the pedagogic reviewers (who had an overview of submitted materials) identifying examples of good practice and sharing these with authors who needed help with certain aspects of their work, effectively acting as tailored examples that supported the development of authors' skills.Importantly, within this aspect of the project (the only one to focus on excellence rather than standards), it was impossible to separate the quality systems from a mutual process of training and education.
The revised development model is illustrated in Figure 5, which illustrates how a parallel review process and amalgamation of review tasks minimized the amount of author revision's required.This had the effect of reducing the overall development timescale.
Finally, it is worth noting that additional adjustments were required when the prototype for the online system became available.The authors' guidelines were revised to include a Author Academic Review Pedagogic and technical review

7.
8.  specification for the structure of the materials, to ensure they adhered to the system requirements.The unit template was also revised to include more detail about the assessment methods and support requirements for the unit, and feedback forms for the pedagogic and academic reviewers were circulated that set out the final review criteria (Table 3).
As before, these criteria are best viewed as a way of specifying the minimum standards expected.Authors were aware of what the reviewers were looking for, and for reviewers, the criteria became part of a feedback sheet, which could be used as a checklist as the review was carried out.This made the review process quicker, minimizing the need for lengthy and repetitive feedback.

Conclusions
Quality systems for authoring online materials inevitably reflect the complexity of the development process.Moreover, whilst the project set out to provide excellence, much of the system that was put in place relied on the notion of ensuring standards (although excellence was encouraged through the process of peer review).In addition, the initial industrial model of production was revised on pragmatic grounds.All the elements were retained in order to ensure that no areas were omitted from the quality framework.However, in order to speed up the development process, one team undertook the majority of the quality checks whilst a second, working in parallel, attended to the co-ordination of subject coverage and academic standards.
Several important conclusions can be drawn from this experience.Firstly, the model that was developed was effective, in that it allowed marked improvements to be made in terms of the quality of materials.Given its success in this role, it may well prove to be a useful point of reference for other initiatives that involve the development of online courses.
Next, the implementation of a quality system is resource-intensive (particularly in terms of staff time) and must be achievable within the timescale allowed for the project.It should be adequately costed and planned for at the outset of development projects if is to be achievable.In this particular case, the project did not include enough time for sufficient iterations of review and redevelopment of materials; this limited the extent of revision that was possible.The implication of this was that the project had to choose between compromising on the volume or the quality of the materials produced.
In addition, the Learner Support Team came to realize that subject experts may not have any prior experience of authoring, particularly in the context of distance or online learning and may need training, as well as guidance and support.
It also became clear that there is a tension between the specialization permitted by the division of labour model and the practicalities of development.In this case, a compromise position lying somewhere between Peter's (1998) 'craft' and 'industry' models emerged.This involved authors creating draft materials and technical experts implementing it, but one multi-skilled team taking on a range of roles in order to broker information and act as a channel for communication between these groups.Whilst the authors concentrated on subject expertise and the implementation team on technical expertise, the learner support team was required to have expertise in instructional design, editing, multimedia design and a range of communication and management skills.This model may be relatively common for those working as learning technologists in support of academics developing online courses.Some compromises clearly need to be made.The development teams had to reconcile the desire for consistency against a range of authoring styles.Although the online system has imposed a consistent structure on materials, the authors retained considerable autonomy over what they would include (subject to the quality checks outlined above).Because the materials were produced as part of a research project, such variation was actively supported, since it will allow the evaluation team to monitor effectiveness of different approaches as materials are piloted with learners through the delivery centres.
Perhaps the most important conclusion to draw, however, is that achieving quality in the development of courses for a virtual university is a demanding, costly and time-consuming process, which is easily underestimated when planning a project.This is particularly true when aiming at more abstract interpretations of quality, such as quality as excellence as opposed to conformance to standards.The implication of this is that managers of such a development process will need to take hard decisions about how much quality -and of what type -their project can afford.

Figure I :
Figure I: The distributed development and delivery model for the project

Figure
Figure 2: The structure of the core team

Table 3 :
The final criteria for checking submitted materials