Cost-effectiveness analysis of computer-based assessment

The need for more cost-effective and pedagogically acceptable combinations of teaching and learning methods to sustain increasing student numbers means that the use of innovative methods, using technology, is accelerating. There is an expectation that economies of scale might provide greater cost-effectiveness whilst also enhancing student learning. The difficulties and complexities of these expectations are considered in this paper, which explores the challenges faced by those wishing to evaluate the costeffectiveness of computer-based assessment (CBA). The paper outlines the outcomes of a survey which attempted to gather information about the costs and benefits of CBA.


Introduction
In response to the opportunities created through widening participation and within an environment operating under the pressures of limited resources, the higher education sector needs more cost-effective and pedagogically acceptable combinations of teaching and learning methods to sustain increasing student numbers.With traditional assessment and delivery methods, economic frugality forces reductions in the staff-student ratio and declining unit costs per student, potentially resulting in a reduction in the quality of the learning experience for students.The challenge is to find an alternative cost curve to allow for increasing numbers without a loss of quality and with greater efficiency (Dearing, 1997).Within this environment the use of innovative methods, utilizing developments in technology, is accelerating, in the expectation that economies of scale might provide greater cost-effectiveness whilst also enhancing student learning.
The uptake of computer-assisted teaching, learning and assessment remains somewhat limited and measures designed to establish the true development and adoption costs and related benefits (economic and educational) are equally restricted (Boucher, 1998).To date, little conclusive evidence exists for the cost-effectiveness of such approaches due to a lack of rigorous studies (Jung and Rha, 2000) and competing methodologies that make comparison difficult (Bakia, 2000).Lack of consensus on costing models, observable variation in cost estimates (Scott, 1997), problems in quantifying costs and benefits and variable quality (Boucher, 1998) present a major challenge.The sector needs to develop appropriate methodologies to identify and quantify the educational, institutional and social benefits claimed.

Challenges and issues
At an institutional level in the UK higher education sector, all investment in technology will be used for many purposes and costs may be met across many sources and absorbed under different budgets.Therefore, apportioning these costs specifically to computerassisted assessment (CAA) is likely to be extremely complex.Another problem in evaluating cost-effectiveness is the differing time scales between the initial investment in the technology and supporting infrastructure, and the realization of benefits.Even if costs are amortized over the expected life of the technology and duration of course materials produced, insufficient data currently exists on which to base any firm assumptions.
Political sensitivity to the 'cost of costing' (Bacsich, Ash, Boniwell and Kaplan, 1999) in the academic community combined with the destabilizing effect of the exposure of costing programmes (Oliver, Conole and Bonetti, 2001) may reveal the cost of everything and the value of nothing and provide inadequate arguments for not costing (Bacsich, Ash and Heginbotham, 2001).
The immaturity of computer-assisted teaching, learning and assessment methods and the problematic nature of evaluating cost-effectiveness means that there is not an accurate or reliable model that might predict the precise costs and benefits to be expected.A major study on the costs of networked learning (Bacsich et al., 1999) revealed many intangible, or 'hidden', costs.Recent developments arising from this project (Bacsich et al., 2001) resulted in guidelines and resources for costing courses (Ash, Heginbotham and Bacsich, 2001).The model uses an activity-based costing model (equally suitable to costing conventional and networked learning) applied at the level of the institution, faculty, course or module.However, this methodology is complex and extremely resource-intensive in terms of time scales, software and professional support costs.
Cost-benefit analyses of computerized teaching and learning have also proved problematic.The procedure is intended as a decision tool prior to undertaking the investment on the assumption that all costs and benefits are known with certainty, and can be quantified over the known lifetime of a project.Although cost-benefit analysis is based on sound economic principles, at this stage there is no suitable procedure for evaluating the cost-effectiveness of computer-assisted methods.Not only are most benefits difficult to quantify and the lifetime of a project sometimes difficult to ascertain, but also the majority of evaluations tend to be carried out retrospectively.The existence of intangibles in education requires modification, in the form of cost-effectiveness analysis.Although based on similar methodology, benefits are not valued in the same terms as costs and need not be evaluated in purely financial terms, but include statements of aspiration and social concerns, equality of opportunity, for example.
Benefits may be pedagogical, social or psychological.Some of those uncovered in the literature suggest that learning technologies may: • be motivating, time-efficient and effective in learning conceptually difficult topics (Atkins, 1993); • reduce failure through self-paced practice (Pollock, Whittington and Doughty, 2000;Scott, 1996); • enhance the quality of the learning experience; • promote a shift from passive to active learning; • improve transferable skills, such as independent study or IT knowledge; • lead to improved quality of teaching material; • allow increasingly flexible student access to learning materials; • allow learning experiences not otherwise ethically or economically practical (for example, simulations); • represent an important culture shift, rather than a one-off, short-term benefit (Oliver et aL, 2001); • develop greater institutional visibility or academic kudos (HEFCE, 1997).
Therefore, evaluation of cost-effectiveness is more important than a purely financial evaluation due to the potential benefits to be gained -cost is only one element in the equation (Bacsich et aL, 1999;Rumble, 2001).Developmental costs for computer-assisted methods are usually higher than for traditional methods (Hunt and Clarke, 1997;NCIHE, 1997) but, if innovative methods allow for increased numbers of participants, have a longer shelf life and improve educational quality (Cukier, 1997), cost-effectiveness can still be demonstrated.

Computer-based assessment (CBA)
The Computer-assisted Assessment Centre {http:llcaacentre.ac.uk) was funded by the Teaching and Learning Technology Programme (Phase 3) to provide advice, guidance and good practice in the implementation and evaluation of CBA in the UK higher education sector (Bull and McKenna, 2000).A review of published literature undertaken by the CAA Centre, regarding cost-effectiveness of CBA, revealed only one directly relevant article in the field (Pollock et aL, 2000), while the CAA Centre's annual report in 2000 highlighted some of the main issues as hidden costs, existing infrastructure and support costs (Bull, 2000).
At this stage in the development and implementation of computerized assessment, for reasons outlined above, a retrospective, cost-effectiveness analysis is appropriate.An institutional approach to CBA has been in place at the University of Luton for a number of years and, in common with the ITATL project (HEFCE, 1997), it is unlikely that institutions, such as the University of Luton, will be able to provide accurate figures for the cost of equipment, maintenance and replacement, or in respect of the supporting infrastructure and access costs.
Benefits arising from the implementation of CBA may be more important than costs suggested by responses to a national survey (McKenna and Bull, 1999).For example, the benefits to academic staff include automatic marking, statistical analysis of results and speed of feedback to students.Further, formative or self-assessment can provide benefits for students by allowing them to study at their own pace, as frequently as necessary and with the advantage of instant feedback.At an institutional level this potentially equates to improved access and retention.Other benefits which seem to be emerging include improved awareness of underlying pedagogy, increased efficiency through front-loading, spreading the academic workload, testing of the full breadth of course material and contextual congruence with other forms of online learning and student record and management systems.It is also important to consider opportunity costs in relation to the failure to use the resource in its best possible alternative use, particularly staff time.Time spent developing and delivering computer-assisted methods could have been spent on teaching, research, professional development, updating course material, or with students, for' example.
Certainly, an important driver for the adoption of CBA is the time saved as a result of automatic marking.Although, perhaps, not typical, Pollock et al (2000) suggest the time spent preparing and marking examinations has been, reduced to one-sixth of the original time.If a comparison is to be made with more traditional approaches the methodology adopted needs to be sufficiently flexible to incorporate all types of assessment, particularly as evaluation of the cost-effectiveness of traditional methods at the same level of transparency is rare (Bacsich et al, 1999).
The literature recognizes a number of significant challenges in the evaluation of costeffectiveness and benefits of computer-based teaching, learning and assessment.There are a range of issues which need to be evaluated in order to conduct a full cost-benefit analysis and which were beyond the scope and resourcing of this study.However it was seen as particularly valuable to explore specific issues relating to time saved as a result of automatic marking, hidden time costs and the level of staff experience in order to gain an understanding of the cost-effectiveness of CBA.
The CBA cost-effectiveness study This paper reports the results of a study designed to evaluate the cost-effectiveness of CBA.The focus of this study was only those costs that are directly attributable to CBA, over and above the provision of other technology within the institution, including CBA specific software costs.On the basis that academic staff time is the principal cost of teaching in higher education (Dearing, 1997), the major area of the study involved the time, and therefore costs, associated with using CBA in comparison with more traditional forms of assessment.It was critical to evaluate cost-effectiveness in terms of staff time including all stages of the process from planning and implementation through testing, post-testing, moderation, analysis and evaluation.
During the development and implementation stages, CBA is resource-intensive and costs will be greater at this stage than for subsequent maintenance.In addition to software costs, examples of hidden costs (Bacsich et al, 1999) specifically relevant to CBA were addressed including: time for question and test design, including the necessary staff development (trainers and trainees) and demand on technical support staff, which can increase overall costs.However, once questions are developed they can be saved for future use as long as the material stays relevant to the objectives of the course.For example, quality assurance guidelines at the University of Luton currently require a change in examination questions of at least 10 per cent per year although, in practice, it is often more.In the early stages of implementation this requires the design of new questions while, in time, the development of question banks may provide a sufficient number of questions to facilitate the necessary variation in examination content.Together with software costs, it therefore becomes possible to amortize these costs over their useful shelf life.Experience in CBA is also a critical factor and it was anticipated that the history of CBA at this institution would allow inclusion of the temporal effects of experience.Approximations of economies of scale may also have been possible.

Aims
Drawing on the findings of the literature review and acknowledging the complexity of the issues surrounding the costs and benefits of CBA, the study involved a number of aims.
The aims sought to address the key issues in the definable quantitative costs and also more qualitative benefits, such as staff experience and improved quality of teaching and learning experiences.The following aims were identified: 1. evaluation of the cost-effectiveness of implementing and maintaining CBA at a modular level; 2. evaluation of benefits arising from implementing and maintaining CBA at a modular level; 3. estimation of the effects of experience over time in relation to the cost-effectiveness CBA; 4. estimation of cost-per-student for CBA; 5. comparison of cost-effectiveness for conventional methods of assessment; 6. evaluation of evidence for improved retention rates where CBA is used.

Methodology Original methodology
The researchers' original intention was to explore cost-effectiveness within one institution, the University of Luton.Based on expertise developed within the CAA Centre over the previous three years and an extensive literature review, they identified that the most appropriate methodology to achieve the objectives of this evaluation would be focus groups and semi-structured interviews involving key members of staff.This would allow collection of historical data on implementation and maintenance of the CBA process through participation of staff with varying experience of using CBA.The aims were to evaluate the effects of module tenure and experience on the time, and therefore costs, involved as well as the benefits emerging.Data would be obtained for comparable modules using alternative assessment methods.Retention data would be extracted from the student records system, management information system or from field managers.
Problems with this original methodology emerged, however, due to resistance of staff within the two targeted faculties to participate in focus groups or interviews.Provision of historical retention data also proved problematic within the available time scales.It is likely that the problems encountered were, perhaps, due less to the nature of the study and more to unforeseen restructuring within the institution, including major changes to the CBA process at an institutional level -coincidental timings, which could not have been anticipated at the outset.

Revised methodology
Within the remaining faculties it was not possible to collect historical data on implementation and maintenance of CBA or to identify staff with the varying levels of CBA expertise necessary to fulfil the original aims of this study.In anticipation that staff at the institution would be more amenable to less time-consuming data collection methods, the only feasible alternative was to conduct a survey.Also, the use of a survey would provide cost-effective collection of as much data as possible within limited remaining resources.This would allow access to all staff within the institution, both users and nonusers of CBA, and, more specifically, to the CBA User Group, a sub-group of the University Teaching and Learning Committee.Given the changed nature of the study it was decided also that there would be merit in attempting to collect similar data from other academic institutions.A survey could effectively be administered to a network of CAA practitioners through the CAA Centre's national mailing list.
The survey was designed as far as practicable to obtain data that would closely match the original aims of this study.Survey questions requested details of position, faculty, academic experience, module level, student numbers, instruction time, methods of assessment and time needed for each stage of the assessment process.Additional questions were included for users of CBA with the aims of evaluating the effects of experience over time.The survey was piloted to all staff within the-CAA Centre and to several users and non-users of CAA within the institution.Following feedback on the survey, it was adapted prior to being made available more widely.
The survey (see Appendix ) was distributed online to the CBA User Group, comprising academics and key support personnel reporting to the Teaching and Learning Committee, all academic staff within the University of Luton, and approximately 700 members of the CAA Centre's national mailing list.Both the CBA User Group and members of the CAA Centre's national mailing list provided relevant and convenient samples to obtain valuable data in relation to computerized and other methods of assessment.At the institutional level, CBA has been in use at the University of Luton for a number of years with varying levels of uptake and expertise within the individual faculties.Distribution of the survey to all staff at the institution provided the potential to obtain data on both CBA and other assessment methods.The survey was made available online for a period of two weeks following email messages to the above sample.No personal identifiers were requested due to potential sensitivity of the type of data requested and, therefore, all responses remained anonymous.

Participants
Eighty participants responded, thirty of whom comprised current users of CBA, the survey successfully reaching both users and non-users of CBA.This represents a low response rate that is likely to be attributable to the limited availability of the survey.
Respondents frequently failed to answer all questions, as reflected in the analysis of the data.The breakdown of respondents is somewhat biased towards higher levels of seniority and experience, with three-quarters of respondents reporting more than ten years' academic experience.Respondents' positions are shown in Table 1.

Student numbers
Details of student numbers for each module were requested from all respondents.Reported frequencies are summarized in Figure 1.For all types of assessment, the majority of modules reported comprised student numbers of less than one hundred.It is, however, interesting to note that the relative use of CBA increases proportionately to higher student numbers.

Module levels
Reported frequencies for the academic levels of the modules are summarized in Table 3. Disproportionate CBA usage at lower levels was reported, as would be expected from the (mistaken) perception that CBA is only suited to the assessment of lower learning levels.

CBA
Questions numbered 8-23 in the survey were only relevant to current users of CBA (n=30), in order to obtain more detailed information about this method of assessment.The duration of CBA use and the level of experience of academic staff in using CBA clearly suggest that over half the participants have more than five years' experience with this method of assessment.
Almost two-thirds of respondents reported module weighting of 50 per cent or less for CBA, leaving just over one-third that reported weighting of between 51 and 100 per cent.The majority of respondents (24) reported using CBA for summative assessment closely followed by formative assessment (17), with lesser use for self (9) and diagnostic (8) assessment.
Preparation time CBA users were asked to respond to questions on the time involved for CBA preparation, per module and per annum.Based on usable data provided by twenty-eight of the thirty respondents using CBA, overall frequencies of preparation times reported in respect of CBA preparation are summarized in Figure 2.More than one-third of respondents reported preparation times of less than ten hours per module and per annum with almost two-thirds reporting preparation times of less than thirty hours.
Analysis of reported preparation times in relation to student numbers (Figure 3) necessarily results in low frequencies for each of the resulting fifteen categories.Reported preparation times are clearly not dependent upon student numbers, as would be expected for one of the main advantages of CBA.Analysis of reported preparation times based on learning levels (Figure 4), although based on very low numbers within each of the twelve categories, does start to form a pattern, For example, at level 0 both reports were for less than ten hours.At postgraduate level there are no reports of less than ten hours.However, between these extremes, no clear relationships emerge.Similarly, analysis of reported preparation times according to the weighting of CBA (Figure 5) suggests no clear relationships.It is likely that more complex relationships exist between these and other factors.In terms of CBA expertise, more than half the respondents reported more than five years' experience.Contrary to expectations, no clear relationship emerges between experience and preparation times.Also, the length of time that CBA has been used in modules is not clearly related to preparation times (Figure 6).
CBA users were also asked questions on the numbers of examinations used, new examinations prepared, questions per examination, new questions written and question bank use, the reported data for which is summarized in Table 4.
One aim of the study was to evaluate the effects of experience of using CBA over time and to include other factors, such as weighting, student numbers and learning level of the module.Questions in the survey to explore this were based on the assumption that the front-end loading of CBA means that it is likely to be more costly than other methods of assessment on implementation, but that with practice and experience in question design and test construction the process should become more cost-effective over time.It was further anticipated that the point at which this occurred could have been approximated from the data provided.Exploration of correlations between any number of these factors proved equally insignificant.

Times per student
Drawing on the data obtained it becomes possible to calculate approximations of time (and, therefore, cost) per student in respect of CBA preparation.As shown in Figure 7, based on the data provided by half of the respondents who use CBA, the preparation time involved is less than twenty minutes per student.Data from only five respondents indicates preparation time in excess of one hour per student.Reference to the source data reveals that two of these involved modules with low student numbers (0-50), and three of the modules reported on involved between 50 and 100 students.Two of the modules were at postgraduate level and two represented weighting of up to 75 per cent.CBA had been used for less than one year in respect of three of the modules.It should be noted that these calculations are based on the minimum number of students and, therefore, represent maximum preparation times based on reported data.Figure 8 shows these approximations in comparison to times per student in respect of the more traditional coursework essays.In so doing, the argument concerning the very different skills assessed between these assessment methods is recognized.

Time savings
Respondents were also asked direct questions (questions 18-23) on time savings and other benefits arising from the use of CBA.Of CBA users, 75 per cent reported noticeable net time savings (n=21), and 40 per cent reported making time savings within the first two years (n=12).A further 40 per cent attributed time savings to both experience and the use of a question bank (n=12).

Other methods of assessment
Question 24 in the survey requested estimated times for examination preparation in addition to or, other than, CBA and questions 25-9 requested an estimation of times for marking, invigilation, moderation, evaluation/analysis of results and staff development for all methods of assessment.The purpose of this was to obtain comparable data for all stages of the assessment process across all assessment methods.The extent of responses to the majority of these questions is evident from the range of preparation and marking times shown in Table 5.
Table 6 provides a summary of total times reported for all stages of the process in respect of all assessment methods reported.This data is based on the mean or median as deemed most appropriate for each set of data arising from responses.Extreme high and low values results in a wide range for the majority of data sets.Therefore, the median, where equal numbers of cases fall above and below the value used, is more representative.In many instances, the median and the mode produced identical values.Almost all cases resulted in the mean producing a higher value than the mode.Therefore, the use of the median presents a fair and realistic evaluation.

Discussion
Cursory interpretation of the results in Table 6 suggests that, in terms of time and cost, CBA is placed towards the higher end of the range of reported assessment methods, second only to coursework essays and, with the caveat that hardware and software costs have been intentionally excluded on the basis that hardware is widely used for many other purposes in higher education, the costs of which, together with software costs, are likely to be distributed across diverse budgets.At the modular level, costing of hardware and software becomes even more difficult to quantify with any accuracy.Comparison between different assessment methods is interesting and informative, although caution with interpretation is advisable as reduction of data in this way necessarily results in a loss of richness and complexity.
Of greater direct relevance are the data on preparation times in respect of CBA use (Figure 2).Over one-third of respondents (n=l 1) reported preparation times of less than ten hours per module, per annum.Almost two-thirds of respondents (n=19) reported preparation times of less than thirty hours per module, per annum.This needs to be considered together with the absence of marking time for CBA.Preparation and marking times for CBA are not dependent on student numbers and, therefore, are suitable for modules that attract large student numbers.This is supported by the analysis of reported preparation times by student numbers (Figure 3) and the relative proportional increased use of CBA for higher student numbers (Figure 1).Large variations were apparent between times reported for conventional assessment methods (Table 5).Variations in student numbers are reflected in the marking, moderation and, possibly, analysis/evaluation stages of the assessment process but these are not relevant at the preparation, invigilation or staff development stages.The extent of the variations reported questions the validity of the reported data.For example, even for low student numbers it is unrealistic that essays can be marked in an hour or two.Some participants reported on the basis of 'time per student' and it is possible that this applies to many of the reports.For these reasons, the data reported in Table 5 has been based upon the median being representative of the data.
For conventional methods of assessment the only meaningful unit of analysis is cost-perstudent.CBA is much less subject to variation based on student numbers and the mean becomes far more meaningful but calculation of cost-per-student is more complex.This clearly demonstrates the major problem in comparing CBA with conventional assessment methods, whilst at the same time highlighting one of the major advantages of CBA for high student numbers.Approximations of time-per-student based on reported data (Figure 7) suggests maximum preparation times for CBA for almost half the respondents of less than twenty minutes per student (n=14); only five respondents fall into categories in excess of one hour per student.This partially addresses aims 4 and 5 of this study.Given the generally low preparation time and the absence of marking time, CBA compares favourably with more traditional assessment methods.-Detailed information was requested from CBA users in anticipation that the resulting data would address aims 1 and 3 of this study.It was expected that preparation time for CBA would decrease with experience after allowing for variations in the number of examinations and number of questions per examination and giving consideration to other factors such as the learning level of the module, weighting and question bank use.This should have allowed estimations of preparation times for the assessment processes over time for comparison with conventional methods of assessment.However, analysis of participants' responses quantifying all stages of the CBA process (Table 2) suggests the relationship between these factors is far from direct.No relationships are suggested based on experience and no clear relationships are apparent between any of the variables relevant to the aims of this study.Any relationship that may exist is likely to involve complex interactions between any number, or all variables.A slight indication of the greater time needed per question at higher learning levels is suggested, although the low numbers on which this is based results in this being far from conclusive.Inconsistencies in responses are evident.For example, some participants report no new questions or examinations per year, others report new questions but not new examinations, whilst still others report a greater number of new examinations than examinations actually used per year.
These indications are far from conclusive and cannot be used as the basis for the level of complexity necessary to fulfil aims 1 and 3 of this study.The data resulting from this survey has, therefore, not been successful in achieving the aims of evaluating the effects of experience nor of being able to amortize the costs of implementing CBA over time.This is partly due to a bias towards experience of respondents, 75 per cent of whom reported academic experience in excess of ten years, and with 57 per cent of CBA users reporting experience in excess of five years.On reflection of the survey design, this may also have been due to inadequate wording of some of the survey questions to enable differentiation between preparation times for examination preparation, and new question design and examination preparation and insufficient differentiation between the implementation and maintenance stages of the CBA process.What this study has been successful in doing is highlighting complexities in evaluating the cost-effectiveness of CBA in comparison to conventional methods of assessment, in addition to those that were evident from the start, from the extensive literature review (see Bakia, 2000;Boucher, 1998;Jung and Rha, 2000;Scott, 1997) and expertise within the CAA Centre.
However, costs only represent one part of the equation for the cost-effectiveness of CBA, the other side being benefits arising (Aim 2 of the study.).Responses to direct questions on time savings and other benefits arising from the use of CBA (Q18-23) appeared very positive.Of the thirty CBA users who completed the survey, twenty-one (70 per cent) reported noticeable net time savings, and twelve (40 per cent) of these reported time savings having become apparent within two years of implementation, the same number attributing these savings to both experience and the use of a question bank.Further potential benefits not specifically addressed in this study are that the front-end loading of CBA spreads the workload of assessment methods, enhancement of the student learning experience, improved access through diagnostic assessment and improved retention rates through formative assessment, all of which potentially contribute to cost-effectiveness at an institutional level.

Recommendations
This study has not been successful in achieving all of its original aims.Importantly, the need emerges, congruent with the original methodology, for stratified sampling that provides a balanced range of experiences (academic, CBA and module tenure) in order to estimate the effects of experience over time in the CBA process.A longitudinal approach across a number of case studies may be more successful in achieving these aims, subject to available resources as this methodology would be extremely resource-intensive.Sampling for such studies presents a real problem.For example, whilst this study may be considered to have used convenience sampling, this was making use of the many contacts of the CAA Centre, all of whom have expertise in the use of CBA.The resistance of academic staff, which resulted in a forced change to the methodology as well as the limited resources available, largely hindered the study.Whether staff resistance was due to coincidental timing with other major changes within the institution, or reluctance to explore the costing issues cannot be known with certainty (see Bacsich et al, 1999;Oliver et ah, 2001).
On the basis of the study as it was conducted, the need to differentiate more clearly between the implementation and maintenance processes for CBA emerges.In particular, distinguishing between examination preparation, new question design and incorporation and preparation of new examinations is important.Greater clarity is also needed in respect of data requested to evaluate more traditional assessment methods.
The relative immaturity of computerized teaching, learning and assessment methods, combined with the institutionalization of traditional methods and resistance to change all remain problematic in any attempt to evaluate the cost-effectiveness of CBA at this point in time.For these reasons, it becomes extremely difficult to obtain hard data that conclusively demonstrate the cost-effectiveness of CBA.It also becomes necessary to consider whether the ultimate goal is to demonstrate that CBA is efficient in terms of utilization of available resources, or whether it is effective in contributing to the goals of quality education.Above all, the success of CBA is dependent upon whether it is pedagogically acceptable.This necessitates extensive research that demonstrates, not only that CBA enhances student learning, but, specifically, how it does so.Current developments towards the assessment of all learning levels, possibly through a more holistic approach, combined with text analysis, for example, may provide the key to the future of effectiveness and cost-effectiveness of CBA.To date, the uptake of CBA appears to have been adopted in a largely piecemeal fashion that also suggests a need for the management of change and of the diffusion of innovation process through staff awareness and development.
Figure I: Reported numbers of students

Figure 3 :
Figure 3: CBA: reported times by student numbers

Figure 6 :
Figure 6: Number of years CBA used on this module

Figure 8 :
Figure 7: CBA: approximation of time per student What is the level of this module?

Table I :
Academic positions of survey respondentsA breakdown of the frequencies of use for all assessment methods reported by respondents is given in Table2.Frequencies reported exceed the total number of respondents as the majority used more than one assessment method for each module.

Table 2 :
Reported frequencies of assessment methods

Table 3 .
Module levels for all assessment methods

Table 5 :
Preparation and marking times (hours per annum) for assessment methods

Table 6 :
Average* total times for assessment methods (hours per annum at a modular level) Approximately how long (to the nearest hour) does it take to design the questions and create the tests for CBA in this module?In your experience of CBA do you notice any net time savings overall (i.e.extra time for question design and test construction compared to time saved marking)?