A Model for Evaluating the Institutional Costs and Benefits of ICT Initiatives in Teaching and Learning in Higher Education

Strathprints is designed to allow users to access the research output of the University of Strathclyde. Copyright © and Moral Rights for the papers on this site are retained by the individual authors and/or other copyright owners. Users may download and/or print one copy of any article(s) in Strathprints to facilitate their private study or for non-commercial research. You may not engage in further distribution of the material or use it for any profitmaking activities or any commercial gain. You may freely distribute the url (http://eprints.cdlr.strath.ac.uk) of the Strathprints website.


Introduction
Higher education (HE) institutions are currently investing heavily in information and communications technology (ICT) to support teaching and learning.As a result, senior managers, policy-makers and the funding councils are becoming increasingly concerned that more systematic information be made available about the costs and benefits of these activities so as to inform investment decisions (HEFCE, 1998).
In recent years, there has been growing interest in developing models for identifying and analysing the full costs associated with ICT-supported teaching and learning activities (for a review, see Ash and Bacsich, 2002).Over this same period, there has also been considerable research into ways of evaluating the benefits of ICT for teaching and learning.However, with perhaps a few exceptions (for example, Collis and Moonen, 2001), there is still almost no published literature linking costs and benefits together within a single framework or model that could be used to compare the relative merits of different investment decisions.Most cost-benefit studies are limited in scope usually involving comparisons of a course delivered at a distance with the same course delivered face-to-face; and there are few studies that deal with situations where computer-supported and face-to-face learning methods are integrated rather than separate.
There are two interrelated reasons why research on costs and benefits in HE have progressed separately.Firstly, each area (costs and benefits) has its own distinct methodologies.The rationale for cost evaluation is based on systematic data collection, quantitative analysis and measurable outputs (for example, monetary, staff time) whereas benefits evaluation relies on indirect and subjective data collection that leads to qualitative analysis (for example, reports of learning gains rather than actual measures).These differences make it difficult to construct a single framework or model that might relate the two areas together.Secondly, within institutions, those tasked with cost analysis normally have little expertise in benefits evaluation and those with benefits expertise have little costing experience, and these groups rarely work together.This paper describes the outcome of a UK project that was aimed at developing a model and methodology to evaluate the costs and benefits of the use of ICT in teaching and learning in HE.This work was funded by the Joint Information Systems Committee (JISC) and the cost-benefit model that is proposed is called the INSIGHT model.
The INSIGHT cost-benefit model was conceptualized as a decision-support tool for senior managers in HE tasked with making investment decisions about initiatives in technologysupported teaching.The model comprises a general framework for cost-benefit evaluation and an associated methodology.A basic assumption of the model is that decision-makers rely as much on qualitative information (for example, about impact or benefits) as on quantitative cost information to inform their reasoning (Oliver, Conole and Bonetti, 1999).The model is flexible in that the cost and benefit categories utilized can be adapted to meet the needs of different stakeholders and institutional contexts (see Patton, 1997 on utilization-focused evaluation).It is also assumed that decision-makers would be actively involved in discussions about strategic priorities and about the output values.It is considered unlikely that the output of any cost-benefit evaluation would ever be accepted unquestioningly.Therefore, undertaking the 'journey' through the analysis that the INSIGHT model proposes is considered as valuable as getting to the final 'destination' (the output ratio of a cost-benefit evaluation).

Research on cost and benefits of ICT-supported learning
The first step in developing a cost-benefit model is to identify and apportion all costs associated with a specific ICT-supported teaching and learning activity and to identify the benefits that derive from that activity.There are, however, barriers to the achievement of these conditions.
Existing financial systems rarely contain the required cost information Moonen (1997) has identified some reasons why costs information is difficult to access and quantify in HE.One problem is the lack of systematic collection of cost data.Institutional finance systems are usually organized in relation to the dual requirement of recording financial transactions and providing budgetary control.Unlike large commercial organizations, it is rare to find sophisticated costing systems geared to providing routine information on the costs of specific activities such as courses or classes (Bacsich, Ash and Heginbotham, 2001).Moreover, few institutions have systems capable of apportioning central costs, such as the cost of a network, to local activities at academic department level.
Other reasons given by Moonen for the inability to cost ICT initiatives include disagreement amongst researchers about which costs should be taken into account, the instability and evolving nature of costs and the confidentiality of some costing information.However, cultural barriers are probably the most important impediment to systematic costing practices within HE (Bacsich et al., 2001;Cropper and Cook, 2000).Systematic costing, as applied in commercial models, would require that institutions adopt procedures whereby staff keep records of time spent on different activities (for example, teaching, research, administration) and this is counter to the culture of HE.Nonetheless, recent developments in HE such as the Transparency Review are challenging this culture (Joint Costing and Pricing Steering Group, 1999).
Evaluation of benefits tends to be too narrowly defined The benefits of ICT innovations in HE are normally evaluated from a fairly narrow pedagogical perspective (Gunn, 1997).The main concern is usually with the learning benefits deriving from an ICT innovation although these measures of learning gain are almost always indirect (for example, confidence logs where students rate their grasp of underlying principles) rather than direct (for example, examination marks).Few studies evaluate the benefits from the wider institutional perspective; for example, the benefits that an ICT initiative in teaching might have for organizational processes within the institution (for example, improved communication), or for the external standing of the institution (for example, its public image).There is a need for a framework that goes beyond educational benefits and starts to identify 'hidden benefits' at other levels within an HE institution.
Benefits are rarely considered in relation to the strategic objectives of the institution The benefits of an ICT initiative are rarely considered in relation to the overall goals of the HE institution as defined in its mission statement, strategic objectives or learning strategy (Collis and Moonen, 2001).An ICT initiative that helped progress an institution's strategic objectives should be judged of higher value (the benefits should be considered greater) than a comparable initiative that did not progress those objectives.For example, consider an innovation which involved giving students laptops with radio cards to improve their access to learning resources which in turn required that a wireless network be constructed in a specific department.In this case, the benefits accruing to this initiative might be rated more highly if the university had to develop, as one of its strategic objectives, a wireless networked campus than if this was not an objective.
The contribution of ICT in teaching and learning cannot be isolated It can be difficult to attribute benefits to a specific aspect of a teaching and learning situation.For example, Draper, Brown, Henderson and McAteer (1996) have argued that all factors in a teaching and learning situation interact and that it is not possible to isolate the benefits of ICT from the wider context or to determine causal relationships.When ICT is introduced to support an existing educational system there will necessarily be associated changes in other areas, for example, teacher roles, student motivation, resource availability or course organization processes.

Comparing benefits across ICT projects is problematic
Another important issue is the absence of evaluation methodologies, or criteria, that would allow one to compare benefits reliably across different ICT projects (Draper, 1997;Jones, Barnard, Calder, Scanlon and Thompson, 2000).Comparisons across ICT projects are crucial to the making of investment decisions, yet, in most evaluations of ICT-based education the methodologies are tailored to the specific teaching context.Moreover, triangulation, using multiple evaluation instruments, is a recommended practice in order to be able to draw more robust conclusions about benefits (Rossi and Freeman, 1993).Yet, some commonality of evaluation methods or criteria would seem to be a prerequisite for cross-project comparisons.A further difficulty is that the value assigned to benefits may vary depending on the stakeholder, for example, teacher, senior manager (see Anderson, Haywood, Heywood, Land and Macleod, 2002).

Commentary
There are no simple solutions to the above problems.Decision-makers and researchers must accept that it will not be possible to obtain exact measures of costs and benefits and that there will always be a great deal of subjectivity in cost-benefit evaluation.Furthermore, there is no point in having precision in one area (costs) when evaluations in the other area (benefits) are likely to be highly subjective.The INSIGHT model tries to strike a balance in relation to costs and benefits by providing a framework that is pragmatic and flexible.On the costs side, a high-level approach is adopted that should be workable within current institutional financial systems.On the benefits side, the model attempts to support subjective judgements by suggesting ways in which evaluation criteria can be made more explicit and weighted in relation to institutional goals.The overall focus of the model has been to develop procedures that would support more systematic thinking about cost-benefit relationships.

Identifying costs
The main difficulty in costing ICT-supported teaching and learning activities is identifying the full range of costs associated with those activities.Where a course is Web-based the ICT-related resources (human and material) utilized in providing that course extend beyond those incurred by the academic department.The network hardware, the software, the ICT support services, and even the cost of housing the Web server, are all part of the cost of that Web-based provision.
In industry, activity-based costing (ABC) is a widely adopted technique to assess the full costs of different activities within an organization.ABC assigns costs in relation to different types of activities taking into account that these activities will often cut across organizational boundaries.Although ABC is not common in HE, this method of costing apportionment has been investigated and trialled by Bacsich and his co-workers (Bacsich et al, 1999;Bacsich et al, 2001).Implementation of ABC is very costly, complex and timeconsuming.This has resulted in significant resistance to its introduction to HE.In response to these difficulties the approach adopted here has been to take a pragmatic and high-level approach to the attribution of costs to activities.

Apportioning costs
Three distinct types of cost centre have been identified as relevant in the development of a costing framework for ICT-supported teaching and learning activities: infrastructure; value-added and support.Infrastructure cost centres are used to collect the total cost of ownership for ICT assets.For example, the total cost of ownership of a network server would encompass not only the acquisition costs but also licensing costs and external support contracts.
Value-added activities comprise any activity in an institution which results in the generation of external benefit (financial or otherwise) and which is undertaken in pursuit of the institution's primary objectives.Teaching and learning, research, commercial exploitation of knowledge, residences and catering are value-added activities in higher  , 1997).All other activities within an institution can be defined as 'support activities'.This would encompass administrative and academic support functions including central ICT support functions.The focus of this paper is value-added activities relating to teaching and learning.
Given that not all costs can be directly attributed to value-added activities, the ABC approach involves apportionment of costs from other cost centres.In Figure 1 this is depicted by the horizontal arrows from infrastructure and support costs, to value-added activities.

Cost categories
In order to develop a workable model for costing it is necessary to categorize costs into generic types that are dealt with in broadly similar ways.Figure 1 also depicts the four types of costs that have been identified as critical to the model -capital, revenue, staff costs and overheads -although other costing categories could easily be accommodated.
Staff costs account for 58 per cent of all expenditure in higher education (HESA, 1999), with academic salaries accounting for 33 per cent, therefore it is crucial that the model incorporates realistic figures for staff time.The model assumes that institutions can produce reasonable estimates based on category costs (for example, lecturer, senior lecturer).Rumble (1997) discusses the difficulties associated with this approach.
Capital costs refer to items of non-recurrent expenditure that are written off over an expected useful lifespan.The model proposes that capital costs include time spent on initial course or software development.The actual number of years over which capital costs are written off would depend on the factors that make reinvestment necessary (such as changes in technology or in course content) and on accounting conventions (for example, amortization).
Revenue costs in this model are defined as non-capitalized expenditure occurring in academic departments or for IT support activities.Having identified staff, capital and revenue costs, all other institutional costs (for example, utilities) are dealt with in much less detail and are apportioned as overheads.

Commentary
In contrast to true ABC, which assumes an institution's financial ledger would be reanalysed to extract precise costing information, the model here advocates that the figures for costs are sourced from much higher-level documentation (such as departmental estimates of staff time spent on specific activities, annual budgets and forecasts, planning documents, expenditure summaries).This methodology facilitates a quicker analysis of expenditure but places a higher burden on the user of the model (than the mechanistic ABC approach) to identify all the relevant expenditure.
The degree of accuracy that is lost in taking a high-level approach to costing in the model is justifiable for at least two reasons.First, it reduces the costs of conducting the costing exercise.Second, the effort spent in collecting detailed costs is negated when these figures are subsequently combined with the more subjective qualitative benefits information in order to derive cost-benefit relationships. -

Framework for the evaluation of benefits
Classifying benefits Initial work suggested that three broad categories of benefits would need to be considerededucational, organizational and external benefits.This categorization, as shown in Figure 1, recognizes that most institutions seek to deliver the best education that they can (educational objectives), to be effectively and efficiently organized and managed (organizational objectives) and to have a high standing within their external communities and within society (external objectives).These benefit categories would obviously overlap and interrelate in various ways in relation to a specific activity: for example, an ICT initiative in teaching might not just incur educational benefits for students but might also improve information flow within the organization or improve the standing of the institution with a particular external community.Tables 1, 2 and 3 illustrate a range of benefits within the three categories: educational, organizational and external and their associated evaluation methods.

Student satisfaction/motivation
Enhanced

Weighting and rating benefits
In applying the model, an institution would weight the selected benefits in a systematic way, in relation to its strategic objectives.For example, an institution with strategic motivations that emphasize community involvement might initially weight criteria such as 'strategic partnering' more highly than, say, 'student satisfaction'.This is consistent with the HEFCE (1999) paper, Appraising Investment Decisions, which makes the following observations: The institution must first decide what criteria [benefits] are significant and how to score them; scores can be weighted to reflect their relative importance.The results are only an aid to decision making, and can never relate to any absolute measure.However, even if the scores and weightings are fundamentally subjective, the institution will be in a better position to explain the rationale for its decision.

Commentary
The three broad benefits categories are provisional at this stage but they do provide a starting point for benefits analysis within a framework that is reasonably broad in scope.
The results of a case study of cost-benefits of online delivery conducted by Oliver et al. (1999) would suggest that the types of benefits we proposed are realistic.It is assumed that the actual benefit criteria would be chosen by the users of the model and would differ depending on the HE institution and its strategic objectives.Furthermore, the benefits could in practice be more broadly or narrowly defined.For example, instead of 'improvements in organizational efficiency' one might focus more narrowly on 'effort needed to keep information about students up to date'.

The INSIGHT cost-benefit model
Simply identifying costs and benefits does not allow a comparison of ICT investments; to achieve this, an evaluation methodology is required.In this section the six steps of the INSIGHT cost-benefit model will be outlined and illustrated using a hypothetical example.
Step I: identify the scope of the activities for each option to be included in the costing The INSIGHT model is based on 'option appraisal' techniques, where several competing options are evaluated and a relative, as opposed to an absolute, assessment of their value is made.
In our example, a fictitious institution, 'Kelvinglen University', wishes to evaluate the costs and benefits of three classes that utilize ICT in different ways, in order to determine which model should receive further strategic investment.A fourth class uses traditional teaching methods and is included as a baseline comparison.All classes are first-year, in social sciences disciplines, and have similar student numbers.
• Class 101 -'The Laptop Project'.This class utilizes university-owned laptop computers in order to promote group working and enhanced communication.
• Class 102 -'The Virtual Learning Environment Project'.This class uses the facilities provided by a centrally maintained Virtual Learning Environment (VLE).
• Class 103 -'Personal Response System Project' (PRS).This technology provides instantaneous feedback on learning to students and helps teachers manage large class discussions.

• Class 104-Traditional Teaching Methods
Step 2: determine the total cost of the activities for each option Table 4 assumes that the total cost of running each class has been carried out, including the direct costs within the teaching department and the apportioned infrastructure and support service costs.Step 3: identify and weight the evaluation criteria (benefits) A limited set of benefits (criteria) is identified against which the classes are evaluated (see Table 5).Kelvinglen University has a strategy that, among other things, emphasizes balancing innovative use of ICT with high student satisfaction and on developing a high public profile in ICT.Hence student satisfaction, innovation and public perception are given the highest weightings on a scale of 1-10.It should be noted here that the number of criteria to be used in the model is not fixed and decision-makers can determine the number that works best for their institution.However, we believe that in practice it will be best to agree on 3 to7 key criteria, those that matter most to the institution.Table 5: Benefits sought after by the University ofKeivingien and the weightings of these benefits based on the institution's strategic priorities

Class
Step 4: evaluate the options using the weighted criteria For each of the chosen evaluation criteria the users of the model must decide on an appropriate evaluation methodology.The model is not prescriptive about what methodology should be used.However, in the interests of comparability the same criteria should be applied across each option.For example, Kelvinglen University has decided to use the results of a comprehensive student survey to inform the evaluation of 'student satisfaction'.For 'staff satisfaction', survey data exists for only two of the options and therefore the evaluation group have decided to judge this criterion on the basis of anecdotal evidence.No quantitative evidence exists to inform the evaluation of 'public perception' criterion, so the group relies on judgements of an 'expert witness' -the institution's press officer.
For each option the results of these evaluations are translated into a rating on a 1-10 scale.
The rating for each evaluation criterion is multiplied by the weighting for that criterion to arrive at a weighted score.The sum of these weighted scores represents a 'measure of benefits' for each option or class (see Table 6).Step 5: determine the total cost-benefit ratio for each option

Student satisfaction
Having determined the cost and the benefits of each option (class) these can be compared to provide a relative assessment.The simplest method of doing this is to divide the benefits measures by the annual costs to produce a cost-benefit ratio (see Table 7).Step 6: plot the results graphically and discuss their meaning These results can plotted on a graph (Figure 2), with the 'Expected Benefits' line showing an average level of benefit that one might expect to accrue per £ of investment.Options above and to the left of this line give a better than average effective return per £ of cost, whereas options to the right and below the line give a poorer than average return.Potentially, if enough data were accumulated about existing and proposed services, this 'Expected Benefits' ratio could be used to judge the value of any ICT project in isolation.effective (measured against the institution's objectives) but it is also the cheapest, giving it a reasonable cost-benefit analysis.These results, however, cannot be taken at face value and would have to be explored by those charged with making investment decisions.

The INSIGHT model as a decision support tool
The hypothetical example of a cost-benefit analysis outlined in this paper described the procedures in somewhat mechanical terms.However, the real value of the model is in the way that it would be utilized by decision-makers.It is envisaged that decision-makers would be involved, at an early stage, in proposing both the costing and benefits categories as well as the benefits weightings.This means that there is considerable flexibility in the way that the model might be constructed.Indeed, it could be argued that users actually build their own decision-making model tailored to their own institutional context.Research on 'toolkits' in other decision-making domains (for example, course design) has demonstrated that this kind of model-building process has significant benefits (Conole and Oliver, 2002).It helps users to 'reflect upon and structure their thought processes' while making decisions in areas of professional practice.
It is also assumed that decision-makers would be involved in discussions after cost-benefit figures are produced.It is inconceivable that a decision-making group would make investment choices based on a single calculation.The group would invariably wish to explore the cost-benefit results that derive from different strategic priorities.This would require examining the effects of changes in benefits weightings and in ratings.So, while the model produces quantitative outputs it is assumed that these figures are provisional and that they merely act as triggers for debate and discussion.A final point to note here is that although we have described the model from the strategic perspective of an HE institution, the model could just as easily be constructed in relation to other organizational levels (such as faculty or department).

Issues to be addressed in future developments
The cost-benefit model outlined in this paper is currently being piloted across six HE institutions.However, conceptual testing has already helped us identify a number of issues that must be addressed as the model is developed.

Scope versus depth of impact
In the Kelvinglen example, it was assumed that the numbers of students participating in each class were broadly similar.However, if the numbers had differed significantly a question arises: 'is it better to implement an ICT project that has a small impact on large numbers of people or that has a big impact on a smaller group'.For example, if the costbenefit ratios were the same, would it be better to invest in a communications infrastructure that has a small effect in supporting teaching and learning across the whole institution or to invest in a highly developed system in one faculty.This emphasizes the importance of dialogue and of weighting objectives closely to strategic priorities.

Timing of cost-benefit analyses
A second issue concerns the timing of cost-benefit analyses.The Kelvinglen example assumed that the options appraisal was carried out after implementation.The advantages of retrospective appraisal are that it is possible to collect harder evidence of costs and benefits.If the appraisal were carried out prior to the start of a new initiative (for example, to obtain funding) then the costs-benefits data in the model would necessarily be provisional and more subjective.Nonetheless, new initiatives are normally run as pilots and hence increased subjectivity at this stage would normally be expected.Another concern is that the costs and benefits deriving from the first run of a new initiative might be quite different from those that occur when the change has become embedded into operational procedures.While the costing model suggests gradually writing off the initial start-up costs (amortizing) over an extended time-span this has not been shown in the example and no corresponding adjustment has been suggested for benefits.
Deriving evaluation data that can be used for ICT project comparisons A third issue concerns the difficulty of obtaining measures of benefits that would allow comparison across different ICT projects.As the Kelvinglen example shows, there is no simple solution to this problem and especially when one might be comparing investment scenarios that produce quite different benefits.However, in practice, with a small number of criteria (aligned to strategic objectives), evaluators should be able to rank order the benefits of different projects in relation to these criteria.Moreover, benefits evaluation data could be made more systematic and comparable.This could be done by employing an independent evaluator to scrutinize the findings of the evaluation reports from different ICT projects, by ensuring that each project is evaluated across a common set of criteria, and by having different project evaluators discuss their findings as a group before making recommendations to the decision-making body.

Conclusion
The cost-benefit model described in this paper was devised to help decision-makers evaluate choices regarding ICT investments in teaching and learning.Key features of the model are that it can be adapted to quite different organizational, financial and ICT structures; that it concentrates on high-level costs; that it takes an institution-wide perspective on benefits; and that it focuses decision-making on a small number of strategic priorities, those that matter most to the institution.Finally, the model is a work in progress and we welcome feedback from readers to help us refine our thinking.

Note
An earlier report on the INSIGHT project that was submitted to JISC which includes a software model can be found at: http://www.mis.strath.ac.uk/predict/projects/insight/ index.htm.
Figure I: The cost benefit model depicting the allocation of costs to value-added activities and the categories of benefit deriving from those activities

On examining Figure 2 ,Figure 2 :
Figure 2: The relationship between the costs and benefits of each option

Table I :
Educational benefits and example evaluation methods

Table 2 :
Organizational benefits and example evaluation methods

Table 3 :
External benefits and example evaluation methods

Table 4 :
Total costs of each class including direct and indirect costs

Table 6 :
Measure of benefits for each class/option (sum of the weighted scores)

Table 7 :
The cost-benefit ratio for each option (benefits divided by annual costs)