The Online Faculty Development and Assessment System

This article evaluates the role of the Online Faculty Development and Assessment System (OFDAS), created at universities in the Canary Islands, Spain, in staff development. The evaluation indicates that the system helped staff in learning to teach curriculum and teaching capacities. The tasks, online resources and opportunities for discussions provided within the learning environment created for the system helped shape their attitudes towards learning curriculum and teaching capacities and enabled them to share their concerns about students’ classroom learning environment assessment.


Introduction
As with other European higher education institutions, Spanish universities have experienced an exponential increase in the number of staff professional development courses offered online.At two public Spanish universities in the Canary Islands, distance education initiatives and faculty development courses have been used recently in targeted disciplinary subjects.The Online Faculty Development and Assessment System (OFDAS) is a voluntary education programme designed to enhance practitioners' pedagogical awareness and understanding.It includes topics such as teaching beliefs, educational quality and pedagogical excellence in the university profession (Caffarella & Zinn, 1999).It also includes topics such as planning, organizing, structuring, tracking, reporting and communicating assessments (Nijhuis & Collis, 2003).As discussed by Fitzgibbon and Jones (2004), the social dimensions and organizational factors of online faculty professional learning are also important and were taken into account when creating the system.The instructional design approach used in the OFDAS was guided by the principles of instructional systems design (Oliver & Herrington, 2003), considering both the technical and pedagogical factors associated with the development of quality digital materials.

Aim and objectives of the research
The aim of our research was to develop and validate an online framework of CTC called the OFDAS as an intelligent tutoring system.We set out to examine three objectives: the learning experiences of faculty in the online system, evaluation of what the participants learnt about CTC, and students' evaluation of the classroom learning environment.The overall research question was 'how did the OFDAS elements affect staffs' opinions and what was its impact on teaching attitudes and on the resultant learning environment?'This question was examined via three subquestions (Figure 1).
1. Was there a difference in the practitioners' opinion about the quality of the OFDAS? 2. Did the practitioners learn how to teach CTC having completed the OFDAS? 3. Was there a relationship between the teachers' attitudes and students' learning environment after completion of the OFDAS?The OFDAS model The OFDAS model consisted of the following elements: (a) A four-hour, face-to-face mentors and participants workshop.(b) A sequence of structured and comprehensive lessons.Learning activities were designed to engage and direct the participants through the process of CTC knowledge acquisition, ensuring that the development of the instructional CTC transferred to practical classroom settings.(c) A communication support system to scaffold the teaching and learning process, as well as a mechanism for providing meaningful feedback and sharing of ideas/ problems with colleagues.(d) A set of associated learning resources to complement the learning activities and to provide guidance.(e) A specific inventory of students' evaluation of the classroom-learning environment generated by the participants to provide them with feedback on matters relating to participant instructors' learning.
2. They interpreted resource materials that were divided into 10 CTC lessons, released on a weekly basis.These focused on seven basic dimensions, similar to the teaching competencies framework proposed by Tigelaar et al. (2004).The structure of a CTC was described as having a four-phase cycle: purpose, uses, educational setting and case study.The resources formed a substantial body of material, including 156 pdf and html documents (on relevant teaching and learning facts, concepts and theories), 114 web sites, 10 PowerPoint presentations, and over 500 glossary items on relevant educational concepts and references with embedded hyperlinked in the materials.Mentors also used voice-over Internet protocols (see Service and Content Information in Figure 1).3. Participants were also expected to take part in two discussion topics in the asynchronous forums: 'European convergence issues' and 'Student mental effort to cope with the new European credit system'.These themes were organized and released on a fortnightly basis, but remained accessible throughout the remainder of the course.The last forum included reflective questions.The forums were a core element of our design approach as we believed that participation in the forums was a crucial element as it fostered collaboration, social dialogue and negotiation of meaning between the participants.Blignaut and Trollip (2003, p. 152) note that 'Determining the elements of faculty participation and involvement can lead to the development of improved skills, which in turn may lead to improved learner satisfaction, instructor satisfaction, and the lowering of attrition rates'.4. They used email for one-to-one interactions with mentors or other participants.5.They could browse the curriculum materials and resources containing URL links to related articles and institutions, notes and grades from any location, at a time and pace that suited them.6.They could download the presentations, key concept maps, study guides and resources onto their personal computer.7.They were able to submit their learning activity assignments via a Web-forms interface or by email.These assignments were designed to be authentic activities that had real-university relevance linked to classroom practice and that presented complex teaching-learning tasks to be completed over a sustained period of time.
The assessed activities aimed to be realistic representations of the tasks in which the authors wanted to demonstrate capacity; and therefore participants had substantial freedom in selecting activities relevant to them.This is in line with the literature on the importance of authentic assessment (Uhlenbeck et al., 2002).8. Participants were expected to complete 10 online tests; the answers were recorded in a database on the server.Each CTC test was programmed via random selection to be unique and provided instant feedback to the participants.
Participants evaluated the quality of instructional materials and of the training process as a formative evaluation for course revision, so that an authentic assessment was included, which was seamlessly integrated into the learning activity assignments, and which provided a formative assessment of their understanding of basic concepts and theories, aiding them to gain a sense of progress.9.They were asked to provide evaluative feedback on the course and the multimedia OFDAS environment by completely the Attitude Towards Course Learning Questionnaire (ACLQ) (see Table 1).10. Finally, they also received feedback on the students' evaluation of their classroom learning environment; the students completed the Assessment of University Teaching Activities Questionnaire (AUTAQ) system (see Table 2).

Faculty competence
Our OFDAS framework emphasizes a student-centred approach to teaching and learning (Villar, 2004), which we believe is innovative and helps in terms of 'embedding' e-learning into everyday classroom practice.The OFDAS focus was on student learning experiences and processes (e.g.major concepts, assumptions, processes of inquiry and ways of student knowing) within the university social context (Badley, 2000).Participants were expected to have a deep understanding of their subject discipline field as well as the necessary pedagogical and didactic skills.Core elements in the OFDAS programme were development of participants' competence in the design of curriculum and course material, along with development of their generic didactic and guidance skills (Tigelaar et al., 2004).In addition, participants were expected to be applied researchers in charge of constructing and interpreting activities, having beliefs and a voice of their own (Wildman et al., 2000).

Classroom learning environments
The general literature regarding students' evaluation of classroom environments has increased over the past decade (Aldridge & Fraser, 2000) and the types of learning environments has undergone remarkable 'diversification and internationalisation' (Fraser, 1998, p. 7).Evidence (derived largely from on-demand university teaching quality assessment) had been accumulating regarding the potential of classroom learning environment assessments to improve university teaching and learning as well as staff development (Dallimore et al., 2004).In addition, some results have showed that students' 'sense of belonging' was an important predictor of satisfaction measures (Thomas & Galambos, 2004).Thus, a list of assumptions underpinned the development of the system.First, student evaluations are influenced by the students' demographic characteristics and background factors (Worthington, 2002;Barfield, 2003).Second, students' interpersonal skills are also an important factor in enhancing their academic focus and their perceived satisfaction with the social environment of the class (Lindblom-Ylänne et al., 2003).Third, we wanted to incorporate the students' evaluations into the OFDAS model and as such the evaluations were used as feedback to the course participants (Schelfhout et al., 2004).Fourth, we also believed that the students' evaluations were valuable as they represented one means of assessing teaching quality (Wierstra, 1999).Fifth, students' evaluation was used as a mechanism to improve the quality process (Villar & Alegre, 2004).Finally, we built on previously research in the field, which provides a means of validating our approach (Dorman, 2000).

Method
The study focused on the experiences of staff and students at two campuses.Participants in the study included academics from 24 tenured and contracted instructors who enrolled on the OFDAS.All participants were volunteers and met the following selection criteria: located on a university campus, associated with a particular Note: (+), items are scored one, two, three, four and five, respectively, for the responses 'almost never', 'seldom', 'sometimes', 'often' and 'almost always'; (−), items are scored five, four, three, two and one, respectively, for the responses 'almost never', 'seldom', 'sometimes', 'often' and 'almost always'.discipline, and had professional merits.The faculty were full-time instructors at two public Canary universities: 11 from the 'research-led old' University of La Laguna (www.ull.es)(46%) and 13 from the 'technological-led new' University of Las Palmas de Gran Canaria (www.ulpgc.es)(54%); 10 participants (42%) were men and 14 participants (58%) were women.With regard to teaching experience, 19 participants (79%) were experts (with more than five years of teaching experience) and 14 had a Ph.D. (58%).Participants were from a range of different disciplines: social science (eight participants), experimental sciences (five participants), healthcare sciences (four participants), humanities (three participants) and technical sciences (four participants).The OFDAS programme took place during the spring quarter of the 2006 academic year and lasted 11 weeks.
As emphasized by other researchers, faculty opinion and student perceptions are two of the best means of evaluating the quality of CTC (Pratt, 1997).Three evaluation tools were used: • CTC quality scale.This scale was used to measure participants' ability to understand the CTC and the degree to which individuals or groups wished to practice and apply them (Cronbach's alpha = 0.944).The scale consisted of 10 items about the CTC structure, conditions, technologies and teaching practices, and was designed to assess participants' perspectives on the extent to which their personal CTC learning was enhanced.A five-point scale was used: one = 'strongly agree' to five = 'strongly disagree' in items one to seven; items 8-10 had tailored five-point scales.The 10 items covered aspects such as the relevance, usefulness and appropriateness of the CTC, suggested adaptations, tips, structure, pertinence, reading, impact and time-consuming (see Figure 1, OFDAS Delivery).• Attitude Towards Course Learning Questionnaire.The ACLQ measure was designed to be used as a generic tool for the participants using CTC learning activities (Cronbach's alpha = 0.950).The ACLQ consisted of 20 items using a five-point Likert-type rating scale (one = 'strongly agree' to five = 'strongly disagree') (see Table 1).• Assessment of University Teaching Activities Questionnaire.The AUTAQ was administered to 78 students taking courses at the two universities.The sample was representative of gender, age, area of study, level of study and other academic and social characteristics.The questionnaire focused on the students' perceptions of the classroom learning environment (Cronbach's alpha = 0.958).The questionnaire consisted of six scales and 22 items scored on a five-point Likert-type rating (see Table 2).
A variety of data analyses were completed on data gathered through three instruments.These included descriptive statistical summaries, alpha reliabilities of the subscales of the ACLQ and AUTAQ surveys, and t-tests to compare the means of participants.In addition, a 10-point scale was applied by the two mentors to all of the CTC participants' activities based on an interpretation of script expressions: 'maximum distinction (9-10)', 'important for its intensity (7-8), 'suitable (5-6)', 'minimum qualification (3-4)', and 'differed execution (0-2)'.Content analysis was chosen as a methodology for analysing the participants' online learning activities, which involved comparing, contrasting and scoring them.
The OFDAS was designed to make teaching and learning more of a collegial activity.Decisions about what was desirable and feasible pedagogically and technologically were made through telephone conversations and emails.The participants involved in the programme explained the purpose of the research to their students and assured the students that all data would be anonymized.Ten multiple-choice items were used to assess the participants' knowledge and understanding of CTC.Online data were collected during the course (i.e. the CTC quality scale, CTC learning tests and the ACLQ); the AUTAQ was collected after participants had completed the OFDAS programme.

Results
The first research question asked participants about their opinion of the quality of the materials and the OFDAS online learning environment.Responses ranged from 3.08 (item eight, reading: 'I read Web sites and pdf documents which were linked to the capacity') to 1.33 (item one, relevance: 'The capacity was relevant for my teaching').Standard deviations varied from 1.52 (item eight, reading) to 0.76 (item 1, relevance).All item mean scores exceeded the midpoint scale (3.00, normal), and item eight (reading) exceeded the midpoint scale (3.00, frequently).
A t-test was conducted on each item.In terms of gender differences, there were significant differences in five of the CTC quality items (usefulness, adaptation, tips, structure and pertinence).In terms of the level of educational achievement, a significant difference was found in eight of the CTC quality items (relevance, usefulness, appropriateness, adaptation, tips, structure, pertinence and timeconsuming).In terms of the level of teaching expertise, differences were evident for five of the CTC quality items (usefulness, appropriateness, adaptation, tips and structure) (see Table 3).
Question two focused on whether the participants felt that the OFDAS facilitated their learning in this area.This question was divided into two subcategories: facilitating learning activities and assessing the cognitive domain of CTC learning.
Descriptive summaries provided information about the activities undertaken by the participants for each of the 10 CTC.The results showed that overall the participants completed 1587 learning activities.The coaching and scaffolding for the course was done by the two OFDAS mentors, who diagnosed the strengths and weaknesses of each participant and tailored their support accordingly.Activity transcripts were scored, and the scores were then actively discussed to arrive at a final version in which most scored learning activities had been brought into alignment.All CTC were approved by participants except for the last two: CTC9, 'Knowledge of formative and summative evaluation'; and CTC10, 'Capacity to conduct own self-assessment process') (see Figure 2).Means and standard deviations on the 10 self-assessment test scores are shown in Figure 3, which indicates that results overall were positive.However, objective testing of the CTC showed that performance was more effective in CTC7, 'Teaching and didactic skills for large groups', than in CTC1, 'Knowledge of student motivation and ability to promote students' positive attitudes'.There were significant gender differences in the learning of CTC3, 'Capacity to solve students' problems' (t (15) = 2.520, p < 0.018).Also, significant differences were found between instructors with and without previous educational knowledge in: CTC1, 'Knowledge of student motivation and ability to promote students' positive attitudes' (t (15) = −3.119,p < 0.008); in CTC3, 'Capacity to solve students' problems' (t (15) = −2.477,p < 0.027); in CTC4, 'Capacity to develop metacognitive skills in the trainee' (t (15) = −2.385,p < 0.032); in CTC7, 'Teaching and didactic skills for large groups' (t (15) = −2.449,p < 0.028); and in CTC8, 'Knowledge of questioning skills' (t (15) = −2.590,p < 0.022).Finally, with regard to teaching experience, significant differences in learning CTC3, 'Capacity to solve students' problems', were found between novel and expert participants (t (15) = 2.800, p < 0.015).The student sample consisted of 78 undergraduate students from a variety of disciplines from the two Canary Island universities.The first index of validity was scale reliability.Estimates of the internal consistency of the actual and preferred forms of each AUTAQ scale were calculated using Cronbach's alpha coefficient.Data were reported separately for the two forms using the individual as the unit of analysis.The scale values obtained for the alpha coefficient ranged from 0.083 to 0.830.These data together suggested that each AUTAQ scale had adequate internal consistency, except for 'Use of resources'.It appeared that the AUTAQ measured distinct, although somewhat overlapping, aspects of classroom environment.The scale values for the ACLQ obtained for the alpha coefficient ranged from 0.739 to 0.911.These data together suggested that each ACLQ scale had adequate internal consistency.Inter-correlations showed the association between the students'

Discussion
Overall participants' evaluation of the programme was positive.The manner in which the mentors managed the OFDAS had a direct impact on their appreciation of the programme and also on the CTC learning results, which mirrors findings from other Web-supported courses (Nijhuis & Collis, 2003).As discussed in the findings section, some differences in attitudes were evidence in terms of gender, level of qualification and teaching experience, which supports our first research question about the participants' opinions with respect to value and ease of use of CTC in the OFDAS.
The usefulness of learning activities can be inferred from the fact that the participants actively engaged with the programme, completing 1587 learning activities.Caffarella and Zinn (1999, p. 253) pose the question 'Do professional development activities assist in a faculty members' professional success?'We argue that from our study the answer is yes; all but two of the CTC learning activities were rated positively by participants (Figure 2), and all 10 CTC self-assessment test means exceeded seven on the 10-point scale used (Figure 3).Significant differences in CTC learning were found between participants in three nominal variables: gender, prior educational knowledge and level of teaching experience.
The results stressed two somewhat different but conceptually related measures and brought about new perspectives on assessing learning environments in higher education settings.The 'climate' scale, particularly, emphasized the importance of the development of mature, interpersonal relationships and friendships, social bonds, and connections with other students, as a vector of behaviour of student development (Lounsbury et al., 2005).The results from the AUTAQ were given back to participants; Kember et al. (2002) adopted a similar approach with the Student Feedback Questionnaire used in their study.Correlations between the two measures used suggested that the attitudes of faculty towards teaching were not particularly related to the students' perception of the classroom environment, except on the 'Clarification with learning' and 'Evaluation' scales.Contrary to the results of Fraser's (1998) research, the findings in our study did not support the conclusion that there was a close relationship between faculty teaching attitudes and student classroom perceptions.
Overall, the findings are encouraging; all CTC were perceived to be useful and easy to use, although at varying levels depending on specific factors such as how time-consuming they were.As Fitzgibbon and Jones (2004) report, appropriate coordination of such an online programme is crucial.With directed and purposeful design efforts, and by determining which activities are best suited for different disciplines, participants completing the programme are likely to enhance their understanding of CTC and be able to apply them in their own practice.
Finally, the variables explored in this study seemed to directly address ongoing concerns about the need to improve online training in higher education as well as emphasizing newer ideas about important variables that might be measured as alternatives to the more traditional approaches in the evaluation of staff development (Ellett et al., 1997).In comparison with other questionnaires (see, for example, Tucker et al., 2003), the ACLQ and AUTAQ online systems facilitated timely data collection, feedback and evaluation.In general, participants reported that collecting feedback online with the AUTAQ system was convenient, as had already been remarked by researchers regarding other online systems (Bullock, 2003).
One of the limitations found in this study was that it only examined one online programme and its use in two public urban universities over an 11-week period.Because the participants volunteered their classes for inclusion in the study, these were not randomly selected.Therefore it is questionable to what extent the findings can be generalized to other university contexts.Another limitation was that response rates for the student online evaluation were low, a common problem in studies of this kind (see, for example, Ballantyne, 2003).Finally, it is worth noting that although the AUTAQ consisted of two sections, only section two was used; Barfield (2003) did use both in a related study.Oliver and Herrington (2003) note the value of focusing on learning tasks and the 110 learning activities included in the system were chosen with care to get across the key concepts of the course.Good resources, however, are not enough; given the huge time pressures and work commitments of staff, an online staff development system also needs elements of 'enticement' and inclusion of appropriate support (Salmon, 2005).We contend that one of the most important issues associated with online development programmes is not so much the quality of the resources, but the ways in which they are used to enable instructors from different disciplines to enhance the status and level of their pedagogical practice.

Conclusion
Our evaluation has demonstrated that the OFDAS programme was an effective tool to improve reflective practice on teaching and learning.No correlation was found between staffs' teaching attitudes and students' classroom learning environment at the end of the programme.Our main conclusions about the use and effectiveness of the system are that: it was evident that there was positive mentor-instructor interpersonal relationships with respect to the development of the instructors' pedagogical content knowledge (e.g.instructional approaches, strategies and representations of subject-specific topics); there was insufficient time provided for the online professional development given the time and commitment required to engage with and apply CTC to their practice; and there was some personal beliefs about the association between the CTC online professional development and actual student evaluation of the learning environment.

Figure 1 .
Figure 1.The OFDAS model and sample variables

Figure 2 .
Figure 2. Learning activity qualifications Test means varied from a high score of 7.4 (CTC1, 'Knowledge of student motivation and ability to promote students' positive attitudes') to a very high score of 10 (CTC7, 'Teaching and didactic skills for large groups').All 10 test means exceeded score seven on the 10-point scale used.Means and standard deviations on the 10 self-assessment test scores are shown in Figure3, which indicates that results overall were positive.However, objective testing of the CTC showed that performance was more effective in CTC7,

Table 1 .
Description of scales and a sample item for each scale of the ACLQ

Table 2 .
Description of scales and a sample item for each scale of the AUTAQ

Table 3 .
Significant t-test results for demographic and academic factor comparisons Online Faculty Development and Assessment System 227 perceptions (AUTAQ scales) and the participants' teaching attitudes (ACLQ scales).Only two correlations were statistically significant (p < 0.05), negative in direction and somewhat moderate: r = −0.548,Clarification, actual/learning; and r = −0.548,Clarification, actual/evaluation.