Identification of critical time-consuming student support activities in e-learning

Higher Education Staff involved in e-learning often struggles with organising their student support activities. To a large extent this is due to the high workload involved with such activities. We distinguish support related to Learning contents, Learning processes and Student products. At two different schools surveys were conducted to identify the most critical support activities, using the Nominal Group Method. The results are discussed and brought to bear on the distinction between content, process and product related support activities.


Introduction
Modern higher education curricula increasingly make use of information and communication technologies and thus increasingly implement e-learning.This applies to open, distance education as well as more traditional forms of education (Guri-Rosenbilt, 2005).It is well known that the introduction of e-learning often leads to an increase in staff time spent on tutoring (Bartolic-Zlomislic, 1999;Bacsich et al., 1999;Bie, 2002;Romiszowski & Ravitz, 1997as cited in Fox & MacKeogh, 2003).Rumble (2001, pp. 81-82) mentions even a doubling of the load.One of the most important reasons for this is that often an extended classroom model is applied (Beaudoin, 1990;Salmon, 2004).That is, in addition to the usual lectures and availability during office hours, a teacher creates a website to support the course and is available for email help between classes.Staff who use e-learning environments to organise student support this way profit little from the economy of scales: students are treated as help seeking individuals, rather than as a group.The problem is even exacerbated as online students -rightly or wrongly -expect to be able to send emails to support staff and receive answers quickly (Salmon, 2004).Beside this the characteristics of the questions for support change.As Anderson (2004) points out, tutors in an e-learning context are no longer restricted to well-defined and pre-planned tasks but have to adopt on the fly to student needs.A tutor has to make provisions for negotiation of his activities to meet unique learning needs and equally well has to stimulate, guide and support the learning in a way that responds to common and unique student needs.A way out of this predicament is to develop technologies that make support activities significantly less time consuming.This is not as easy as it may sound.Although it isn't hard to involve technology in teaching and learning, this does not necessarily lead to more efficient practices.One should not simply develop software that offers technological support without determining first whether there is a true need (Coopers & Saunders, 2000).This kind of approach readily becomes costly, more time-consuming and thus ineffective.So the challenge is to select e-learning based support activities that are worthy of technology support and to make sure that the perceived need is a genuine one (Koper, 2004).
Three categories of support activities in blended learning environments are distinguished: 1) support related to the learning content (CONT) 2) support related to the learning process (PROC) and 3) support related to the learning product (PROD) (Reid & Newhouse, 2004).Content related support refers to all tutor activities that pertain to the subject matter.Cases in point are answering content related questions, providing additional explanations or examples with regard to the subject matter, et cetera.As a result of the introduction of e-learning, staff reports an increase in the flow of content questions and answers from and to students.This is partly due to an increase in opportunities to reflect as a consequence of written interaction instead of oral interaction with the result that a content question is discussed on a deeper level.In addition, the existence of an interaction pattern that involves all students instead of only a few of them also intensifies the tutor-student communication.After all, a question posed by the staff to enable students to rehearse the subject matter now reaches each student individually and not only the ones who are paying attention in class (Coppola, Hiltz & Rotter, 2002).
Process related support refers to all tutor activities that have to do with the learning process of individual learners or collaborating of groups.Examples are providing 'study aids' or moderating group discussions.In blended learning environments staff sees their role change from the "Sage on the stage" to the "Guide on the side".Instead of being responsible for student acquisition of knowledge their responsibility shifts to moderating student activity in, for example, collaborative groups (Coppola, Hiltz, & Rotter, 2002).In other words, due to the introduction of e-learning the emphasis of the support activities shifts from content related support (i.e., transmission of knowledge) to process related support (i.e., facilitating and guiding students) (Hardless & Nulden, 1999;Beaudoin, 1990).As Denis et. al. (2004) claim, an e-tutor is "… someone who interacts directly with learners to support their learning process when they are separated from the tutor in time and place for some or all these direct interactions." Product related support refers to all tutor activities that pertain to the summative assessment of student products, such as checking for authenticity of the product or correcting tests.Although the emphasis on process related support in blended learning environments also results in a greater emphasis on formative assessment (Hardless & Nulden, 1999), the summative assessment of student products remains an important support activity (Beaudoin, 1990).Nowadays, with the arrival of Internet and the availability of online journals, there is an eye-catching problem that warrants the attention of tutors in blended learning environments, to wit plagiarism.It appears that the detection rates of plagiarism are only 1.5 % and that approximately 20 % of the tutors ignore obvious plagiarism because of the hassle of dealing with it (Bennett, 2005).It would seem then that extra effort is necessary to put a stop to these practices.
Three categories of support activities -content related support, process related support and product related support -have been discussed and illustrated by the changes that tutors have to face when e-learning is introduced in their curriculum.The present study aims at identifying the critical support activities in both distance and e-learning enriched regular education that can be supported by technical solutions.With regard to the activities identified, it furthermore investigates whether tutors in blended distance education identify problems that are different than those identified by their colleagues in blended traditional education.Finally, it explores whether the categories of support activities differ in perceived importance.Here too, the opinions of tutors in blended distance education are contrasted to those of tutors in blended traditional education.

Materials and methods
Two separate brainstorm session were organised, each structured according to the Nominal Group Approach (Dunham, 1998).One involved a group of stakeholders at an open distance learning institute (Open University of the Netherlands, OUNL), the other stakeholders from a traditional teacher training school (Fontys University for Professional Education, Fontys).The latter have been practising forms of blended learning for several years.The participant groups at the two schools did not interact with one another at all.
Both schools are situated in The Netherlands, but cater for different students.The OUNL serves mainly students who study at their own place, pace and time.Some are degreestudents, but most are life-long learners, who study to improve their job qualifications or even for pleasure.Students range in age from 18 up to 80 and over.Ever more, the OUNL relies on ict-tools for its learning management, such as newsgroups and an in-house-built virtual learning environment (VLE).The Fontys teacher training institute teaches full-time (daytime) as well as part-time students.The full-timers outnumber the part-timers only by a margin.For both groups e-learning tools such as e-mail, the web, a VLE (N@tschool!) are used intensely.
The OUNL group consisted of 12 persons, among them faculty members, educational designers (5) and staff of the teacher training institute.They were not randomly drawn from the OUNL's faculty, but rather carefully selected for their expertise and experience, which included course design, tutoring, help-desk support, educational research and software development.Here 3 of the participants provided no scores at all or 'illegal' scores and were subsequently ignored, bringing the OUNL-group down to a total of 9 members.The Fontysgroup consisted of 7 persons, all working at the teacher training faculty.They were not chosen at random either, but selected because they fulfilled managerial duties beyond their teaching responsibilities.Not only would the group of all teachers have become unruly large, it was felt that teachers with managerial duties would have a more informed opinion on the matter.Here 2 of the participants provided 'illegal' scores and were subsequently ignored, bringing the Fontys-group down to a total of 5 members.
By way of preparation, the organisers of the brainstorm invited the participants in each group to consider the following questions: Which support activities currently lead to staff workload problems?Which support activities do you find relevant but are not common practice because of workload constraints?
At the subsequent face-to-face sessions (a separate one for each group), the brainstorm organisers (the same people for each group), briefly introduced the nominal group approach to the participants.Referring to the preparatory questions, every participant was then asked to draw on his or her personal experiences and describe as many situations as possible of critical student support activities.They were stimulated to be creative and take risks i.e. also to take into consideration situations they believed to be critical but had not had experienced so far.The participants within a group were stimulated interact with each other, if they so wanted.However, they were asked not to criticise any contribution, all ideas were to be considered equally good or bad.
With a great many ideas on display, the participants were then asked, as far as necessary, to briefly explain them.After eliminating the apparently synonymous items (this was discussed in the group), they were asked to add new ideas, if they felt they had any.In the end, the OUNL-group came up with 37 items, the Fontys-group with 13 items.
Finally, the participants were asked to give votes to the items, as a reflection of an item's degree of importance.They could attribute either 1, 2, … 9, or 10 votes.Moreover, each number of votes could be allocated once only.This means that, for each individual participant, 10 items would each receive a unique number of votes and the remaining would receive none.It also means that each participant has a total of 55 votes to distribute.
The method followed is in accordance with the Nominal Group Approach.Clearly, another allocation scheme could have been followed, for instance one in which participants could freely distribute votes over categories.In principle, they then could have allocated all votes to one item only or, conversely, have distributed their votes evenly over all items.We chose the present scheme with limited distribution possibilities as we wanted to force the participants to consider a sizeable number of items (10), but not so large a number that they could effectively refrain from choosing.The latter could happen easily, as the participants themselves came up with the various items and might thus be inclined to favour their own items over those of others.
An effort was made to categorise the variety of items that resulted from the Nominal Group Approach.4 Experts, all authors of this paper, allotted each of the 50 items to one and only one of 3 categories (CONT, PROC, PROD).After the first round of allotment, no effort was made to align the opinions of the experts.The language of the items that the experts were ambiguous about, clearly was to blame, rather than confusion among the experts about the categories themselves.In order to estimate the inter-observer reliability, Cohen's kappa (1960) was used.The computation is straightforward for 2 observers (and several categories) or several observers and 2 categories.The present case has 4 observers and 3 categories.Following Fleiss (1981) and Landis and Koch (1977), first 3 separate kappas were computed, one for each of the 3 categories (case several observers and 2 categories).Effectively, for the computation of each individual kappa, one of the 3 categories was set apart and the remaining two categories were lumped.These 3 kappas provided insight in the inter-observer reliability per category.Subsequently, the overall kappa value was computed as the weighted average of the resulting kappas (again, following Bonnardel).This provides insight in the inter-observer reliability for the entire allotment exercise.

Findings
Tables 1 and 2 show the items that came out of the nominal group session for the OUNL and Fontys, respectively.Also the total number of votes each item received is indicated.This number ('score') is an indication of how relevant the participants a particular item deemed to be.
Table 1.Critical situations as identified by selected staff members of the Open University of the Netherlands.Each entry lists an item suggested by one or more of the participants.Votes reflect the cumulative votes of all the participants.Items are ordered by the number of votes they received.See the Material and Methods section for how the totals have been computed.See the text for an explanation of the Category column.Table 2. Critical situations as identified by selected staff members of the Fontys Teacher Training School, Sittard, Each entry lists an item suggested by one or more of the participants.Votes reflect the cumulative votes of all the participants.See the Material and Methods section for how the totals have been obtained.See the text for an explanation of the Category column.

ID Support activities Fontys brainstorm
Score Category 43 How to help students who are in the final phase of their study, organise their own support activities.
1 PROC 50 The administrative systems need to support the existing diversity of different training programmes.

PROC
42 Give support in laboratories while students are working and walking around.9 ** 48 Individual problems can be clustered in order to provide support in a group session.These should be traceable both individually or for a whole group.13 PROC 40 Staff should limit the number of visits to their placement students by using efficient technical solutions like video observation and feedback via a Virtual Learning Environment.

PROD
46 Students can communicate using IT about their own questions and problems, without intervention of staff.

PROC
38 Individual students might be helped in their preparation of lessons (being teacher-trainees) with a database of video materials, with explanation and well-designed assignments.

CONT
39 Support systems should be organised so that requests for support from students can be met adequately.

PROC
49 The demand for support is rising due to the individual learning arrangements, therefore staff time needs to be organised more efficiently.

PROC
47 Individual student's products can be shared by a group, fellow-students can comment (electronically).

PROC
45 Just-in-time support on the basis of questions from students is growing; staff availability needs to be organised cleverly.

*
44 As students are expected to study independently, staff often receives no questions at all or too late; so staff cannot keep track of the study progress.
41 Portfolios of students should be easily accessible and well structured, so that staff can easily find what they are looking for and do not lose much time.

PROD Total number of votes 275
Are all items equally important?
The first question to be answered, for each school separately, is whether the participants consider some items more important than others.Inspection of the tables suggests clear agreement on the side of the participants on what items matter and what not.The possible range for the scores on each item in Table 1 is 0 to 90 votes, in Table 2 it is 0 to 50 votes (the maximal score is the maximal score per participant, 10, multiplied by the number of participants, 9 and 5 respectively).Actual ranges are 0 to 49 and 1 to 41.
Closer inspection of the data in Tables 1 and 2  Looking at the results this way, little in the way of a pattern may be discerned.Although is it very useful to know that 'the prevention of fraud with student papers' or of 'repeatedly having to answer a similar question' are high on the agenda of items to be resolved, one would have liked to know what support issue in general staff at either school is concerned with.Also, it would be interesting to know whether the schools differ in what matters to them and what not.
Unfortunately, the nominal group method is ill-suited to answer this kind of question.As discussed, according to it, participants themselves formulate items.This has the benefit that one maximally taps into the creativity of the participants.It has the drawback that it is impossible to establish any trend in the participants' judgements; nor can one make intragroup comparisons, for that matter.

Do the opinions of participants within a school and between schools concur?
In order to be able to draw more general conclusions and to compare schools, first each of the items produced by either group of participants was categorised by experts as being an instance of either content related support (CONT), learning process related support (PROC), or student product related support (PROD) (see the final column in Tables 1 and 2).This way a common denominator was established.The 4 experts received the following instruction when logging their category choices: Content related support refers to all those activities support staff undertake that are related to the subject matter; for example, answering content related questions, providing additional explanations or examples with regard to the subject matter et cetera.
Process related support refers to all those activities support staff undertake that are related to the learning process of individual learners or collaborating groups; for example, providing 'study guides' or moderating group discussions.
Product related support refers to all those activities support staff undertake that are related to the formative assessment of student products; for example, checking for authenticity of the product or correcting tests.
The subsequent analysis will bear on the categories and the number of items (frequencies) that have been allotted to them.The judgements of the experts that have carried out the allotment thus is a decisive factor.If the experts disagree on the categories to which the items belong, further analysis is of little usse.Fortunately, the weighted mean value of Cohen's kappa equals 0.61, which according to Fleis (1981) reflects a 'good' degree of inter-observer reliability.
With the expert judgements in hand, items were categorised according to the following rules: Items that according to 3 or 4 of the experts should belong to, say, category A were allotted to category A. Items to which this applies are labelled CONT, PROC, or PROD in tables 1 and 2.
Items that less than 3 of the experts allotted to the same category were ignored.So all permutations of scores (2,2,0) and (2,1,1) were left out.In Tables 1 and 2 these items are labelled with one or two asterisks, respectively.These 7 items are ignored henceforth, leaving a total of 43 items for further analysis.Table 3 shows the number of items that ended up in the different categories.A Chi-square test for deviations from a uniform distribution of items over categories, which one would expect under the null-hypothesis of no prevalence, made no sense.The number of categories ( 3) is too small that convey any power on such a test.The sensibility a test is further comprised by the low frequencies in two of the cells (1, 2).Similar arguments go for a test of independence, which could show Fontys staff to hold a different opinion than OUNL staff.The Table nevertheless suggests that both schools believe that the process category is the most pressing, as it contains by far the most issues.

Discussion & Conclusions
The use of the Nominal Group Method led to a bewildering diversity of critical situations, identified by staff at both schools.It did become clear, though, that collectively staff deemed some of the situations important and others not important at all.Tables 1 and 2 pointed to the particular importance of: − Preventing fraud in papers − Giving feedback on the progress of students' work − Filtering out 'repeated' questions − Making portfolios of students easily accessible and well structured.
It is remarkable that 3 out of 4 are process related support activities, preventing fraud being the exception (it is a product related activity).The suggested importance of the process category is further reinforced by the observation that the four activities directly below preventing fraud are all process related (see Table 2).Table 3 further underscores this finding.
It shows that at both schools the overwhelming majority of critical situations belonged to the process category.We may therefore safely conclude that staff at either deems process related support activities most critical.As both schools surveyed are in transit from traditional forms of distance learning and face-to-face learning to their e-learning enhanced pendants, this conclusion is fully in line with the earlier reported literature finding that, upon the introduction of e-learning, the emphasis shifts from content related support to process related support (Hardless & Nulden, 1999;Beaudoin, 1990).Parenthetically, the high score of the fraud item is also in line with literature findings (Bennett, 2005).
As for the unimportant items and categories, one had better not infer too rapidly to their irrelevance, however.First, the method followed makes it hard to distinguish between items that are found genuinely unimportant and those that are formulated unattractively.The kind of interaction process that participants of the sessions are engaged in leaves little time for reflection and careful consideration.At least this implies that some items will attract few votes merely because of their opaque language or their being ill thought trough.Clearly, lack of votes then does not reflect lack of importance but, perhaps, difficulty exactly to pinpoint what the issue amounts to.
Second, an argument may be made that process problems are first-order problems in that they screen off other problems, particularly content related problems.A transition to e-learning, whether in a distance learning environment or more traditional learning setting, is fraud with difficulties, organisational, technical, pedagogical (cf.Sloep et. al., in press).Inevitably, students will experience some of these too.These difficulties, irrespective of whether they pertain to content, process or product, will primarily make themselves felt in the learning process.After all, it is while learning that students see themselves confronted with the imperfections of their learning environment.The argument thus is that staff is indeed confronted with process issues, but that this is at least in part a reflection of the transition process that they as a school are going through.
In summary then, when making the transition from traditional forms of learning to e-learning enhanced, blended forms of learning, one had better pay particular attention to process related demands for support as well as fraud prevention.However, content and product related support issues should not be forgotten about.They might well resurface once the transition nears completion and process issues have been resolved.
reveals that some items are deemed highly relevant: − Fraud in papers (ID 14) − Give feedback on the progress of students' work (ID 16) − Filter out 'repeated' questions (ID 12) − Portfolios of students should be easily accessible and well structured, so that staff can easily find what they are looking for and do not lose much time (ID 41), whereas others are seen as highly irrelevant: − Coaching of novice teachers: clarifying problems, generating alternatives (ID 7) − Facilitate putting together groups with specific characteristics and demands (ID 11) − Monitor complex group processes (ID 24) − Score assignments and papers (ID 27) − Support for writing of papers (ID 35) − How to help students who are in the final phase of their study organise their own support activities (ID 43)

Table 3
Critical situations categorised.The absolute frequencies of items for the categories content, process, and product.