How accurately do instructors judge students ' attitudes online ? A measurement of expectations and level of satisfaction with an Online Information Systems masters program

In order to run a successful educational program, instructors as well as staff members must constantly review and adapt to the expectations, concerns, demographics and satisfaction level of their student consumers. This study was conducted in order to examine these issues in an online educational setting. First, interviews were given to the program instructors in order to determine their opinions about the students' expectations and satisfaction levels. This information was then used to create a student survey that assessed the students' expectations and level of satisfaction. These two sets of results were then compared This comparison revealed that the online instructors did have a good grasp of the online students' expectations, concerns, demographics and satisfaction level. The only areas where the instructors' concepts of student views were slightly less accurate was student concerns and student feelings about the program administration, where the instructors overestimated the level of concern the students had about successfully returning to the learning environment and underestimated the students' satisfaction with the program's administration. This leads us to conclude that, even with the added online factor, instructors strongly understand student expectations, satisfaction levels, demographics and concerns.

email: machtl@umbc.edu,preece@umbc.edu In order to run a successful educational program, instructors as well as staff members must constantly review and adapt to the expectations, concerns, demographics and satisfaction level of their student consumers.This study was conducted in order to examine these issues in an online educational setting.First, interviews were given to the program instructors in order to determine their opinions about the students' expectations and satisfaction levels.This information was then used to create a student survey that assessed the students' expectations and level of satisfaction.These two sets of results were then compared This comparison revealed that the online instructors did have a good grasp of the online students' expectations, concerns, demographics and satisfaction level.The only areas where the instructors' concepts of student views were slightly less accurate was student concerns and student feelings about the program administration, where the instructors overestimated the level of concern the students had about successfully returning to the learning environment and underestimated the students' satisfaction with the program's administration.This leads us to conclude that, even with the added online factor, instructors strongly understand student expectations, satisfaction levels, demographics and concerns.

Foreword
David Squires was my colleague for almost twenty-five years.During that time I watched David's stature as an academic grow.David brought depth and rigour to the evaluation of educational software and in recent years he introduced techniques and concepts from

Introduction
Understanding student concerns, demographics, expectations and satisfaction levels is essential to running a successful educational program.Within any educational program, instructors as well as staff members must constantly determine, review and adapt to these student variables (Harasim, Hiltz, Teles and Turroff, 1995;Squires and Preece, 1996).This task, while difficult within itself, is made increasingly difficult when the educational environment is placed online and educators must concern themselves with the usability of the software as well as with content and pedagogy (Squires and Preece, 1999).The importance of this task, however, is not lessened by the online factor.
It is because of the importance of student consumers' expectations, concerns, demographics and level of satisfaction within an educational program that this study was conducted.A second reason for doing this study is that there is surprisingly little work reported that directly examines whether faculty and staff perceive students' satisfaction of online courses or lack thereof accurately.The main goal of this study is to compare the instructors' views of student expectations, concerns, demographics and level of satisfaction with the same variables for the students within the context of one online educational program.It was our hope that this type of analysis will be done for other programs in other educational settings, since it is obviously desirable that instructors should accurately interpret the concerns and feelings of their students.
The study occurs in three parts.The first part involves conducting instructor interviews and then analysing the results to identify what the instructors think are the students' concerns.The next part entails creating a survey based partly upon the analysis from part one and administering this survey to students within the program.The data collected from this section of the study is then analysed.The third and final part of this study involves the comparison of the students' data and the instructor data.

Experimental method
A set of open-ended interview questions was created in order to access the instructors' perceptions of student expectations and satisfaction level within the Online Information Systems Masters program.The questions were as follows: 1. What do you think the students expect from the program as a whole?
2. What do you think the students expect from your course(s)?
3. What are your general feelings about the students' level of satisfaction with the program as a whole?
4. What are your general feelings about the students' level of satisfaction with any courses that you have been involved in?
5. Is there any part of the program that you feel that the students are mainly satisfied with?
6.Is there any part of the program that you feel that the students are mainly dissatisfied with?

Participants
For this part of the study, seven out of the eleven course instructors were interviewed.However, two interviews had to be conducted through email due to the fact that these instructors were on vacation.All the instructors have extensive experience teaching masters level courses but one had not taught online before.

Research setting
Each interview took place in the instructor's office.Every session was recorded onto an audiotape and notes were taken.Each session lasted between fifteen minutes and one hour, with most averaging forty minutes.

Discussion of results
All instructors appear to have similar concepts of student expectations and levels of satisfaction.All perceived the students as being either satisfied or very satisfied with the program as a whole.Many aspects of student level of satisfaction were noted, such as the students getting quick and timely responses to email, finding the course Website easy to use, participating in many useful interactions, feeling that the course material and the course itself are useful, and appreciating the program's flexibility of time schedule and location.It was also noted that there is a very low drop-out rate for the program, which also supports the notion of the students being satisfied.Also, the instructors who have taught an online course within this program mentioned that they are satisfied with the experience, which often suggests that students are also satisfied.
Even though all instructors saw the students as being satisfied with the program, if not very satisfied, they did mention some aspects of the program that may lower the overall student level of satisfaction.These consisted of the students dissatisfaction with many administration procedures that were not completed as smoothly as expected, disappointment when course books and other needed materials did not arrive on time, frustration when their course enrolments were not correct or timely, dissatisfaction when program accounts were not set up fast enough, and when admissions procedures were executed at what appeared to be the last minute.
In conjunction with the view of students generally being satisfied, the instructors also saw the students as having most, if not all, of their expectations met or surpassed.It was noted during the interviews that many different expectations are involved within each student's perception of this program.Also, while students may not share all expectations, many of the students will approach the program with quite similar expectations.Specifically, most instructors indicated that they believed the students to have the following expectations: • to be able to enhance the skills that they already have as well as to acquire new skills; • to gain a basic introduction and familiarization with IT terminology and tools; • to enhance further the students' placements within the job market or to assist with career advancement; • to gain enrichment, to be able to get assistance from instructors and other staff members when assistance is needed; • to pass course hurdles without excessive trouble and without wasting time, to receive as much support as possible; • to complete the course fully without wasting too much time; • to find that the course work is useful instead of wasteful, that an online course should be no harder to complete than an equivalent face-to-face course; • to have as much student-teacher and student-student interaction as possible; • to receive a degree at graduation that has as much status as the equivalent degree from a face-to-face program, to be able to do everything from a distance; • to be given all needed materials aiid clear instructions of what is expected from them, for the administration procedures to run smoothly, a learning environment filled with other committed and competent students; • to be supplied with thorough lecture notes; • to be given sufficient guidance on all projects; • to receive feedback on assignments and work; • to gain an in-depth and wide understanding of current IT; • to learn about industry-related topics so that they can apply them to real-life scenarios, to have the administrative procedure be almost invisible; • to have the teachers stick to the syllabi; • to have all accounts set up and ready for use before courses begin; • to have instructors keep their promises; • for the courses to be set up on a flexible time schedule; • to gain hands on or applied experiences.
Instructors also had similar concepts of the demographics of the student population.Most indicated that the majority of the students had some level of computer skills prior to entering the program and have been working in the industry for a while.It was also believed that almost all of the students are returning students, many of whom are older and lived out of state.The instructors also believed that most students did not have higher education degrees, especially within high-tech fields.
Some students' concerns were also noted during the interview process.Getting back into the learning environment and succeeding in it was noted as a major student concern by most of the instructors.Many also saw the online factor, which is new for most students, as being a cause for student concern.

Limitations
Even though all efforts were made to strengthen the soundness of the research design, methods, and findings, some limitations need to be noted.First, even though a 64 per cent response rate was achieved for the interview participants, the rate was not 100 per cent because not all instructors were able to participate in the interview process.The response rate was more than ample for this kind of research and a good deal of information was collected.Another limitation lies in the fact that two of the instructors were interviewed through email.While this could not be helped, since the instructors were on vacation and our goal was to collect as much information as possible, the research setting could have been controlled more if these interviews had been conducted in person.Also, when interviews are conducted through email or other distance mediums, information such as body language and reactions is lost.Having such information for these two instructors could have been useful.
While a sufficient number of interview questions were presented to each instructor, the questions were quite vague in nature.The questions were designed in this way because very little prior research has been done in this area and there was not enough information that could have been used to make the questions more specific.However, if time had permitted, it might have been beneficial to interview each instructor a second time with a more specific set of questions, based on the answers from the first set of instructor interviews.This way each instructor could have supplied us with information on each possible expectation and notion of satisfaction.

Survey creation
The findings from the first part of this study were used to create a survey that was administered to the online students.The survey contained two parts.The first section of the survey contained a measure of student demographics, a general measure of student expectations as well as whether they are being met, and a single overall measure of level of satisfaction.
The questions that measured student demographics were based on the instructors' perceptions of student demographics, provided within the interview process.Other common demographic measures, such as age and gender, were added to the question list as well.These questions were presented in Likert, open-answer and check-box form.The purpose of these questions was not only to gain an understanding of the demographics for the students, but also to be able to evaluate how well the instructors' views of student demographics matched actual student demographics.
The first part of the survey also dealt with general measures of program expectations and whether they were being met.Within this part of the survey, the survey user was first asked through an open-answer question to list any of their expectations that pertained to a specific aspect of the program (such as course software, admissions, instructors, material, etc.).Then, the user was asked to indicate, through the use of a check box, whether this expectation had been met.Lastly, in open-answer form, the user was asked to discuss why the above answer was given.The purpose of these questions was to gather a list of expectations as determined by the students.This list could then be compared with the list of student expectations provided by the instructors.These questions also enabled us to gather an unbiased list of student expectations that was not influenced by any of the expectations that the instructors had already listed.
The end of the first part of the survey contained a general level of satisfaction measure where the students could indicate if they were very dissatisfied, dissatisfied, neither satisfied nor dissatisfied, satisfied, or very satisfied with the program.The students were also asked to provide information on why they chose the above answer.The purpose of this question was to gain a general and almost baseline measure of level of satisfaction from each participant before they were asked to think about more specific aspects of satisfaction and expectations.The second part of the survey consisted of questions that accessed either an aspect of satisfaction or an expectation that had been mentioned by one or more of the instructors during the interview process.
All of the questions within this section of the survey were in 'I statement' form.Each question also had answer choices ranging from 'very dissatisfied' to 'very satisfied'.These five answer choices are presented to the survey user within a Likert scale format.
It is important to note that, once survey users had moved on to the second part of the survey, they were not allowed to change their answers to the questions from the first part of the survey.The survey was purposely designed this way in order to make sure that the integrity of the expectation and overall level of satisfaction questions within the first part of the survey was not jeopardized.In other words, we did not want survey users to base their answers to the questions within the first part of the survey on the expectations and aspects of satisfaction that are mentioned within the second part of the survey.In order to make sure that this did not occur, it was necessary to make sure that the answers to the questions within the first part of the survey could not be accessed by the survey users after they had viewed the questions within the second part of the survey.

Participants
All sixty-nine students on the Online Information Systems Masters Program were contacted through email and asked to complete the online survey.Twenty-eight students filled out the survey.However, five of those surveys had to be discarded because the answers were either incomplete or were received too late after the deadline.The total number of usable surveys was twenty-three, giving a response rate of 33 per cent.

Research setting
The survey was available online, and was completed by the students in their own time and therefore it was impossible to control the environmental conditions while the students were completing the survey.The online survey consisted of two separate surveys connected by a link.Survey users could complete each survey part at leisure, although most completed both parts within a one-hour period.

Discussion of results
Much student demographic data was collected throughout the survey process.It was found that the majority of the online students are males (68 per cent) between the ages of 41 and 50 (55 per cent) or 31 and 40 (31 per cent).The majority of the online students also live within the United States (95 per cent), mainly outside Maryland.
Most online students within this program have a strong background in computer use as well as computer structure and operations (77 per cent).The undergraduate areas of study for these online students are very diverse, ranging from Chemistry and Engineering to Psychology and Education, with the majority of the areas of study not falling within a computer-related field (90 per cent).Most online students reported their highest level of degree earned to be a Bachelor of Science degree (60 per cent), with the next most popular level of degree being a Bachelor of Arts (23 per cent).In conjunction with that, most online students indicated that they did not hold any computing degrees or certifications prior to enrolling in this program (64 per cent).For those who did have computer degrees or certifications, their types were diverse, ranging from Information Systems Management and an MCSE certification to a BS in Computer Programming and CNE (Novell) certifications.The year that each online student had last taken a college level course varied, with the majority having been out of the learning environment for at least four years (58 per cent).
Online students within this program also varied in their current occupations.Many had only spent between two and six years working within their current job (50 per cent).However, most had spent either between eleven and twenty (38 per cent) or twenty-one and thirty (28 per cent) years working within the job market.
Online students within this program seemed to be divided on issues pertaining to program concerns prior to beginning classes.A small majority of students indicated that they were concerned with getting back into a college learning environment successfully (45 per cent), although many did not think about it (18 per cent).A slight majority also noted that they were at some point concerned with the fact that the program is offered strictly online (55 per cent), and no students indicated that they did not think about this possible concern.
Through this survey process, online students also indicated what expectations they had for the program and its parts.In order to do so, online students were first asked to list their expectations.Students noted such expectations as that all materials should be supplied on time, communication with the instructors should be often and thorough, the course work load should be time-and location-flexible, professors should be knowledgeable, administrative staff should be friendly, organized and helpful, and that the degree received at the end of this program should be valuable.The large majority of online students also indicated that they felt as if their expectations were being met.
The online students were then asked to rate their overall level of satisfaction with the program as a whole.Using the Likert scale mentioned above, the average level of satisfaction rating was a 4.04, which means that the online students were satisfied or very satisfied with this program.
Next, online students were asked to rate each expectation within the provided expectation list that was collected from the instructor interviews.A reliability analysis was run and alpha was shown to be 0.90.According to this statistic, each question or expectation appears to measure the concept of general online student expectations without being redundant.A frequency analysis run with this data shows that the average score for each of these expectations was a 4.03 or 'agree'.In other words, the online students agreed that the student expectation list that the instructors created reflected their own expectations.Within this expectation list, the expectation score ranged from 3.03 or 'neither agree nor disagree' to 4.73 or 'strongly agree'.The expectation receiving the lowest rating of 'neither agree of disagree' was that 'Courses should not take up too much of my time'.Some other expectations that received the same rating were 'I expect that I will interact with my peers on a regular basis throughout the course of this program', 'I assume that through this program I will be able to gain hands on experience', 'I expect that this program will help me to further enhance my placement within the job market', and 'I anticipate that it will be easy to pass the hurdles within this program'.The expectation with the highest rating of 'strongly agree' was 'Instructors do have to provide relevant feedback'.Other expectations that also received some of the highest ratings were 'I assume that I will find the work assigned to me within my courses to be useful', 'Instructors and other program staff members should provide me with help when it is needed', 'Students should receive guidance from their instructors on projects', and 'I assume that my program credits and grades will be properly recorded'.
Finally, the online students noted their ratings of the aspects of satisfaction.A reliability analysis was run and alpha was shown to be 0.92.According to this statistic, each question or aspect of satisfaction appears to measure an aspect of general online student level of satisfaction without being redundant.A frequency analysis run with this data shows that the average score for each of these expectations was a 3.83 or 'agree'.In other words, students seemed to agree that they were satisfied with the aspects of satisfaction as a whole.
The scores on each aspect of satisfaction ranged from 3.17 or 'neither agree or disagree' to 4.26 or 'agree'.The aspect of satisfaction that received the lowest rating of 'neither agree or disagree' reads 'I believe that this program does not demand too much from its students'.
Other aspects of satisfaction that received the same rating are 'The program courses provide me with just enough information on each topic', 'Instructors and other program staff members are providing me with the proper assistance that I need', and 'I expect that I will not interact with my peers on a regular basis throughout the course of this program'.
The aspect of satisfaction that received the highest rating or 'agree' reads: 'I am pleased with the program administration.'Other aspects of satisfaction that also received this rating are 'I find the courses within this program to be interesting', "The materials within each course seem appropriate', and 'This program is helping me to learn new skills'.

Limitations
Even though one of the main goals of this research process was to strengthen the accuracy of this study's design, methods and findings, some limitations still need to be noted.First, although a response rate of 33 per cent for this type of research is quite acceptable, it would have made this study's findings stronger if the response rate had been higher.
Another limitation stems from the fact that the subjects were unsupervised and in a location of their choice when filling out the survey.Because of this fact, subjects may have been distracted at times or may have rushed over parts of the survey without caution in order to finish quickly.Both of these situations could have lessened the strength of our findings by leaving the researchers with missing and/or incomplete data or with data filled with mistakes.Even though it is impossible to tell whether or not any of this undesirable data can be found within this study (it is assumed that if present such incorrect data will be outweighed by correct data during the analysis), it would have improved this study if the possibility of such undesirable data could have been eliminated.

Comparison of results
It is apparent that the instructors within the Online Masters Program have a strong and truthful concept of student expectations and satisfaction level, as well as student concerns and demographics.This was determined by directly comparing the student responses to the survey and the instructor interviews.Even though the information collected from the online survey technically applied to the student sample and not the student population as a whole, one can assume that they generally reflect the demographics within the whole student population.It is based on this that we can make a comparison.
The first thing that we compared was the instructors' concepts of student demographics and actual student demographics as collected from the survey process.The instructor views of student demographics reflected the collected students' demographic information in many respects, such as noting that the majority of students within the program are students returning to the learning environment from an educational break, are older students falling within the 31 to 50 years old range, having some level of computer skills before beginning this program, living outside the state of Maryland, working within the industry for a number of years, and not having higher educational degrees, especially within a computer-related field.From this comparison it is apparent that the instructors have an excellent grasp of online student demographics.
Student concerns were then compared with the instructors' concepts of student concerns.Instructors noted that they believed the student population to be concerned with successfully getting back into the learning environment and overcoming the differences and difficulties that the online factor can add to learning.Through the survey process, it was learned that online students seem to be very divided on these issues.Most online students did note that they were nervous about how the online factor will affect their learning experience, but a fair number of online students also noted that they were not concerned about that at any point.The majority of the online students indicated that they were not concerned with getting back into the learning environment, with a few students even noting that they did not even consider this concern at all.It can be concluded from this section of the analysis that the instructors did not have a perfect concept of student concerns.
Next, we compared instructor concepts of student expectations with the student-given expectations.This part of the procedure actually involved two steps: comparing the list of students' expectations collected from the instructors with those collected by the students in order to make sure that the list of expectations used within the second part of the survey was complete; and analysing the answers to the expectations within the second part of the survey in order to make sure that the listed expectations truly reflected those of the student population.
The expectations that the students listed on their own turned out to be quite similar to those student expectations that the instructors had previously listed.The few student-given expectations that were unlike those provided by the instructors were irrational, such as wanting to receive the degree within a year and wanting to do very little to no work for the degree, and therefore discarded.The students also appeared to agree that the expectations that the instructors came up with reflected their own.This can be seen by the average score of 4.03 or 'agree' that the online students assigned to the instructor expectations list.As mentioned above, there are some expectations that receive a higher score of 'strongly agree' such as, 'Instructors do have to provide relevant feedback', 'I assume that I will find the work assigned to me within my courses to be useful', 'Instructors and other program staff members should provide me with help when it is needed', 'Students should receive guidance from their instructors on projects', and 'I assume that my program credits and grades will be properly recorded' while some expectations received a lower score of 'neither agree or disagree', like, 'Courses should not take up too much of my time', 'I expect that I will interact with my peers on a regular basis throughout the course of this program', 'I assume that through this program I will be able to gain hands on experience', 'I expect that this program will help me to further enhance my placement within the job market', and 'I anticipate that it will be easy to pass the hurdles within this program'.It is also important to note that no instructor-given positive expectation received an overall score below a three ('neither agree or disagree').All of this together suggests that the instructors understand student expectations well.
Our next step was to compare aspects of student satisfaction.This was accomplished by analysing the scores that the online students assigned to the aspects of satisfaction that were supplied by the instructors.The average score for the aspects of satisfaction was a 3.83 or 'agree'.As mentioned above, some aspects of satisfaction, such as, 'I believe that this program does not demand too much from its students', 'The program courses provide me with just enough information on each topic', 'Instructors expect that I will not interact with my peers on a regular basis throughout the course of this program', received a lower rating of 3 or 'neither agree nor disagree'.Other aspects of satisfaction received a higher rating of 4 or 'agree', such as, 'I am pleased with the program administration', 'I find the courses within this program to be interesting', 'The materials within each course seem appropriate', and 'This program is helping me to learn new skills'.It is also important to note that no aspect of satisfaction received an overall score less that 'neither agree or disagree' or higher than 'agree'.In other words, students seemed mainly to agree that they were satisfied with the aspects of satisfaction that the instructors noted, suggesting that the instructors do have a fairly good grasp of student concepts of level of satisfaction.
There were a few aspects of satisfaction that the instructors believed that the students were very unhappy with, such as the administration and their procedure, the arrival of books and the timely creation of accounts.While the students agreed that they were disappointed with the arrival of books and the timely creation of accounts, they noted that they were very pleased with the program's administration.The discrepancy between the students' satisfaction with the program administration and the instructors' concept of that satisfaction leads us to believe that while the instructors have a very good grasp of satisfaction within parts of the program that relate to their teaching experiences, perhaps they do not have the same strong concept of satisfaction when it comes to aspects of the program that they are often not involved with.The majority of the instructors noted that they believe that most students were concerned about successfully returning to the learning environment and about succeeding within an online course.
Instructors believed that the majority of students shared many expectations, such as that instructors have to provide relevant feedback, the course work will be useful, instructors and other program staff members will provide assistance, and that program credits and grades will be properly recorded.
Instructors indicated that they felt that the majority of students shared the same aspects of satisfaction, such as being pleased with finding the courses interesting, seeing the course materials to be appropriate, and learning new skills.Instructors thought that the students were less satisfied with the program administration.
Instructors noted that they believed that the students were either satisfied or very satisfied with the program as a whole.
The average student response matched those suggested by the instructors.
Student responses reflected that the majority of students were concerned about the online factor, but were not worried about getting back into the learning environment The majority of students strongly agreed with the expectations that the instructors suggested.
Most of the students agreed with the instructor-determined aspects of satisfaction but they were more satisfied with the program's administration than the instructors realized.
Students indicated that the majority of them were satisfied with the program as a whole.

Table I: Summary of results
Lastly, we compared overall level of satisfaction.During the interview process, all instructors indicated that they perceived the online students as being very satisfied with the program as a whole.However, while most students did indicate that they were very satisfied, the overall score for level of satisfaction was only a 4.04 or a score of 'satisfied'.This difference indicates that, while instructors had a good concept of student level of satisfaction, their concept was not flawless.However, the overall level of satisfaction score and the scores for aspects of satisfaction did correlate significantly, suggesting that the instructors were not too far off.It is a strong possibility that these findings might merely reflect the fact that the list of aspects of satisfaction given by the instructors is just not complete.Table 1 summarizes these reports.

Conclusions and future work
From this study it can be seen that overall the instructors working within the Online Masters Program possess a very good understanding of online student expectations, level of satisfaction, and demographics.They also possess a good but not flawless understanding of student concerns and aspects of satisfaction dealing with parts of the program where the instructors are not often involved.Instructors and students both appear to be satisfied with the program itself.Students also feel that the program is meeting a good majority of their expectations.
The online factor has not negatively affected the ability of instructors and other staff members to determine, review and adapt to the expectations, concerns, demographics and satisfaction level of their students.This is important to determine, since the online factor eliminated face-to-face communication, which was the most common way that instructors and other staff members were informed of student expectations, concerns,-demographics and satisfaction levels.
For future work in this area, it would be beneficial to all education institutions, especially those with unique factors such as the online element, constantly to assess the level of satisfaction, concerns, demographics and expectations of their consumer students.It is only after knowing this that any program can expect to be the best that it can be.Such selfreflection is particularly important for online programs (Hiltz, 1994) in which it is only too easy to fall into the trap of 'just putting material on the Web' with little thought about pedagogic structure.Approaches like this can all too easily lead to digital diploma mills (Noble, 1998), particularly when profit forms part of the underlying motive as it has done for many online programs.
For this and many institutions it will be beneficial to repeat this type of study in the near future.This is due to the fact that levels of satisfaction, concerns, expectations and demographics are not constant elements within an institution.In order for this program to be able to continue satisfying its consumers it needs to be able to continue to know what they want from the program and if they feel that their needs are being met.This is true for any kind of program but it is particularly so for Masters programs in Information Technology where the curriculum changes rapidly, driven partly by technical developments and partly by market forces.