A Comparative Analysis of the Effects of Instructional Design Factors on Student Success in E-learning: Multiple-regression versus Neural Networks

This study explores the relationship between the student performance and instructional design. The research was conducted at the E-Learning School at a university in Turkey. A list of design factors that had potential influence on student success was created through a review of the literature and interviews with relevant experts. From this, the five most import design factors were chosen. The experts scored 25 university courses on the extent to which they demonstrated the chosen design factors. Multiple-regression and supervised artificial neural network (ANN) models were used to examine the relationship between student grade point averages and the scores on the five design factors. The results indicated that there is no statistical difference between the two models. Both models identified the use of examples and applications as the most influential factor. The ANN model provided more information and was used to predict the course-specific factor values required for a desired level of success. Introduction For the past few decades, we have witnessed the widespread application of technological developments in the educational systems. The integration of education technologies with communication technologies introduced e-learning as an alternative or complementary approach for the traditional education methods that are subject to time and location constraints. The necessity of having easier access to a wider range of information, coupled with the demand for effective and low cost education, paved the way for the increased use of the Internet in learning environments. One of the challenging problems that society has been facing is the need to improve the quality of e-learning systems. A number of studies have been conducted to gain a better understanding of the factors that affect the success of e-learning systems and the interrelations of these factors. The literature can be categorised into three main groups, namely, student and


Introduction
For the past few decades, we have witnessed the widespread application of technological developments in the educational systems.The integration of education technologies with communication technologies introduced e-learning as an alternative or complementary approach for the traditional education methods that are subject to time and location constraints.The necessity of having easier access to a wider range of information, coupled with the demand for effective and low cost education, paved the way for the increased use of the Internet in learning environments.One of the challenging problems that society has been facing is the need to improve the quality of e-learning systems.
A number of studies have been conducted to gain a better understanding of the factors that affect the success of e-learning systems and the interrelations of these factors.The literature can be categorised into three main groups, namely, student and instructor characteristics (Bozarth, Chapman, and LaMonica 2004;Greene et al. 2004;Hillsheim 1998;Kerr, Rynearson, and Kerr 2006;Liaw, Huang, and Chen 2007;Marom et al. 2003;Soong et al. 2001), instructional design factors (Bozarth, Chapman, and LaMonica 2004;Wegner, Holloway, and Garton 1999), and demographic variables (Diaz 1999;Diaz 2000;Dille and Mezack 1991;Hillsheim 1998;Lynn 2002;Marom et al. 2003;Muse 2003).
Of these success factors, good instructional design is seen as a necessity for the establishment of efficient learning environments.Instructional design factors can be defined as the optimum design of the instructional environment to meet the requirements of both the students and the instructors (Qureshi 2004).Instructional design, from the Web-based learning perspective, is a three-dimensional process involving designers, students, and instructors (Wegner, Holloway, and Garton 1999).The designers provide the students with the appropriate course content through working collaboratively with the instructors.Fresen and Boyd (2005) conducted a study regarding the critical success factors of Web-based learning systems and they synthesised a list of success categories and associated factors from several e-learning resources.For the category of instructional design, they identified 15 factors, which it is possible to divide into two groups: content design-related factors and other instructional design factors (Table 1).The first group of factors is more about the interface design aspect of e-learning contents while the latter group is associated with the usability of content and student interaction.
The literature already offers a range of recommendations for the first group of interface design factors, also known as content development standards, concerning issues such as the design of Web interface (Rollins 2002), texts (Ardac and Unal 2008;Bullock, Geraci, and Bear 2003;Ipek 2001;Lee 1999;Qureshi 2004;Pomales-Garcia and Liu, 2006) multimedia applications such as graphics (Angeli and Valanides, 2006;Jennings 2001;Qureshi 2004;Swan 2001), animations, video files, and colours (Angeli and Valanides 2006;Bhowmick et al. 2007;Bullock, Geraci, and Bear 2003;Ipek 2001;Jones 2002;Paul 2001;Qureshi 2004), practical exercises and illustrative examples (Jennings 2001;Paul 2001).We already know that animations should be designed to be interactive, complementary, and supportive to the text documents (Bullock, Geraci, and Bear 2003;Ipek 2001), but that the excessive usage of graphics and animations may distract the attention of the students from the course (Ipek 2001).This study will assess the relative importance of such factors on student performance.

Factors related to content design
Other instructional design factors

Methodology
The success of a learning system depends on the level of efficiency in which students can grasp the contents of their courses.Since measuring this efficiency may not be easy to formulate, grade points are seen as indicators for students' success.The aim of this study is to establish a relationship between instructional design factors and the student success level.This relationship is investigated by making use of statistical and neural networks techniques.This section will describe issues regarding the variable selection, data collection and sampling, and data analysis.

Creation of criteria
Having considered the instructional design factors particularly related to content design from the relevant literature, a list was created of 13 variables that are thought to be have an effect on student success.These variables were emailed to 20 interviewees a week ahead of a semi-structured individual interview.Interviewees were experts who had been working on the project for more than five years.Eighty percent of the participants had an academic background in instructional design while the rest mostly had a design-oriented background.The email included an invitation for the semi-structured interview together with information about what they were asked to carry out before attending the interview.A text file including some explanation of the variables and their brief description was also attached.The main aim of the email communication was to ask the interviewees to consider the suitability of those variables as instructional design criteria.
During the interview, the interviewees were also allowed to combine or eliminate some of the variables to create a reduced set.All possible instructional design factors suitable for the current project were extracted.The decision criterion for selecting those factors was based on the major agreement of the participants for each criterion.Of these 13 variables, two were eliminated by all the participants and the rest were reduced to five factors.
At the end of the interview stage, the following criteria were selected as predictors of success: the length of the texts, the usage of examples and applications, exercises, quiz and homework, multimedia applications, and visual integrity.These criteria were utilised by the select models as independent variables, while the average grade of students in different courses was used as a proxy for student success.However, in the second implication of the artificial neural network (ANN) model, a different operationalisation was employed, which will be presented later.

Sample
The data used to perform the analyses were collected from the e-learning school of a university in Turkey.The school consists of five different study programmes with more than 100 courses, serving more than 1000 students in total.Only one study programme is business-related while the rest of them are technically oriented.
A sample of five courses was selected from each programme for the data analyses, giving a total of 25 courses.Each of the five courses is programme-specific and considered as representative for the corresponding programmes.The sampling process ensured that the courses chosen have different types of content covering a wide range of subjects and the sample includes courses with a variety of difficulty level for each specific programme.More than 100 students were enrolled in each course.

Data collection
The data collection had two parts.The first was conducted to obtain the instructional design scores.The courses were evaluated by the experts on each of the selected instructional design factors.Second, students' grade point averages were collected for the corresponding courses directly from the student registration office of the university.
The first part of the data was collected from the experts in the form of a survey.The survey was used to obtain the scores on the five instructional design criteria, for each of the 25 courses.The experts were asked to evaluate each course based on the selected criteria by giving points on a scale between 0-10.The average grade was based on overall 100 points.The results of these evaluations as an average of all opinions (for five random courses) can be seen in Table 2.In Table 2, the first column shows the codes which were developed for the purpose of differentiating the courses.
The length of the text was compared to already determined standards.The related literature suggests that the optimal length of the texts should be 8-10 pages, given that anything between six and 12 pages is acceptable (Bullock, Geraci, and Bear 2003;Ipek 2001).The length of the texts was categorised into three classes and for each of them a ranking point was assigned.Conversion of the total average rankings into the 10-point scale provided the final results for this variable.
Examples and applications, and exercises and homework were rated by dividing the number of weeks where these factors were demonstrated by the total number of weeks in the course.Regarding the usage of multimedia applications, this factor was assessed based on the usage of graphics and animation.Two usage categories were considered and the rankings corresponding to that categorisation converted to an average usage of both animation and graphics, which was taken as a final point value.
The experts evaluated visual integrity intuitively as an average of the evaluations given to the quality of colour competence, the inclusion of multimedia applications and the texts in the whole content, and the errors occurred in the graphics and animations used in the texts.
The final column represents the weighted average of the final marks of homework, quiz, and final examination for a given course.The percentage of final examination is 80% while homework and quiz marks take 20% overall.The study assumes that the questions that were asked in the quiz and final exam were extrapolated from the course content.

Data analysis
The relationship between the criteria and the success of students was analysed using two different prediction methods, a multiple-regression analysis technique and an ANN model (multi-layer perception).These prediction methods have already been used in several comparative studies in different areas (Baker and Richards 1999;Gaudart, Giusiano, and Huiart 2004;Guiterez-Estrada et al. 2004;Hardgrave, Wilson, and Walstrom 1994;Hyvarinen and Bingham 2003).Recent studies have considered ANN as an alternative method of traditional data analysis techniques or perhaps a more-developed version of these techniques.Although the solution of an ANN may not be the optimum one, its results can be considered rather satisfactory as compared with other traditional methods (Hertz, Krogh, and Palmer 1994;Oztemel 2003).
The ANN prediction models used in the current study are presented below.The models' network typology consists of three layers, namely input, hidden, and output layers.In the first model, the input and the hidden layers have five nodes each while the output has one node only (see Figure 1).The second model's typology consists of one input node, one hidden node, and five output nodes (see Figure 2).

Numerical illustrations
This section provides some illustrations with regards to the implementation of the two prediction models on the dataset as well as the associated results obtained from the analyses.

Multiple-regression model
After employing the regression analysis, the coefficient of determination (R 2 ) was found to be 0.8067 and it indicated that the five criteria used in the model can explain 80.67% of average grade points of all courses.The resulting F-value, which was 15.859 with a probability of 0.00003, leads us to reject the null hypothesis (H 0: The regression coefficients of the model are all equal to each other that have a value of zero).Furthermore, a t-test analysis was performed within a 95% level of confidence to check whether the regression coefficients were significant or not.The results of the t-test can be seen in Table 3.In the table, X 1 , X 2 , X 3 , X 4 , and X 5 were used to refer to the length of the text, the usage of examples and applications, exercises and homework, multimedia applications, visual integrity respectively.
As it can be seen from the Table 3, all coefficients are statistically significant.Therefore, the regression model can be as follows: The expected and the obtained values were compared using a t-test, and it was concluded that there is no statistical difference between them (t = −0.006,p = 0.99).The results of the regression analysis indicated that the five instructional design factors are related to students' success.It can also be concluded that the most effective factor is the usage of examples and applications.

Multi-layer perceptron models
The ANN prediction model was designed in two different ways.It can be applied in a way similar to that of the multiple-regression model, which considers the instructional design factors as input and then tries to predict average grade points.It can also be applied inversely, where the average grade points are used as an input and therefore ending with the instructional design factors as an output.In the following sections, the implementation of these models on the selected data will be demonstrated.

Feed-forward prediction
The first ANN model predicts the average grade points via the multi-layer perceptron method.Hence, the dataset was scaled within the interval of 0-1.The dataset was divided into three sub-sets, the training set, the test set, and validation set.Accordingly, the data collected from the 25 samples were divided into 10 training, five test, and 10 validation sub-sets.For homogeneity purposes, the dataset was divided over the courses in the three divisions.Given the unavailability of a large sample, the error value was selected as high as 0.07.Table 4 shows the prediction input sets, the expected outputs, the predicted outputs and the error percentage, which represents the difference between the prediction results and the expected values.After a series of simulation studies, the network reached 100% learning and 100% test success in the 100000th iteration with a learning coefficient of 35% and a momentum coefficient of 75%.In order to evaluate whether there was a significant difference between the expected and the predicted values, a t-test analysis was performed.The t-test value (t = 0.04, p = 0.96) revealed statistically significant results.
It is possible to see from Table 4 that the predicted values are close to their corresponding expected values.Only for the second and the fourth cases, the error values are slightly high.A reasonable explanation for this could be that the sample size for training was not large enough.Even though sample size for training was not sufficient, the worst prediction result of the network had 7.8% of error value.Hence, it can be said that multi-layer perceptron model was performed successfully.

Back-forward prediction
In this part of the study, the second network model uses the first model's inputs as output and ends up with the inputs of the previous model.When the previous output values were used as input, the network was able to produce the instructional design factors' values for a desired interval value of an average grade point.Similar to the first network model, a 0.07 error value was selected.The network trained had a 100% of learning success and 35% and 75% of learning and momentum coefficients respectively.With these parameters, the network produced the results as shown in Table 5.
In general, the model produced very close results to the expected values, except for output 3. The reason for the high difference between the expected and the network output is that output 3 had a wide range of value varieties, which caused a learning problem in the network and an explanation difficulty.
The back-forward prediction model provides insights about the type of instruments that should be included in the courses.For a desired success level, a reasonable combination of the instructional design factors levels can be predicted.

Discussion
This study investigated the effects of instructional design factors on students' success using multiple-regression and ANN approaches as prediction models.These models were developed and compared using the same prediction dataset.The results indicated that all of the five factors have positive effects on students' success.Additionally, the multimedia applications and the visual integrity factors have strong relationship with the success of the students, and the usage of examples and applications is the most effective among all factors.By means of the multiple-regression and the feed-forward multi-layer perceptron model, a decision concerning the predicted level of success of a course can be made before displaying the course online.For example, necessary improvements for a course can be taken into consideration during the pre-displaying process.These models can be used as a strategic tool to decide whether to present a course online or not as well as which aspects of it should be improved.Table 6 summarises the results of both models and the comparison between them, along with their expected outputs based on the error values for the dataset.
A paired t-test analysis was conducted to evaluate whether the results of two models were different from each other or not.According to the t-test results, (t = 0.696, p = 0.504), there was no statistical difference between the two models.
However, it should be noted that the results of the second ANN model (backforward) cannot be possible to be produced by the multiple-regression model.In this manner, ANN models can be chosen instead of a multiple-regression approach due to their capability of giving different aspects of this kind of problem.The back-forward neural network prediction model stimulates us to look at the answers for the question of what design structure a course content should have to make the learning outcomes adequate for the students to lead them to success.
The contribution of this study is two-fold.By introducing the application of ANN to the design of learning environments, it might be possible for the researchers to introduce new problem domains which utilise the capability of this technology.With regards to the contribution of the study for practitioners, the study brings out the implications for an ongoing e-learning system.To be able to reach the highest success level on student performance, several simulation studies can be performed by investigating the effects of different success predictors, not just instructional design factors, on students' success.The simulation could be performed based on the feed-forward neural network model.Furthermore, the simulation can be expanded through the usage of back-forward network model by including the attributes of the learners together with the other success measurements.As a result of this, an appropriate design structure can be tailored for each group of learners.

Conclusion
Even though the success of an e-learning system is explained by factors such as the number of students and lecturers in the system, their learning and teaching efficiencies, and education technologies used, students' success level is another significant indicator.Students' success level is affected by socio-economic, demographic and instructional design factors.In this study, only instructional design factors were investigated.As a result of the study, it is recommended for instructional designers and lecturers to prepare course contents within a specified standardisation and to make efficient use of multimedia applications.Furthermore, visual design of a course should always be taken into consideration.During this research, uncontrollable variables such as socio-economic and demographic factors, which are indirectly related to students' success, have not been considered.This remains a limitation of the study.For future work, a statistical analysis of how these factors affect the students' success can be investigated along with the investigation that has been done in this study.Furthermore, as an indicator of success, not only grade point averages but also some other success measurements after graduation should also be included in further studies.

Figure 1 .
Figure 1.Typology of feed-forward neural network model.Figure 2. Typology of back-forward neural network model.

Figure 2 .
Figure 1.Typology of feed-forward neural network model.Figure 2. Typology of back-forward neural network model.

Figure 1 .
Figure 1.Typology of feed-forward neural network model.

Figure 2 .
Figure 2. Typology of back-forward neural network model.

Table 2 .
Scores for each course for the five instructional design factors.

Table 4 .
The differences between the expected and predicted results for the first multi-layer perceptron model (five inputs, one output).

Table 5 .
The differences between the expected and predicted results for the second multilayer perceptron model (one input, five outputs).

Table 6 .
Comparison of regression and ANN models.