An inferior source ? Quantitatively analysing the production and revision of five technology-enhanced learning-related terms on Wikipedia

Research in Learning Technology 2018. © 2018 M. Flavin and K. Hulova. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.


Introduction
This article analyses the production and revision of five Wikipedia entries relating to technology-enhanced learning, compared with the publication of academic journal articles on the same topics.The article draws its understanding of technology-enhanced learning from the Universities and Colleges Information Systems Association's definition of the term, encompassing in-house, commercial or open source tools (UCISA 2016).It focuses on the following five terms: virtual learning environment (VLE), Second Life, massive open online course (MOOC), learning analytics and bring your own device (BYOD).The article does not aim to evaluate the intrinsic quality of the Wikipedia entries studied but to look at the volume of activity on Wikipedia and in academic journals, aiming to evaluate aspects of the value of Wikipedia as a source to support learning and teaching.
Wikipedia is open source (UCISA 2016), 'the largest free reference work ever created' (Konieczny 2016(Konieczny , p. 1531) ) and 'an icon of the contemporary internet age' (Okoli et al. 2014).Since it first appeared in 2001 it has become an established technology for formal and informal learning, as well as for entertainment.Unlike academic journal articles, Wikipedia does not feature original research (Xiao and Askin 2014) and thus the two are different forms of publication, but while the latter is published for an academic audience, the former is, nonetheless, used plentifully by students and lecturers.Wikipedia is often regarded as an inferior or a dubious source, but this paper argues Wikipedia may be more useful than academic journal articles in certain respects, especially in relation to information on new and emerging technologies and practices.
The specific questions addressed in this article are as follows: -How does the pattern of publication on Wikipedia relate to the pattern of academic journal publication, for the five technology-enhanced learning-related terms examined?-Are Wikipedia entries and their histories more useful than academic journal articles as a barometer of interest in technologies?
The questions will be examined via two 2-way analyses of variance with standardised Wikipedia article lengths, yearly edits and the number of academic journal articles published, as indicators of the popularity of the terms.The variables of each ANOVA are described in Table 1.The authors hypothesise that the interaction and year terms will be significant in both analyses: it is expected that, after Year 5, the number of journal articles will still be rising, while the Wikipedia edits will have peaked and started to drop, and article lengths will be stable after the edit peak.Such a finding would support a hypothesis that would answer the second question: Wikipedia entries could be a more useful barometer of interest than academic journals because Wikipedia activity peaks early, providing a substantial body of information.
The research for this article is relevant because of the extent to which Wikipedia is used by students and lecturers, yet Wikipedia is often regarded as a dubious, inferior source, because it is not subject to peer review and editorial oversight.

Literature review
The use of Wikipedia by students in higher education is widespread.Head and Eisenberg's (2010) research showed 82% of their undergraduate respondents using Wikipedia to support academic work.Furthermore, Colon-Aguirre and Fleming-May (2012) interviewed 21 undergraduate students, all of whom used Wikipedia for their research.In addition, Flavin (2012Flavin ( , 2016) ) shows Wikipedia being used widely by participants.Wikipedia was also used widely in a survey of 1658 students in Australia (Henderson et al. 2015).Moreover, students' usage of Wikipedia is generally efficient and responsible, treating it as a valuable support for learning; Selwyn and Gorard (2016) show Wikipedia 'fulfilling a supplementary and/or introductory role, providing students with initial orientation and occasional clarification on topics and concepts …' (p.33).That said, Dunn, Marshman and McDougall (2018) evaluated five Wikipedia articles on basic statistical concepts, finding the articles to be poor in general.Their findings notwithstanding, Wikipedia is clearly established in higher education in practice, used by students to get jobs done.The use of Wikipedia in higher education is not limited to students.Knight and Pryke (2012) in a study at Liverpool Hope University found both students and lecturers making widespread use of Wikipedia (75% of the sample), and Konieczny (2016) argues: We also are too often concerned with 'coming out', admitting to our students and even more, our colleagues, that the site is a valuable resource that we also consult (We do so, too.)As long as such attitudes persist, we are denying ourselves access to one of the most revolutionary tools for educating the general public, and our students, about the topics we have spent years, if not decades, teaching about.(p.1531) Similarly, Flavin (2017) argues Wikipedia is used by lecturers as well as students (pp. 75-76).
Wikipedia comprises a challenge to academia because it alters the way knowledge is produced, with the role of the expert challenged by the citizen.Wikipedia's mode of production thus contravenes established academic practice, the latter's system of peer review (absent in Wikipedia) comprising quality assurance.Eijkman (2010) argues, 'Wikipedia … is inimical to conventional power-knowledge arrangements in which academics are traditionally positioned as the key gatekeepers to knowledge' (p.173).Bayliss (2013) and Konieczny (2016) also note a negative attitude toward collaborative knowledge produced outside academia.
Di Lauro and Johinke (2017) argue that unwarranted stigma is attached to the use of Wikipedia in higher education (p.478), while Bayliss (2013) identifies '" philosophical tensions" between the closed nature of academia and the university, and the democratic nature of Wikipedia' (p.42), tensions unpacked by Kittur et al. (2007), who analyse the distribution of authorship in Wikipedia; in the beginning, elite users contributed most of the articles.However, and commencing in 2004, there was a shift in the distribution of work to the common users, with a corresponding decline in the influence of the elite (p.8).
Wikipedia does not conform to established models for academic publication but it is not completely unregulated; Staub and Hodel (2016) draw attention to administrators, arbitrators and helpers on Wikipedia, providing support and a measure of control (p.351).That said, Wikipedia does not have peer review in the sense understood by traditional academic publication.As Yasseri et al. (2012) note, 'the editorial policy is in favour of consensus over credentials'.Messner and DiStaso (2013) argue that Wikipedia, through its ground-up mode of production, challenges, '[t]he "gatekeeping model," in which elites select the information that is published ' (pp. 482-483).Moreover, Konieczny (2016) argues that Wikipedia is, 'disrupting the power-knowledge arrangements which in the past gave power to the academics through the peer-review and academic-press systems' (p.1527).Wikipedia is an established and useful technology, but its value is still questioned, especially because its mode of production is inimical to that of academic journal articles.This article therefore attempts to address aspects of the relative value of Wikipedia and academic journal articles in specific contexts, through quantitative analysis.

Method
The specific terms researched for this article are 'virtual learning environment', ' Second Life', 'massive open online course', 'learning analytics' and 'bring your own device'.The rationale for selecting these terms is that they are representative of different phases of development in technology-enhanced learning.The Wikipedia entry for VLEs was created on 23 January 2004; the entry for Second Life was created on 11 April 2004; the entry for MOOC was created on 10 July 2011; the entry for learning analytics was created on 23 August 2010; and the entry for BYOD was created on 01 January 2012.Furthermore, the five terms include in-house and commercial technologies (UCIA 2016).A meta-study of articles on Second Life covering the period 2005-2011 (Wang and Burton 2012) found that academic journal articles on Second Life rose in number from 2005 but declined from 2009, suggesting diminishing interest in the platform as a means of supporting learning and teaching.Regarding MOOCs, Diver and Martinez (2015) show that there was one published academic journal article on MOOCs in 2008, rising to 26 in 2012, suggesting an increasing interest in MOOCs.
BYOD is an emerging practice, underpinned by students' increasing ownership of smartphones; the EDUCAUSE Center for Analysis and Research (2015) shows smartphone ownership at 92% in 2015 in the USA, albeit in the context of the undergraduate student population, whose ownership of smartphones exceeds that of the general population (Educause 2015, p. 13).Furthermore, Kobus, Rietveld and van Ommeren (2013) show 96% of Dutch students, from a sample of over 3000, having a laptop, tablet or smartphone.The same study showed ownership of devices being high for all students, including those from lower income groups.Similarly, learning analytics is a relatively recent area of interest and enquiry in technology-enhanced learning; the leading academic event in the field, the International Conference on Learning Analytics and Knowledge, held its eighth conference in 2018 (the first having taken place in 2011).
'Virtual learning environment' is a term that spans the majority of the period during which digital learning has been present in higher education.A search on the academic journal aggregator Academic Search Complete showed articles on VLEs being published from April 2001 onwards (Ngor 2001), with 28 articles produced by a search on Academic Search Complete for 2016, based on searches for 'virtual learning environment' within 'author supplied keywords'.
As for Wikipedia itself, Okoli et al. (2014) show that there was one scholarly study of Wikipedia published in 2003Wikipedia published in , and one each in 2004Wikipedia published in and 2005.Thereafter, the number increased, to over 20 by 2009 (p. 20).Mesgari et al. (2015) note that the number of articles on Wikipedia stabilised at around 18-21 per year from 2008, though their research stopped after June 2011 (p.236).A search on Academic Search Complete for 2016, searching for 'Wikipedia' within 'author supplied keywords', yielded 37 results.

(page number not for citation purpose)
Choosing the five terms studied in this article makes it possible to look at how technology-enhanced learning-related terms on Wikipedia arise, increase in interest and, in some instances, decline.Furthermore, by looking at terms from different phases in the development of technology-enhanced learning, it becomes possible to see if a pattern of publication and revision is repeated.

Data collection
The data sources for this article were Wikipedia and Academic Search Complete.The latter is an academic journal aggregator ranging across disciplines, including more than 7300 peer-reviewed journals (Wikipedia 2017).It was chosen for this study because of the breadth of subjects it surveys and collates from.Moreover, peer-reviewed journals were used because they are considered to have passed a quality assurance process, which is different to Wikipedia's mode of production, yet Wikipedia has proved itself to be popular with students and academics alike, its mode of production notwithstanding.
Data were collected separately for each of the five terms studied.The authors initially noted the date of the creation of the first Wikipedia articles on each of the five terms.Thereafter, the authors identified article revisions available each subsequent year, on the day of the anniversary of the first article (these were defined as the revisions of the last edit before or on the day of the anniversary).Only data up to the last anniversaries before July 2017 were included.
For each of the versions recorded, the article length, in words, was counted.The authors included the entire article, excluding only the title and the notification that the viewed revision was different from the current revision.Microsoft Word 2013 was used to count the words; each revision was copied into a blank document, and the automatically generated word count was noted.
The number of Wikipedia article edits by year were obtained.The authors included edits from the first creation of the article and included all edits up to and including the last revision as identified above.The edit count for the following year started with the first revision after the day of the anniversary.
Edits on Wikipedia are undertaken by both human contributors and by 'bots', the latter executing decisions without human intervention and operating from dedicated and recognisable Wikipedia user accounts (Tsvetkova et al. 2017).As there is no information on the numbers of edits excluding bot edits available online, the authors counted the edits semi-manually.Microsoft Excel 2013 was used for this purpose.A summary of each edit from the Wikipedia revision history page was copied into a new cell within one column.In the cell adjacent to the right of each summary, the second author wrote a formula that yielded '1' if the summary did not include 'bot' or '0' if the summary included 'bot'.All the yielded values in each year were added up.This final number was noted down as the number of edits in each year, excluding bot edits.
The Academic Search Complete search for each of the five terms commenced from the same month and year as that of the first Wikipedia entry (Academic Search Complete allows for searching by month as well as year of publication).The authors also looked for articles in the year preceding the first Wikipedia entry for each term.The authors searched for each term within the Academic Search Complete category of 'author supplied keywords'.The list of results was surveyed, to ensure the articles were relevant to the topic (for example, the search for 'Second Life' produced a small number of articles about a particular type of battery; these were excluded from the total).

Data analysis
For the data analysis, IBM SPSS Statistics 23 software was used.Data for each term were entered as a separate case.To conduct repeated-measures analysis of variance, all three types of data were converted into standardised scores.For example, for data about article length, mean and standard deviation (SD) were calculated from all lengths of all revisions from the first entry to the entry after 5 years (six time points in total).From each data point, the mean was subtracted; this value was divided by the SD to create standardised scores.For the number of edits, the same procedure was used, calculating only from five time points, starting 1 year after the first entry.For academic journal articles, standardised scores were calculated separately from five and six time points; the latter included 1 year before the first Wikipedia entry.Looking at updates on Wikipedia entries has previously been done by Rubin and Rubin (2010) in a study of company-related information on Wikipedia; the researchers used editing frequency as their approach to data gathering.Similarly, Wilson and Likens (2015) downloaded the complete revision history of entries on selected scientific topics on Wikipedia, also studying updates.Furthermore, Jirschitzka et al. (2017) studied the development of articles on controversial issues in the German-language version of Wikipedia, though their research (undertaken up to March 2013) was focused more on the intrinsic qualities in Wikipedia articles concerning rival perspectives adopted about alternative medicine.

Results
For the following analyses, data from all five terms were aggregated and standardised as described above.All non-standardised descriptive statistics are presented in Table 2.

Article lengths
A two-way repeated-measures 2 (source: Wikipedia article lengths; academic journal articles) × 6 (year) analysis of variance was conducted to compare the effect of these variables on indication of interest (a variable whose value is given by the standardised scores as described in Method, Data analysis).The source term, as expected because the values were standardised, was not statistically significant.The year term was statistically significant (F [5.20] = 9.91, p < 0.001, η p 2 = 0.71).The interaction term was not statistically significant (F [5.20] = 1.10, p = 0.391, η p 2 = 0.22).Descriptive statistics are presented in Figure 1 and Table 3. Descriptive statistics of non-standardised values are given in Figure 2 and Table 2. Partly in line with the hypothesis, both article lengths and numbers of academic journal publications appear to rise year by year.

Wikipedia article edits
A similar two-way repeated-measures 2 (source: Wikipedia article edits; academic journal articles) × 5 (year) ANOVA was conducted to compare the effect on indication of interest, using standardised scores.Again, the source term was not statistically significant.The year term was significant (F [4, 16] = 4.63, p = 0.011, η p 2 = 0.54).The interaction term was also significant (F [4, 26] = 3.09, p = 0.046, η p 2 = 0.44).Descriptive statistics are shown in Figure 3 and Table 4. Descriptive statistics of nonstandardised values are shown in Figure 4 and Table 2.
In accordance with the hypothesis, the number of edits peaks at 3 years after the first creation of the article and then begins to decline.The number of academic journal articles is still rising in Year 5.

Results for the individual terms
Disaggregated data for the individual terms were analysed to identify patterns.The data for VLE show an early peak in Wikipedia edit activity, between Years 2 and 3 (Figure 5).Academic journal articles featuring VLE as an author supplied keyword eventually overtake the number of Wikipedia edits, at Year 8.The results for Second Life show a Wikipedia edit peak at Year 3, with fewer and fewer edits thereafter (Figure 6).The number of academic journal articles peaks at Year 7, gradually decreasing thereafter.The pattern suggests diminishing interest in Second Life, in line with the findings of Wang and Burton (2012).By the final year recorded (Year 13 from the publication of the first Wikipedia entry), the number of journal publications is markedly below the Year 7 (2011) peak.
The pattern for MOOCs indicates that the highest level of Wikipedia edit activity was at Year 2 (Figure 7).The notable rise in academic journal articles from Year 3 suggests increasing interest in MOOCs.The length of the Wikipedia article is broadly stable from Year 2, before which the article length had been increasing noticeably.In contrast to the pattern for Second Life, the pattern for MOOCs suggests ongoing interest, as of June 2017.The pattern for learning analytics deviates from the aggregated data.The number of edits on the Wikipedia entry for learning analytics has fluctuated, while the length of the Wikipedia article has continued to increase (Figure 8).The publication of academic journal articles is more reflective of the pattern of the aggregated data, showing a rise from Year 3 onwards.
Similarly, the pattern for BYOD differs markedly from the pattern for the aggregated data (Figure 9).The length of the Wikipedia article is still increasing.The number of Wikipedia edits peaked around the time the entry was first produced and has thereafter decreased, though with a slight, recent rise.The number of academic journal articles published on the subject has fluctuated.

Discussion
Analysis of the lengths of Wikipedia articles in relation to the number of academic journal articles published on the aggregated five technology-enhanced learningrelated terms indicates that the data are not in line with the hypothesis.There is, admittedly, a rising pattern for both the length of Wikipedia articles and the number of academic journal articles published.Moreover, there is a difference between the Conversely, the analysis of the aggregated data reveals a difference between the patterns of Wikipedia article edits and numbers of academic journal articles published.Wikipedia edits appear to peak at Year 3 from the date of their original production.More activity is apparent in the Wikipedia articles in their first 3 years than in academic journal publications over the same period, supporting the proposition that Wikipedia is a more useful barometer of interest in the technologies identified for this article than academic journal articles.Wikipedia may be a more useful source than academic journal articles for new and emerging technologies, because substantial information is available sooner, and the level of activity on a Wikipedia article may signify a wider public interest in the topic.There may also be an element of 'the wisdom of crowds' (Surowiecki 2004) to enhance the quality of Wikipedia articles, as more contributors develop an article.Considering the disaggregated data, the three most established terms considered in this article (the terms with the oldest Wikipedia articles) follow, broadly, the overall pattern observable in the aggregated data.However, the data for the most recent terms, 'learning analytics' and 'BYOD', differ from the pattern, as the number of edits peak in Year 1.It is possible that the different patterns for learning analytics and BYOD relate to the fact that both terms are relatively recent.As learning analytics and BYOD acquire a longer trajectory, they may begin to resemble the overall pattern for the aggregated data (though it is also possible that they may follow an entirely different pattern); the peak in edits in Year 1 might be an indication of Wikipedia's increasing ubiquity.The aggregated data show that information on the terms studied is available quickly on Wikipedia, emerging well before academic journal articles.Furthermore, and as the literature review shows, Wikipedia is used widely by students and by lecturers.Wikipedia has use value, and its use is enhanced if the user is seeking information on a new or emerging term.

Conclusion
This article has focused on the production and revision of Wikipedia entries, in relation to the publication of academic journal articles, featuring specific technologyenhanced learning-related terms.Addressing the questions posed at the outset of this article, the pattern of publication on Wikipedia is more intense, in terms of level of activity, in the first 3 years than is apparent in academic journal articles, for the five terms studied.Analysing the length of Wikipedia entries in relation to the publication of academic journal entries does not reveal a statistically significant pattern, but analysing Wikipedia edit activity in relation to the publication of academic journal articles does reveal a significant pattern.It is possible that a reader will find out more (in terms of volume of information) through engaging with Wikipedia than by reading academic journal articles, at least at the new and emerging phases of the area being studied.That said, it is acknowledged that the two modes of publication are different: Xiao and Askin (2014) note that Wikipedia discourages original research (p.343), which is still the preserve of the academic journal article, but a reader wanting information about new and emerging technology-enhanced learning-related terms is likely to find their needs satisfied, in the first instance, through Wikipedia.Wikipedia entries are in many instances a more useful barometer of interest than academic journals, as Wikipedia activity peaks early, providing a notable volume of information.Moreover, Jirschitzka et al. (2017) add that '[t]he more an article is edited the more it will become balanced', implying a link between edit activity and intrinsic quality.
Further studies might repeat the general approach adopted in this article but with a larger sample to enhance the reliability of the data.In addition, automated data gathering, such as Wikipedia dumps, could enable analysis of a wider range of topics.Moreover, the recommendation of Okoli et al. (2014) remains valid: 'Any given Wikipedia article will generally improve over time, so it would be quite insightful for a study to examine if individual readers' perceptions might change over time'.That said, this article argues that changes to Wikipedia articles are concentrated in the early stages of its evolution (the first 3 years), so readers' perceptions may change less if the body of information on a topic is relatively stable.The limitations of this article include the fact that it looks at lengths of Wikipedia articles and the number of edits on Wikipedia articles, as well as at the number of academic journal articles published

Figure 3 .
Figure 3. Standardised indication of interest by source.

Table 1 .
Outline of two analyses of variance to investigate the effect of independent variables on indication of popularity (as measured by different sources).
(page number not for citation purpose)

Table 3 .
Standardised indication of interest by source.

Table 4 .
Standardised indication of interest by source.