ORIGINAL RESEARCH ARTICLE
Does technical assessment matter? Functionality and usability testing of websites for ESL/EFL autonomous learners
Natividad Aguayo* and Cristina M. Ramírez
Department of Philology and Translation, University Pablo de Olavide, Seville, Spain
(Received: 29 October 2019; Revised: 21 February 2020; Accepted: 1 May 2020; Published: 14 July 2020)
Abstract
Given the social impact and the transformation of the teaching–learning process enhanced by new technologies, online language learning has been established as a field of study that has been approached primarily from the perspective of pedagogical themes. In the context of the LinguApp research project developed at the University of Córdoba (Spain),1 we aim to evaluate the technical quality of a group of English teaching websites for self-directed learning. The analysis is based on functionality and usability aspects through the use of a specifically designed checklist, created and preliminarily implemented in the early development phase of this study. To complete the design of the checklist before external validation, we offer a comparative study of four renowned websites from the LinguApp corpus: ESOLCourses, BBC, British Council and Cambridge English.2 These preliminary results allow to identify the strengths and weaknesses of these language learning websites by subjecting the data to qualitative and quantitative analysis, while they shed light on the need to strengthen web performance and so reinforce autonomous language students’ experience.
Keywords: technology-enhanced learning; informal English learning; self-directed learning; website technical quality; website evaluation
*Corresponding author. Email: naguarr@upo.es
Research in Learning Technology 2020. © 2020 N. Aguayo-Arrabal and C.M. Ramírez-Delgado. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.
Citation: Research in Learning Technology 2020, 28: 2353 - http://dx.doi.org/10.25304/rlt.v28.2353
Introduction
Language learning using online resources has become increasingly relevant in recent decades. Advantages of e-learning technologies can largely explain the learners’ preference and the institutional support for the use of new technologies in language learning outside formal education (European Commission 2008). Consequently, the English language teaching market is changing to be able to respond to the new demands of consumers for more personalised and flexible services and products (British Council 2018, p. 15).
Information and communication technologies (ICTs) play a key role in language learning outside of the classroom setting since they support different learning styles. As a clear example of a context where these technologies have a significant effect, this study focuses on informal education or self-learning activity. This learning context is defined as the one that ‘is taken to be less organised and structured [than formal and non-formal education], but is nevertheless characterised by the intention to learn’ (Ingham, Ingham, and Afonso 2014, p. 4). Indeed, ‘out-of-class learning with technology comprises an essential context of second language development’ (Lai, Hu, and Lyu 2017, p. 114) to such extent that technology-enhanced language learning (TELL) has received increasing attention in studies on language acquisition in the digital age (Healey 2016).
Among the variety of learning experiences and resources available, we will focus on websites that are considered tools that offer great possibilities in language learning (Kir and Kayak 2013; Son 2005). However, despite significant progress in digital technologies as tools for learning, there is still much work to be done on how to exploit and integrate them to make websites more effective for language instruction (Aguayo and Ramírez 2019; Gütl et al. 2013).
Certainly, there is generally little research on out-of-class, self-directed language learning through the use of online platforms (Ho 2018) and, more specifically, on the pedagogical and technical usability of language learning sites (Stevenson and Liu 2010). Although technical usability has been regarded as a ‘self-evident requirement’ in this type of learning resource (Hadjerrouit 2010b, p. 58), remarkable technical weaknesses have been identified within the context of the LinguApp project (Aguayo and Ramírez 2019). Besides, this research challenges to provide an alternative approach to the traditional studies on online language teaching and learning, frequently explored from a pedagogical perspective (Kessler 2011).
By implementing the draft checklist based on functionality and usability aspects, we aim to gain in-depth insight into optimal web functionality and usability criteria, and complete the development phase of the proposed assessment tool. The preliminary findings reveal that there is a need to concentrate on this dimension while contributing to overcoming the existing lack of research on language teaching website evaluation (Moreno and Risueño 2018).
Website quality characteristics: functionality and usability
According to Rocha and Brandão (2011, p. 376), ‘the technical dimension is related to how the content and services are assembled and made available on a website’. The evaluation of this dimension is based on software quality models which are represented mainly by the ISO 9126 standard (2001) and its successor ISO 25010 (2011) (ibid.). Almost all existing checklists made up by different authors and referenced in our study are based on ISO 9126. However, both standards present a high degree of similarity concerning the quality characteristics we analyse in this paper. The quality model frameworks established by these standards consist of six common characteristics: functionality, reliability, usability, efficiency, maintainability and portability. Security and compatibility are characteristics added in the newest standard ISO 25010, which also includes, as a usability sub-characteristic, a vitally important feature nowadays: accessibility. Since the reviewed checklists rely on ISO 9126, accessibility is not present in them, neither has it been evaluated in this study. We consider it is a very wide issue which covers many areas, so it should be addressed in an independent study. The draft checklist created by the authors (Aguayo and Ramírez 2019), which will be reused in this study, focuses exclusively on two of the quality characteristics proposed by both ISO standards: functionality and usability, characteristics directly related to the user. More specifically, functionality provides functions to satisfy the user’s needs and usability measures the user’s effort when using the product (Dugalic and Mishev 2012).
As a means to analyse the quality of any feature of a website, we first need to take into account that a website is a technology-based product. It is crucial to know the characteristics of a website as a genre, but more importantly, to envision the website as a new reality for teaching and learning: a system with a different way to map, structure and present content with respect to traditional genres (i.e. textbooks).
Shepherd and Watters (1999) declare that web genres are characterised by three main components: content, form and functionality. Content and form are the intratextual dimensions of any text, but functionality is exclusive to cybergenres. In fact, functionality is the feature which best defines them as ‘it encompasses all the capabilities afforded by the web medium’ (ibid.).
Functionality comprises, on the one hand, the technology or technologies involved in the development of a website that must be technically appropriate for the general functions, tasks and objectives (Gledec 2005). Moreover, on the other hand, specific technology required depending on the website particular purpose. This technology will define ‘the extent to which the available e-learning system provides expected results or effects for specified tasks and user objectives’ (Padayachee, Kotzé, and Van der Merwe 2010). This means that functionality determines technical performance and the specific services or functions a site offers (Fogg et al. 2002). These aspects are referred, respectively, to as Adequacy of technology and Accuracy of technology for the specific purpose on the checklist designed by the authors (Aguayo and Ramírez 2019).
Hypertextuality can be considered the most defining characteristic of functionality in the web medium because it represents the basic and most important distinction (content structure and cohesion) between traditional textual genres and web genres. Hypertext is a non-linear, non-sequential modality of content organisation, writing and reading, which affects the way users access, find and use web contents. There is not a defined or monolinear order which determines the way the user must read or consult the content, that is, the ‘movement’ (navigation) between the different textual nodes can be made in many different directions. Thus, the web structure would be, in Brockmann, Horton, and Brock’s words (1989, p.183), the most unpredictable and confusing for the user, with a greater risk of experiencing problems when using and navigating through the content. This can explain the fact that navigation is present on many reviewed web quality evaluation checklists. This dimension is not only conceived as a technical or functionality aspect that defines movement between the information nodes of the website, but also as an important usability feature that allows the user to operate and understand the website.3
Interactivity is another sub-characteristic which applies to both functionality and usability dimensions: it is a functionality-dependent issue, that is, it is based on web functionality quality and technology innovations. Nonetheless, interactivity affects usability quality because it provides capabilities for users to interact with the website in order to operate it.
According to Cassidy (2015, p. 124), ‘the interactivity offered within a website is a positive consumer motivator because it allows the user integration, the engagement with website content, and the dialogue between website and user. But this component gets even more attention in the specific context of autonomous learning: learning is an interactive process between the student and the learning environment, an environment represented by the website in this learning modality.
In the e-learning context, usability requirements are twofold (Hadjerrouit 2007, p. 41): technical usability and pedagogical usability. Technical usability is related to the general concept of usability with regard to the website as a technical product. Technical usability of a website for second language learning is focused on convenience, practicability and use for the learner (ibid.), that is, a virtual environment with good information structure resulting from proper management of contents as well as a correct page design (Nielsen 1999).
On the other hand, pedagogical usability is the part of usability aimed at supporting the learning process (Hadjerrouit 2010a, p. 119). It is directly related to educational aspects and learning theories (Hadjerrouit 2007, p. 40). Fully aware of the interdependence of technology, pedagogy and content as the main features in web-based learning resources (Hadjerrouit 2010b, pp. 56–57), we have identified the need to contribute to research in the technical assessment of language learning websites: ‘technical quality is another important determinant of the quality of e-learning, and technical problems strongly influence the overall success and satisfaction of users’ (Al-Fraihat, Joy, and Sinclair 2018, p. 63). These considerations are reinforced by the ideas of Kukulska-Hulme and Shield (2004, p. 4237):
No matter how pedagogically effective the content may be, it is of little use if the learner is unable to locate it in a poorly organized website. Pedagogical usability, then, is based upon principles of general [technical] usability. […] Technical usability is therefore the basis for the other three levels, whilst not being sufficient by itself.
Thus, the analysis of technical usability becomes the starting point that will also motivate meeting the challenges of pedagogical usability in further research (Son and Park 2012, p. 139).
Nielsen (2012) defines [technical] usability as a quality attribute that assesses how easy a product is to use, linked to methods for improving ease of use during the design process. Ease of use (also referred to as ‘learnability’ in both ISO standards) is a usability sub-characteristic defined as ‘the degree to which the software product makes it easy for users to operate and control it’ (ISO/IEC 25010). This quality attribute is crucial when learning to operate the product, exploring new features, remembering names of commands and performing tasks (Padayachee, Kotzé, and Van der Merwe 2010). To evaluate these elements, checklist indicators for this sub-characteristic must focus on the web structure, content organisation and order, as well as coherence between parts (Aguayo and Ramírez 2019). Other important usability aspects are: (1) intelligibility (also labelled as ‘understandability’), which constitutes a first and decisive step towards the website being easily understandable for the user; and (2) operability, which measures the level of web control or operation by the user with as little effort as possible.
Finally, websites are visual by nature, so their ease and satisfaction of use depend on the user’s visual perception, which, in turn, is determined by design. This explains why all the aforementioned usability aspects depend on web design and why it is included on all the reviewed checklists. Web design is related to the web’s aesthetics, that is, the presentation and disposition of elements in the interface, and it contributes to usability by being visually clean, simple, understandable and consistent.
In the following section, we define the research method used to conduct the study, which has been designed according to the web quality characteristics and sub-characteristics previously described.
Research objectives and methodology
This study builds upon both the technical evaluation process during the first phase of the LinguApp project (stage 1) and the results of our first preliminary study (stage 2). Our paper offers an examination of the technical quality of the selected EFL (English as a Foreign Language) websites currently available online for self-learning. By using the draft checklist generated from the outputs of the previous two stages (Aguayo and Ramírez 2019), we aim to identify their strengths and limitations after assessing the main functionality and usability features. This effort can lead to a better design of the resources for more effective task performance on language learning websites and prevent learners from abandoning a website that may hinder successful learning.
Although the nature of this research is mainly qualitative, as it seeks to explore and interpret the qualitative aspects of the phenomenon under investigation, data collected were analysed using quantitative techniques such as frequency count and percentages. Therefore, opportunities for improvement can be identified in the resources after conducting the comparative study. This analysis will also help to determine the completion of the development phase for the design of the assessment tool before external validation.
Research objectives
More specifically, the specific objectives defined within the limits of this study at stage 3 include the following:
- Qualitatively analyse a selection of three free EFL websites for autonomous learning included in the website corpus of the LinguApp project.
- Conduct a comparative study by quantifying the results from the analysis of the three EFL websites: BBC English, British Council, Cambridge English (WEBTESELCORP corpus4), in contrast with the preliminary data from ESOL Courses.5
- Identify and describe the categories and subcategories with the highest number of deficiencies and strengths. This will mainly contribute to determine what functionality and usability aspects should be qualitatively improved for each resource.
Research key stages
During the development of this research, subsequent key stages are represented in Figure 1 as follows:
Figure 1. Research key stages.
Firstly, the technical evaluation form created and implemented during stage 1 was particularly useful as an initial approach.6 The design and validation process of this starting tool was conducted according to the scientific parameters that assure its validity and usability (Gómez-Parra, Huertas-Abril, and Espejo-Mohedano 2019, p. 74). From this experience, it was agreed that a deeper analysis of the technical dimension was required after determining serious technical weaknesses that considerably prevailed over the content strengths. At this point, the following selection criteria for the website corpus were established according to the aims of the project and also in accordance with this study:
- Web genre: websites (unlike other resources available on the web: i.e., social networks, virtual learning platforms, etc.).
- Learning context: autonomous learning, informal education.
- Purpose of learning: to learn or improve English language learning skills.
- Availability and ease of access: free, open-access websites.
- Authorship: websites owned by official institutions, academic entities or English teaching publishing companies.
In stage 2, a checklist was specifically created and implemented to analyse the ESOL Courses website. This evaluation tool was designed so that the analysis covered the essential aspects of web functionality and usability. Hence, our work is based on an extensive reading of the most relevant literature in this regard – particularly the ISO 9126 and 25010 Quality Models for external and internal quality (ISO 9126-1, p. 2000) – together with an exhaustive selection of checklists from noteworthy studies on technical website evaluation, some of them on learning websites: Boklaschuk and Caisse 2001; Moustakis et al. 2004; Gledec 2005; Aly 2008; Membrate 2010; Padayachee, Kotzé, and Van der Merwe 2010; Hasan and Abuelrub 2011; Devi and Sharma 2016. This literature background has guided the selection, establishing and designation of categories, subcategories and specific evaluation items (Aguayo and Ramírez 2019, p. 6–10). A summary outline of the checklist design and implementation processes to better understand the WEBTESELCORP analysis is included below:
As we can see in Table 1, we distinguish two main categories (Functionality and Usability) established in the checklist design phase. A series of subcategories can be identified in each category. These dimensions are present without exception in the checklists examined and in the ISO 9126 and 25010 standards, and so considered relevant to our research objectives. In the previous section, we have offered the theoretical foundations that support the design and selection of the elements that compose this evaluation tool.
Functionality subcategories and items focus on technological issues: on one hand, the website’s proper hypertextual functioning, that is, the possibility of accessing and navigating all parts of the website and other linked external resources. On the other hand, the level of user performance allowed by the website’s technology both in the general technical and learning-specific dimensions.
Usability is concentrated in aspects related to the user’s understanding and control of the resource so that they can use and operate it with as little effort as possible. For a website, these aspects depend on a proper and logical web structure, coherence between sections, more explicit instructions and user guidance on how to proceed and how to move throughout the website.
The evaluation process was similar to the one carried out in the technical assessment for ESOL Courses at stage 2: the two authors analysed the three selected websites as evaluators so that their adequacy in terms of technical quality could be verified according to the checklist criteria. Taking into account that this study is primarily exploratory, reaching consensus from statistical insights was not intended in this work; thus, both experts tested each website individually and exchanged the information registered. A few slight discrepancies – caused by misrecorded or missing data – were reviewed jointly by performing a second in-depth analysis of the resources. As soon as the responses were sufficiently justified and agreed, they were considered as definitive.
The establishment of specific criteria to complete the checklist helped the evaluators to come to consensus by selecting one of the three suggested fields for each item. These fields for item evaluation are shown in Table 2: total or partial compliance (green and yellow, respectively) or the absence of items being studied within WEBTESELCORP (displayed in red):
By following this methodology, and once the research objectives and the study corpus have precisely been defined, the results from the proposed analysis are discussed in the section below.
Data analysis
In this section, we present the preliminary results of the analysis of functionality and usability (categories 1 and 2) in the WEBTESELCORP corpus. After completing the checklist, we will carry out a comparative study between all the websites, including ESOL Courses from our previous work (Aguayo and Ramírez 2019).
Functionality assessment
‘Navigation’ is the first subcategory within Functionality. It consists of a total of eight items related to the operation of buttons, functioning of links, the loading of content and the visualisation of dynamic components. In short, all those elements that allow us to visualise the content and move through the different sections on the website.
The evaluation for this subcategory has been highly satisfactory in the three resources analysed, with all resulting in positive outcomes, except for the two partially complete subcategories in ESOL Courses (see Table 3).7 These findings indicate that this website shows poorer quality for items 1.1.2. and 1.1.4. corresponding to the functioning of external links and the performance of the links on the website.
Secondly, the results for the ‘Adequacy of technology’ and ‘Interactive functionality’ subcategories are combined and presented in Table 4.
In the first case, all the elements analysed are met for the ‘Adequacy of technology’ subcategory. On the contrary, concerning the technological dimension of interactivity (‘Interactive functionality’), the Cambridge English resource does not include any kind of direct communication technology which allows the user to interact with anybody involved in the learning process.
Moving on to the next subcategory, ‘Accuracy of technology for the specific purpose’, important weaknesses are identified in all resources (see Table 5), although the BBC has obtained a majority of positive results (8 out of 10 items). It must be highlighted that the items included are based on the implementation of specific technology and functions in the website development, which are necessary for its stated purpose. This technology is capable of providing a detailed evaluation of the activities completed by self-learners in the absence of face-to-face teacher assessment.
According to these results, you cannot save or manage progress with most resources. This may hinder the student’s continued use of the tool and so the identification of learning improvements that occur over time. This shortcoming can also lead the user to unintentionally repeat tasks by not keeping track of their overall progress. On the other hand, although the four websites have an automatic assessment system, we must pay attention to item 1.4.3. Important deficiencies have been detected in the evaluation of writing and speaking tasks, understood as core learning areas for the development of the user’s linguistic competence which, in turn, causes ignorance of the aspects on which the student must focus during learning. These outcomes are in line with the general idea that online technologies are considered more helpful in developing receptive skills – listening and reading – than addressing productive skills – speaking and writing – (Kyppö 2017). Additionally, we emphasise the importance of being able to export or download materials which allow the student to work offline (1.4.4.), an item fulfilled by three-quarters of the resources analysed within WEBTESELCORP.
Usability assessment
With regard to the second major category, ‘Usability’, we will firstly examine ‘Intelligibility’, that is, the site’s ability to be understood by the user.
The results for this subcategory are generally positive, especially those related to items 2.1.2., 2.1.3. and 2.1.4., as shown in Table 6. However, negative results (except for the British Council site) were found regarding the inclusion of introductory or step-by-step instructions to help students the first time they access the tool. This is particularly useful in a non-linear and non-sequential resource in which these clarifications are essential to guide users and offer them a logical progression.
On the other hand, there is also evidence of lack of a chat or forum (except for the British Council site) that allows interaction with a virtual teacher or even with other students who use the same resource, thus missing a valuable opportunity to exchange knowledge, materials, doubts or comments about the user experience.
In the second subcategory, ‘Ease of use’, a majority of positive results can be reported. However, this is not the case for ESOL Courses (with five negative and two partial results for a total of 11 items), which denotes a significant deficiency in this subcategory, as shown in Table 7. The importance of the elements in this section should be highlighted since adequate super-, macro- and microstructural website organisation, as well as internal coherence, play an outstanding role in the efficient use of the resource. These aspects also contribute to achieve a satisfactory experience in accordance with the user’s expectations.
Regarding the ‘Operability’ subcategory, that is, the site’s ability to be managed, there are considerable deficiencies despite encountering primarily positive results (see Table 8). We emphasise weaknesses in item 2.3.1.7. related to the ‘back’ option in all the subsections of the website. This feature is crucial to return to previous levels or actions and avoid deviation from the itinerary that the user has been following during the learning process.
In the ‘Navigation’ subsection (2.3.1.), we take a closer look at the need to click more than three times in order to reach more specific levels (2.3.1.4.). This may cause the user to become exasperated in their attempt to move forward within a site extremely large with a deep, complex and inconvenient hierarchical structure, thus making exploration from the homepage difficult.
Finally, we address the ‘Design’ subcategory, related to the most aesthetic part of the website. This aspect is also an important attribute of the resource because it contributes to the tool’s ease of use.
Here, we obtain a large majority (90%) of positive results, with only 10% of usability aspects that must be improved, as shown in Table 9. Identified shortcomings are related to the coherence, clarity and cleanliness of the interface design for ESOL Courses. Additionally, we experience difficulties differentiating some navigation buttons in the British Council resource.
Summary of results
In this section, we provide an overview of the above results by offering a summary with the most important data after assessing each category and subcategory. In order to meet the research objectives, we will determine the strengths and weaknesses of each resource and of the categories (Functionality and Usability) and subcategories in the WEBTESELCORP corpus.
Overall results by website and type of response
After reviewing the results for the two categories ‘Functionality’ and ‘Usability’ for each website, the following graph (Figure 2) has been created according to the type of response (or degree of compliance) recorded:
Figure 2. Overall results by resource and type of response for the Functionality and Usability categories.
As far as the two main categories (Functionality and Usability) are concerned, Figure 2 illustrates the highest degree of item compliance on the British Council website, with 80.3% of positive responses. Furthermore, the sum of negative and partially met responses accounts for a marginally lower rate (19.7%) than that found on the BBC English online resource (23%).
At the other end of the scale, the ESOL Courses website is the resource which has the most shortcomings. Here, the value of negative and partially met responses (42.6%) represents almost half of the total number of items. Moreover, it doubles (or nearly doubles) the other websites’ proportions of unfavourable responses as shown in British Council (19.7%) and BBC (23%).
Results by website and category
Figure 3 indicates the degree of compliance with Functionality items for every website within WEBTESELCORP. The BBC and British Council online learning resources showed a higher percentage of positive responses, proving that the BBC website (at 91.7%) features the highest technical quality for this category. Cambridge English and ESOL Courses reach similar positive values (66.7%); however, the results reveal a significant difference in the number of negative responses, with Cambridge English having nearly double amount of negative results compared to that of ESOL Courses (29.1% versus 16.7%).
Figure 3. Degree of compliance for Functionality by website in WEBTESELCORP.
Regarding Usability, the British Council website is the one that best complies with the items included on our checklist, followed by Cambridge English. Despite showing the highest degree of compliance in Functionality, the BBC website ranks third in the Usability category (see Figure 4). In this context, the web resource with the greatest number of technical deficiencies is ESOL Courses (48.6% including negative and partial responses), and the BBC site comes in second place with 32.4% of the results being negative. Therefore, the British Council website is the online resource with the highest percentage of positive responses (83.8%) and the lowest number of negative and partial results (16.2%). This online learning tool is closely followed by Cambridge English. This site, although listed in last place due to its functionality limitations, received up to 78.4% of positive results and only 21.6% for total and partial usability deficiencies.
Figure 4. Degree of compliance for Usability by website in WEBTESELCORP.
Results by type of response for each category and subcategory
Although registered values are certainly similar for the two broad categories (Functionality and Usability), Figure 5 illustrates that a slightly greater percentage of negative and partial responses are concentrated in the Usability dimension (with 70% of positive responses compared to 75% in the ‘Functionality’ category).
Figure 5. Overall values for Functionality and Usability categories by type of response in WEBTESELCORP corpus.
Considering the negative and partial results in Usability (see Figure 6), the subcategories showing major deficiencies are ‘Intelligibility’ and ‘Operability’ for the negative results (38% in both cases), while ‘Ease of use’ (60%) represents the weakest area with 9 out of 15 of partial responses. In contrast, positive responses are recorded particularly in the ‘Ease of use’ and ‘Operability’ subcategories, with 29% each.
Figure 6. Proportion of negative, partial and positive responses for the Usability category in WEBTESELCORP.
With regard to Functionality, the sum of the percentage of overall negative and partial responses for all websites amounts to 25% of the total, as shown in Figure 5. Of that proportion, the highest values for the negative and partial responses are identified within ‘Accuracy of technology for the specific purpose’, accounting for 94% (17 out of 18 negative responses) and 67% (4 out of 6 partial responses). Regarding positive results in Functionality, it is worth mentioning that the subcategories with the highest technical quality are ‘Navigation’ (42%) (30 out of 72 positive responses) and ‘Adequacy of technology’ (17%) (12 out of 72) with neither negative nor partial values (see Figure 7). Despite accumulating 26% of positive items, ‘Accuracy of technology for the specific purpose’ is not actually considered one of the top two ranking subcategories since the high percentages of negative and partial responses impair its apparent optimal functional capability, thus dropping to the last position in regard to functional quality.
Figure 7. Proportion of negative, partial and positive responses for the Functionality category in WEBTESELCORP.
Results by website, type of response, category and subcategory
Focusing on significant shortcomings, most weaknesses are identified within the subcategory 1.4. ‘Accuracy of technology for the specific purpose’ (Functionality category). Figure 8 demonstrates that negative responses exceed the sum of positive and partial responses for Cambridge English, while there is a balance in these results for the British Council. The amount of positive and negative responses is the same in ESOL Courses, where items that are partially met or not met at all represent 60% of the total number of responses. However, the BBC resource is the one where only 20% of the results are negative for subcategory 1.4. Needless to say that this is an important feature since it is related to evaluation (item 1.4.3.), a crucial aspect for autonomous learning which needs improvement in most resources.
Figure 8. Proportion of strengths and weaknesses for subcategory 1.4. in WEBTESELCORP.
Considering the second main category, Usability, negative results are found to a greater extent in 2.1. ‘Intelligibility’, 2.3. ‘Operability’ and, to a lesser degree, in 2.2. ‘Ease of use’ (see Figures 9–11) within WEBTESELCORP. Findings show strong deficiencies in the aspect of ease of use – that is, being able to use the website without experiencing problems that may lead to giving up learning or cause stress and frustration for the user – .
Figure 9. Representation of type of response for subcategory 2.1. Intelligibility.
Figure 10. Representation of type of response for subcategory 2.2. Ease of use.
Figure 11. Representation of type of response for subcategory 2.3. Operability.
However, with respect to the ‘Design’ subcategory (see Figure 12), almost all of the websites completely comply with all checklist items, with a few exceptions regarding lack of coherence and clarity in interface design, and difficulties in the recognition of navigation buttons.
Figure 12. Representation of type of response for subcategory 2.4. Design.
Conclusions and future considerations
Despite there has been a growing interest in self-regulated learning for nearly two decades in the field of Applied Linguistics (Korucu-Kis 2020), research on the technical evaluation of websites for language self-learning is still quite scarce. Undoubtedly, many factors play a role in the success of online learning; however, technical quality is crucial according to the above analysis focused on the effectiveness of user’s task performance. The results reveal that if functionality and usability criteria are not fulfilled, the learner would hardly be able to effectively access and use the content and this, in turn, cannot serve the pedagogical purpose for which it was made available on the web. Hence, despite technology per se not being sufficient when designing e-resources in order to achieve learning outcomes (Hadjerrouit 2010a), the quality of the technology is truly a key requirement to achieve usability in all its dimensions for these websites (Son and Park 2012, p. 139).
This study also shows that, despite initial expectations, the fact that these online courses are offered by well-established and prestigious institutions – delivering or designing ESL teaching courses and materials – is not accordingly associated with achieving full effective functionality and usability.
Findings from our WEBTESELCORP analysis illustrate this clearly. Significant weaknesses in subcategories 1.4. ‘Accuracy of technology for the specific purpose’, 2.1. ‘Intelligibility’, 2.2. ‘Ease of use’ and 2.3. ‘Operability’ have been identified, resulting in shortcomings concerning technical performance, student support, macro- and microstructure organisation, and coherence of the tool.
- (1.4.) Integrated technology proved insufficient to achieve the specific goal of EFL learning. This means that the function aiming to provide general and individual (per activity) quantitative feedback as well as to assess written and oral production tasks, does not help to achieve the intended outcomes.
- (2.1.) Student assistance: there are few or no step-by-step instructions when using these websites for the first time nor are there enough interactive tools to support learners. Apparently, there is a misconception that independent learners do not need any help when it comes to self-study (assistance in the learning process and also in the use of technology).
- (2.2.) Little relevance of macro- and microstructure organisation within WEBTESELCORP: websites should have a hierarchical, intuitive, simple organisation, being consistent on all pages, while clarity and consistency must also come first regarding content. It is crucial that terminology and graphic elements are coherent, thus contributing to the ease of use of these e-tools.
- (2.3.) With respect to ‘Operability’, obstacles during navigation such as the lack of correspondence between the site map and the actual content organisation cause users trouble while navigating the website. Hypertextuality capabilities should be exploited so as to facilitate and not hinder navigation during the learning process.
Regarding the quality ranking from the WEBTESELCORP analysis, the British Council website is the resource of the highest technical quality except for small improvements that should be made in subcategory 1.4. At the other end, ESOL Courses is definitely the most deficient website, mainly due to its overall usability and its functionality, especially with regard to 1.4. These results show that there is little point in having accurate and reliable content if the website is neither intuitive nor easy to use, causing frustration from incoherence, multiple diversions when navigating and user isolation throughout the learning experience. Lastly, we note that optimal functionality does not necessarily correspond to excellence in usability (i.e., the BBC website ranks first in Functionality, whereas it slides down to third place in Usability).
While aware of the difficulties in reaching categorical conclusions that can be transferable to similar e-learning resources, this is definitely a significant starting point in research on the technical evaluation of language learning websites.
In fact, although the number of websites within WEBTESELCORP may seem limited, this accounts for up to 80% of the total of EFL websites registered in LinguApp – surprisingly, the amount of resources under the selection criteria was notably low (Gómez-Parra, Huertas-Abril, and Espejo-Mohedano 2019, pp. 69–71). Therefore, the evaluation undertaken has covered the main aspects of functionality and usability most frequently found on the currently available websites. Without this formal approach to the technical dimension, the key quality criteria for functionality and usability could not have been identified and incorporated in a single assessment tool. This checklist will serve as the basis for further external validation and refinement in the next steps of research.
Finally, this website quality evaluation brings visibility to the need for interdisciplinary cooperation between the developers behind the design of these sites, and teachers as experts in the learning process and learners’ needs. Institutions must also revise their sites and assure that, beyond appropriate content, technical quality is successfully fulfilled.
Acknowledgements
The authors would like to acknowledge the Andalusian Studies Centre (Fundación Pública Andaluza Centro de Estudios Andaluces) for the financial support under the research project entitled ‘LinguApp: Ensuring Universal and Inclusive Access to Second Language Learning’ (Ref. No. PRY208/17) granted in the 2017 competitive call.
References
Aguayo, N. & Ramírez, C. M. (2019) ‘Evaluación técnica de sitios web para el aprendizaje autónomo de inglés como segunda lengua’, EDMETIC, Journal of Media Literacy and ICT, vol. 8, no. 1, pp. 1–22. doi: 10.21071/edmetic.v8i1.11161
Al-Fraihat, D., Joy, M. & Sinclair, J. (2018) ‘A comprehensive model for evaluating e-learning systems success’, in Distance Learning for Educators, Trainers and Leaders, vol. 15, issue 3, ed M. Simonson, United States Distance Learning Association, Boston, MA, pp. 57–74.
Aly, M. A. (2008) ‘An evaluative study of some online websites for learning and teaching English as a foreign language’, Benha University, Egypt, Available at: http://files.eric.ed.gov/fulltext/ED499586.pdf
Boklaschuk, K. & Caisse, K. (2001) ‘Evaluation of educational websites’, Educational Communications and Technology, University of Saskatchewan, Saskatoon, Saskatchewan, Canada.
British Council. (2018) The Future Demand for English in Europe: 2025 and Beyond. British Council, London. Available at: https://www.britishcouncil.org/sites/default/files/future_demand_for_english_in_europe_2025_and_beyond_british_council_2018.pdf
Brockmann, R. J., Horton, W. & Brock, K. (1989) ‘From database to hypertext via electronic publishing: an information odyssey’, in The Society of Text: Hypertext, Hypermedia, and the Social Construction of Information, ed E. Barrett, MIT, Cambridge, pp. 162–205.
Cassidy, L. J. (2015) Website Benchmarking: A Tropical Tourism Analysis, PhD Thesis, James Cook University.
Devi, K. & Sharma, A. K. (2016) ‘Implementation of a framework for website quality evaluation: Himachal Pradesh university website’, Indian Journal of Science and Technology, vol. 9, no. 40, pp. 1–5. doi: 10.17485/ijst/2016/v9i40/100229
Dugalic, B. & Mishev, A. (2012) ‘ISO software quality standards and certification’, Local Proceedings of the Fifth Balkan Conference in Informatics, Novi Sad, Serbia, pp. 113–116.
European Commission (2008) ‘Multilingualism: an asset for Europe and a shared commitment’, Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, COM(2008) 566 final, Brussels. Available at: https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2008:0566:FIN:EN:PDF
Fogg, B. J., et al., (2002) ‘How do people evaluate a web site’s credibility? Results from a large study’, Report of Research [online] Available at: https://dejanseo.com.au/media/pdf/credibility-online.pdf
Gledec, G. (2005) ‘Quality model for the World Wide Web’, Proceedings of the 8th International Conference on Telecommunications, 2005. ConTEL 2005, pp. 281–287. Available at: https://www.researchgate.net/publication/4151791_Quality_model_for_the_world_wide_web
Gómez-Parra, M. E., Huertas-Abril, C. A. & Espejo-Mohedano, R. (2019) ‘Estudio para la validación de dos instrumentos para la evaluación de sitios web de aprendizaje de lenguas extranjeras’, Revista de Lenguas para Fines Específicos, vol. 25, no. 1, pp. 62–80. Available at: https://ojsspdc.ulpgc.es/ojs/index.php/LFE/article/view/1048
Gütl, C., et al., (2013) ‘Flexible and affordable foreign language learning environment based on Web 2.0 technologies’, International Journal of Emerging Technologies in Learning, vol. 8, no. 2, pp. 16–28. doi: 10.3991/ijet.v8i2.2488
Hadjerrouit, S. (2007) ‘Using an understanding of the learning cycle to build effective e-learning’, in Advanced Principles of Effective E-Learning, Informing Science Press, Santa Rosa, CA, pp. 27–58.
Hadjerrouit, S. (2010a) ‘Developing web-based learning resources in school education: a user-centered approach’, Interdisciplinary Journal of E-Learning and Learning Objects, vol. 6, no. 1, pp. 115–135. Available at: https://www.learntechlib.org/p/44777/
Hadjerrouit, S. (2010b) ‘A conceptual framework for using and evaluating web-based learning resources in school education’, Journal of Information Technology Education, vol. 9, pp. 54–79. Available at: https://www.learntechlib.org/p/111356/
Hasan, L. & Abuelrub, E. (2011) ‘Assessing the quality of web sites’, Applied Computing and Informatics, vol. 9, no. 1, pp. 11–29. doi: 10.1016/j.aci.2009.03.001
Healey, D. (2016) ‘Language learning and technology: past, present and future’, in Routledge Handbook of Language Learning and Technology, eds L. Murray & F. Farr, Routledge, New York, NY, pp. 9–23.
Ho, W. Y. (2018) Translanguaging in Online Language Learning: Case Studies of Self-Directed Chinese Learning of Multilingual Adults, Doctoral Thesis (Ph.D), UCL Institute of Education, University College London, United Kingdom. Available at: http://discovery.ucl.ac.uk/id/eprint/10044705
Ingham, H., Ingham, M. & Afonso, J. (2014) ‘Participation in lifelong learning in Portugal & the UK’, Education Economics, pp. 1–39. Available at: https://eprints.lancs.ac.uk/id/eprint/80008/1/Education_Economics_Revision_2.pdf
ISO (2001) ISO 9126-1:2001, Information Technology–Software Product Quality–Part 1: Quality Model, Geneva: International Standards Organization.
ISO (2011) ISO 25010:2011, Software Quality Model. Systems and Software Engineering–Systems and Software Quality Requirements and Evaluation (SQuaRE), System and Software Quality Models, Geneva: International Standards Organization.
Kessler, G. (2011) ‘Language learning in an era of ubiquitous computing: pedagogy, students and teachers’, ‘New Horizon: Striding into the Future’ Selected Proceedings of the 2011 Michigan Teachers of English to Speakers of Other Languages Conference, Kalamazoo, MI, pp. 6–17.
Kir, E. & Kayak, S. (2013) ‘The evaluation of websites teaching English as a foreign language (EFL)’, Procedia–Social and Behavioral Sciences, vol. 106, pp. 2788–2795. doi: 10.1016/j.sbspro.2013.12.320
Korucu-Kis, S. (2020) ‘An investigation into EFL pre-service teachers’ out-of-class self-regulated learning experiences with ICT’, in Enhancements and Limitations to ICT-Based Informal Language Learning: Emerging Research and Opportunities, eds R. Ahmed, A. Al-kadi & T. Hagar, IGI Global, Hershey, PA, pp. 112–137.
Kukulska-Hulme, A. & Shield, L. (2004) ‘Usability and pedagogical design: are language learning websites special?’, in Proceedings of ED-MEDIA 2004–World Conference on Educational Multimedia, Hypermedia & Telecommunications, eds L. Cantoni & C. McLoughlin, Association for the Advancement of Computing in Education (AACE), Lugano, Switzerland, pp. 4235–4242.
Kyppö, A. (2017) ‘Climbing a mountain. Learning Slovak in new language learning environments’, Department of Language and Communication Studies, University of Jyväskylä, Available at: https://jyx.jyu.fi/handle/123456789/55172
Lai, C., Hu, X. & Lyu, B. (2017) ‘Understanding the nature of learners’ out-of-class language learning experience with technology’, Computer Assisted Language Learning, vol. 31, no. 1–2, pp. 114–143. doi: 10.1080/09588221.2017.1391293
Membrate, T. W. (2010) A Framework for Evaluating Academic Web Site’s Quality from Students’ Perspective, Master’s Thesis, Delft University of Technology, The Netherlands.
Moreno, E. & Risueño, J. J. (2018) ‘Design of a checklist for evaluating language learning websites’, Porta Linguarum, vol. 30, pp. 23–41. Available at: https://www.ugr.es/~portalin/articulos/PL_numero30/3_%20Design%20of%20a%20Checklist%20for%20Evaluating%20Language.pdf
Moustakis, V. S., et al., (2004) ‘Website quality assessment criteria’, Proceedings of the Ninth International Conference on Information Quality (ICIQ-04), pp. 59–73. Available at: http://mitiq.mit.edu/iciq/documents/iq%20conference%202004/papers/websitequalityassessmentcriteria.pdf
Nielsen, J. (1999) Designing Web Usability: The Practice of Simplicity, New Riders Publishing, Thousand Oaks, CA.
Nielsen, J. (2012) ‘Usability 101: Introduction to usability’, Nielsen Norman Group [online] Available at: https://www.nngroup.com/articles/usability-101-introduction-to-usability/
Padayachee, I., Kotzé, P. & Van der Merwe, A. (2010) ‘ISO 9126 external systems quality characteristics, sub-characteristics and domain specific criteria for evaluating e-Learning systems’, The Southern African Computer Lecturers’ Association, University of Pretoria, South Africa. Available at: https://www.researchgate.net/profile/Paula_Kotze/publication/228987388_ISO_9126_external_systems_quality_characteristics_sub-characteristics_and_domain_specific_criteria_for_evaluating_e-Learning_systems/links/0c96051821b7c5e798000000.pdf
Rocha, A. & Brandão, P. L. (2011) ‘Quality of health web sites: Dimensions for a broad evaluation methodology’, in Handbook of Research on ICTs and Management Systems for Improving Efficiency in Healthcare and Social Care, eds M. Cruz-Cunha, I. M. Miranda & P. Gonçalves, IGI Global, pp. 922–936.
Shepherd, M. & Watters, C. (1999) ‘The functionality attribute of Cybergenres’, Proceedings of the 32nd Hawaii International Conference on System Sciences HICSS32, Abstracts and CD-ROM of Full Papers, Maui, HI, USA, 1999, pp. 9–[unspecified end page number]. doi: 10.1109/HICSS.1999.772650
Son, J.B. (2005) ‘Exploring and evaluating language learning web sites’, in Enhancing Learning and Teaching: Pedagogy, Technology and Language, eds J.-B. Son & S. O’Neill, Post Pressed, Flaxton, Australia, pp. 215–227. Available at: https://eprints.usq.edu.au/820/1/Son_ch13_2005.pdf
Son, J. B. & Park, J. Y. (2012) ‘Intercultural usability of language learning websites’, International Journal of Pedagogies and Learning, vol. 7, no. 2, pp. 135–141. doi: 10.5172/ijpl.2012.7.2.135
Stevenson, M. P. & Liu, M. (2010) ‘Learning a language with web 2.0: exploring the use of social networking features of foreign language learning websites’, CALICO Journal, vol. 27, no. 2, pp. 233–259. doi: 10.11139/cj.27.2.233-259
Footnotes
1 With support from the Fundación Pública Centro de Estudios Andaluces (The Andalusian Studies Centre). LinguApp is a research project (ref. no. PRY 208/17) aimed at designing a free, open-access app and web service in order to provide access to tools that have been specifically selected and evaluated to learn foreign languages (Spanish and English).
2 ‘English for Speakers of Other Languages, or ESOL, refers to learning English as a new resident in a English-speaking country. ESOL is similar to ESL. It can be compared with EFL, which refers to learning and using English as an additional language in a non-English speaking country.’ Retrieved from https://www.teachingenglish.org.uk/article/esol (Accessed June 16th 2020).
3 Web Content Accessibility Guidelines (WCAG) 2.1 (2018). Available at https://www.w3.org/TR/WCAG21/.
4 WEBTESELCORP stands for Web Testing for English Self-Learning Corpus.
- BBC–Learning English: http://www.bbc.co.uk/learningenglish/english/ (from ‘Lower intermediate’ to ‘Towards Advance’ sections)
- British Council – Learning English (Skills): http://learnenglish.britishcouncil.org/skills
- Cambridge English – Learning English: https://www.cambridgeenglish.org/learning-english/activities-for-learners/?rows=12
- ESOL Courses: https://www.esolcourses.com/
6 Refer to Gómez-Parra, Huertas-Abril and Espejo-Mohedano (2019, p. 79) to access the final version of the technical evaluation form within the LinguApp project (ref. no. PRY 208/17).
7 Colours used in Tables 3-9 refer to the three fields established for item evaluation as shown on Table 2 (page 8): green stands for positive responses, red for negative responses and yellow stands for partially met responses.