Figure 1. Recommendation of people on the sidebar box.
Luís Pedro*, Carlos Santos, Sara Filipa Almeida, Fernando Ramos, António Moreira, Margarida Almeida and Maria João Antunes
Department of Communication and Art, University of Aveiro, Portugal
(Received 26 September 2013; final version received 28 July 2014; Published 29 August 2014)
Abstract
This paper aims to assess the relevance and usefulness of the SAPO Campus recommender system, through the analysis of students’ and teachers’ opinions. Recommender systems, assuming a ‘technology-driven’ approach, have been designed with the primary goal of predicting user interests based on the implicit analysis of their actions and interactions. The results of this study reveal that although there is some confusion and unawareness about the recommender system, the participants consider that SAPO Campus recommendations are useful and they often find interesting people and content through the results provided by the system. The results also reveal that there is a negative correlation between finding and following people through the platform recommendations and the level of education, that is, the higher the level of education, the lower is the frequency regarding finding and following people suggested by the platform recommendation system.
Keywords: education; SAPO Campus; recommendations; interaction
*Corresponding author. Email: lpedro@ua.pt
Research in Learning Technology 2014. © 2014 L. Pedro et al. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 Unported (CC BY 4.0) License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.
Citation: Research in Learning Technology 2014, 22: 22921 - http://dx.doi.org/10.3402/rlt.v22.22921
In a context where it is recognised that learning occurs throughout individuals’ life, not only in formal spaces but also in informal ones (Saz et al. 2011) and with the improvement and massification of technology – specially with all the tools and applications from the social Web – learners are becoming not only consumers but also producers of content and knowledge (Siemens 2008). The existence of a huge quantity of information makes the process of searching and selecting online content a hard task for the average user, who is generally overwhelmed by information overload (Gemmis et al. 2009). Therefore, recommender systems appear as useful tools in reducing the time and costs involved in the process of searching and selecting online information (Drachsler 2009). In this paper, we will present the main results of a study that aimed to assess the relevance and usefulness of the SAPO Campus recommender system, through the analysis of students’ and teachers’ opinions.
According to Adomavicius and Tuzhilin (2005) cited in Drachsler (2009), the general purpose of recommender systems is to pre-select information a user might be interested in. Thus, this kind of system tries to predict users’ preferences based on an implicit analysis of their activity (Lee 2001), in order to support the process of searching and selecting online information (Gemmis et al. 2009).
The literature in this field tells us that there are two main types of recommender systems: content-based and collaborative (Gemmis et al. 2009). The first one is based on the analysis of users’ preferences in order to recommend content similar to what the user preferred in the past (Drachsler 2009). This recommendation method allows the user to develop a deeper understanding about a given topic but, on the other hand, causes over-specialisation problems because it does not retrieve topics outside that particular cloud of interests. The collaborative recommender systems aim to predict appropriate items based on interaction data of many users with similar interests within the community (Deshpande and Karypis 2004). Although this method retrieves new items based on similar interests in the community, the arrival of a new user or a new item could represent a ‘cold-start problem’, which corresponds to the time that the system needs to understand users’ preferences (Mödritscher et al. 2011).
Nowadays, recommender systems are successfully applied in e-commerce sites like Amazon,1 matching users’ interests with those with a similar taste and creating a ‘neighbourhood’ of like-minded customers (Drachsler 2009). These successful examples may help us to think about the advantages and limitations of these systems and to develop specific ones for educational contexts. In the Technology Enhanced Learning (TEL) research field, recommender systems have to deal with different levels of complexity involving learners and learning activities. Thus, according to Manouselis et al. (2011), it may not be sufficient to merely transfer the recommender systems logic used in e-commerce contexts to educational ones.
In educational contexts, recommendations appear to be useful for empowering learners to set up their own learning environments (Mödritscher 2010), helping them to select content according to their individual needs (Santos and Boticario 2010). Therefore, these systems could offer guidance to learners without limiting their freedom, by mediating the relationship between users’ existing knowledge and potential knowledge acquisition (Lichtnow et al. 2006). According to Drachsler, Hummel and Koper (2008), with recommendations, users are able to be self-regulated and responsible for their own learning, while they also have the opportunity to find peers and/or tools and get suggestions and motivational support from interaction with peers (Mödritscher et al. 2011).
Taking these principles into account, we have developed and integrated a recommender system at the SAPO Campus platform. SAPO Campus is an institutionally supported platform, developed by the University of Aveiro and PT Communications/SAPO for educational contexts, offering social media core services (e.g. photos, videos, blogs, status and links) in order to allow the development of collaborative-based communities of learning (Pedro et al. 2012).
Despite the institutional link – one that provides an acknowledged and regulated environment for content publishing – the platform aims to encourage openness and communication values and attitudes. Therefore, each logged user in the platform has a personal area that automatically aggregates the content he or she has published and a news feed area where the content shared by the people he or she follows is aggregated and where it is also possible to access to notifications, favourite content and recommendations.
In SAPO Campus, the recommender system does not represent the interaction core of the platform. Instead, it works just like an additional element to support users’ navigation and interaction (Pedro et al. 2012), aiming to reinforce the social dimension of the platform. The system developed is a collaborative recommender system based on the open-source technology of Easyrec2 and comprises two main areas (Almeida, Pedro and Santos 2013):
Figure 1.
Recommendation of people on the sidebar box.
Figure 2.
Recommendation of content on the newsfeed area.
To allow for the establishment of a successful relation between the system and the user and to minimise some limitations of the recommender systems (e.g. ‘cold-start problem’), in SAPO Campus, each user is added to a meaningful context (its school space in the platform), being able to interact with other school members and to receive recommendations of people and content relevant in that particular school context (see Almeida, Pedro and Santos 2013).
After making this functionality available in the platform and after a suitable time required for users to start receiving recommendations, a study was conducted in order to understand the relevance and usefulness of SAPO Campus recommender system, through the analysis of students’ and teachers’ opinions.
This paper is part of a larger case study that was developed at the University of Aveiro, focused on assessing the SAPO Campus users’ opinion about the utility and relevance of SAPO Campus recommendations in supporting the interaction of the community and the development of networks based on their own interests.
This particular study had 165 participants (84 female and 81 male) from three different elementary and secondary schools and one university. From these, 144 (87.3%) were students from all educational levels (elementary education [EE] to higher education [HE]) and 21 (12.7%) were teachers from elementary to secondary education (SE). Figure 3 shows the distribution of teachers and students by the different levels of education in which they teach or learn. It is important to notice that there were three missing answers from the group of teachers who did not mention the level of education in which they were involved.
Figure 3.
Distribution of the participants by educational levels in which they teach or learn.
After making the SAPO Campus recommender system fully available and after a suitable time required for users to start receiving recommendations, participants were asked to answer a short online questionnaire (approximately 10 minutes). The questionnaire was composed of 10 questions (four scale questions and six open-response questions) addressing two major sections: (1) general opinion about recommendations and (2) usage habits and the usefulness of the SAPO Campus recommendation system.
The questionnaire was answered online by some members of the informal group that has been working with SAPO Campus team in the pilot study (students and teachers from three different schools). For the HE group, it was made an open dissemination of the questionnaire through the SAPO Campus platform. No particular steps were taken to minimise bias in the sample since this was a pilot study and its main objective was to collect data that would be later applied in the development of the platform. Hence, the research team complemented this analysis with other data that were collected in the usage log files of this feature by this informal group. A further study will be done, with a sample that has minimised potential bias. This study will also include a qualitative scope, in order to explore the reasons underlying the choices made by the participants.
When asked if they remembered to receive some kind of online recommendations and in which services, the majority of the participants (64.8%) of both groups – teachers (61.9%) and students (65.3%), specially from SE (80%) and from the third grade of EE (71.1%) – stated that they did not remember to receive any online recommendation (Figure 4). The results presented in Figure 4 were analysed and distributed by educational levels: EE; SE; HE and the group of teachers (T). EE, in Portugal, is divided into three grades. The first one comprises 4 years (age: 6–10), the second comprises 2 years (age: 10–12) and the last one comprises 3 years (age: 12–15).
Figure 4.
Remembering to receive online recommendations.
The other 58 participants (35.2%) who said they remembered to receive recommendations through online services indicated, as shown in Table 1, mostly Facebook (25.9%).
It is also important to mention that 22.4% of those participants presented some misconceptions, mostly related with online security practices, for example, related to the importance of not sharing personal data, and 19% did not answer the question (Table 1).
As participants were asked to point out one or more examples of recommendations and services, the sum of the percentages in Table 1 is superior to 100%.
The participants were also asked to give their opinion about the importance and usefulness of recommendations, positioning themselves (agreeing or disagreeing) toward the following statements: (1) recommendations are useful to find interesting people; (2) recommendations are useful to find interesting content; (3) recommendations encourage to search for new people and content; and (4) broadly speaking, recommendations are useful. The results presented in Figure 5 show that participants, mostly students from the first grade of EE and teachers, agreed with all the presented statements, specially (2) and (4). Even though the participants have generally agreed, the statements ‘recommendations are useful to find interesting people’ and ‘recommendations encourage to search for new people and content’ were those that gathered lower percentages of agreement in all educational levels, but specially among the participants of the HE group, which seems to be, in general, the most sceptical about the usefulness of recommendations.
Figure 5.
SAPO Campus users’ opinions about the usefulness of recommendations.
When asked about some real cases in which SAPO Campus recommendations were useful, some participants (17%) stated that they were useful to find and follow interesting people, others (16%) stated that they had not received any useful recommendations yet and 11% of the participants said that the recommendations were useful to find interesting content. It is also important to mention that 35% of the participants did not answer this question and 12.3% presented some misconceptions related to the notifications and the navigation support messages of the platform (Table 2).
Participants were asked about their usage habits and their opinion about the utility of SAPO Campus recommendations, in the following topics: (1) attention to the SAPO Campus recommendations; (2) access frequency to recommendations in the newsfeed area; (3) interesting people found with the SAPO Campus recommendations; and (4) useful content found with SAPO Campus recommendations.
Figure 6 shows that all the participants, mostly students from the first grade of EE and teachers, usually pay attention (often and very often) to the SAPO Campus recommendations. Regarding the access frequency to the recommendations in the news feed area, the values seem to decrease a little bit, specially in the group of the HE students (only 27.8% of the HE students access the newsfeed area often or very often). Regarding finding interesting people with the SAPO Campus recommendation, the values also seem to decrease specially in the HE group, with only 44.4% stating that they frequently find and follow interesting people with SAPO Campus recommendations. Finally, in the statement (4), there seems to be an increasing number of participants that frequently find useful content with SAPO Campus recommendations.
Figure 6.
Usage habits and users’ opinion about the usefulness of SAPO Campus recommendations.
It is also important to note that a significant (p≤0.01) negative correlation (−0.21) was found between the action of following people with the SAPO Campus recommendations and the level of education. This means that the higher the level of education, the lower is the frequency regarding finding and following interesting people with the SAPO Campus recommendations.
Participants were also asked to present some real cases in which a SAPO Campus recommendation had been useless. In this particular question, 32.7% of them stated that they have not received any useless recommendations and 44.2% did not answer the question. Only 6.6% of the participants pointed to some cases in which they had received useless recommendations: recommendations of people (2.4%), recommendations of content (1.2%) and all the recommendations (3%). The remaining 16.3% of participants indicated some other aspects related to the navigation support messages of the platform (6%) or their general opinion toward SAPO Campus platform (6.1%). Nearly 4.2% of them also said that they did not usually visit the recommendations area.
The data presented indicate that there seems to be some confusion and unawareness about recommender systems among the participants of this study, since only 58 (35.2%) of them said they remembered to receive recommendations, and 48.3% of those presented some misconceived answers or did not answer the question at all.
It is also interesting to note that, contrary to what is somehow expected, the majority of the participants from the SE group (80%) and the third grade of the EE group (71.1%) stated that they did not remember to receive any online recommendations. A very high percentage of students, especially in these ages, are registered in social networks such as Facebook where it is almost impossible to not receive recommendations of people or content. Thus, it is important to take these issues into account, and it is important also to note that this was the first question and the students might not be familiar with this particular vocabulary, especially the term recommendation.
The results also seem to indicate that students from the first grade of EE (aged 6–10) have presented atypically high percentages of agreement throughout the questionnaire. This may be related in part with suggestibility issues. As Ceci and Bruck (1993) cited in Costa and Pinho (2010) point out, children tend to answer according to the expectations they have toward the person who is asking. Nurmoja (2005) and other authors (e.g. Bruck and Ceci 1999; Candel et al. 2004) add that the suggestibility decreases with age, thus, the younger children are usually more suggestible. Therefore, it is important to analyse and discuss the results taking these issues into account. To deeply analyse the opinion and usage habits of this group of students, it will be desirable to further explore this topic focusing specially on this level of education.
However, despite the confusion and unawareness about the recommendations expressed in the first question, the participants seem to agree with the presented statements about the usefulness of recommendations in general and especially regarding finding interesting content. It is important to look at the first-grade students’ values carefully and take into account the aforementioned suggestibility issues.
Regarding the usage habits and users’ opinions about the usefulness of SAPO Campus recommendations, the participants seemed to find (often or very often) interesting people and content with the SAPO Campus recommendations. From the correlation study, a significant and negative correlation was found between finding and following people suggested by the recommendations and the level of education. This means that the participants of lower educational levels find and follow people through the recommendations more frequently than the participants of higher educational levels. They tend to follow progressively fewer people through the recommendations as the level of education increases. This may denote that younger learners are not as aware as the older learners about their own interests (personal, social, or even learning interests) and as they progress in the learning process, they progressively realise and define their interests, being more selective when they are building and developing digital networks and spaces based on them.
Nevertheless, when we asked the participants to present some real cases in which SAPO Campus recommendations were useful, recommendations of people were the most frequently mentioned, closely followed by content recommendations. Percentages are still relatively low, maybe because it was an open-response question and it was at the end of the questionnaire.
The results of this study seem to indicate that some participants, and maybe due to the fact that the recommender system is not a core system of the platform, have presented some misconceptions and unawareness about it. This lead us to think that it will be important to redesign, optimise and rethink the integration of the system in the platform (e.g. the inclusion of a new tutorial step explaining the existence and the usefulness of the system) because, although the participants have presented some misconceptions about the recommender systems, they seem to simultaneously recognise their importance and usefulness to find interesting people and content.
Also, due to the interest that the badging system is currently raising (see Santos et al. 2013) and taking into account the potential of recommendations in promoting the interaction and the involvement within the SAPO Campus community, recommendations of other types of content, specially badges the user can win, are likely to be introduced at SAPO Campus, in order to explore the interest that both features seem to promote.
For privacy issues, the SAPO Campus recommender system only recommends content shared from users of the same institution. The system would become more interesting if it was possible to recommend content and people from other institutions or even content from other sources, outside the platform. We think that it could be an important and necessary improvement that we will have to further consider in order to make the system more relevant and useful.
The authors acknowledge University of Aveiro and SAPO for the scientific, financial and technical support to the SAPO Campus project and the SAPO Labs at UA R&D activities. This work is part of the Shared Personal Learning Environments (ShaPLE) project that was supported by the Foundation for Science and Technology (Fundação para a Ciência e Tecnologia – FCT, Portugal, ref. PTDC/CPE-CED/114130/2009).
1. amazon.com.
2. easyrec.org.
Almeida, S., Pedro, L. & Santos, C. (2013) ‘O sistema de recomendaç[otilde]es da plataforma SAPO CAMPUS: definiç[otilde]es, caraterísticas e trabalho futuro (The SAPO Campus recommender system: definition, characteristics and future work)’, Challenges 2013: Aprender a qualquer hora e em qualquer lugar, Braga, pp. 205–214.
Bruck, M. & Ceci, S. J. (1999) ‘The suggestibility of children’s memory’, Annual Review of Psychology, vol. 50, pp. 419–439. PubMed Abstract| Publisher Full Text
Candel, I., Merckelbach, H., Houben, K. & Vandyck, I. (2004) ‘How children remember neutral and emotional pictures: Boundary extension in children’s scene memories’, American Journal of Psychology, vol. 117, no. 2, pp. 249–257. PubMed Abstract | Publisher Full Text
Costa, A. & Pinho, M. S. (2010) ‘Sugestionalibilidade interrogativa em crianças de 8 e 9 anos de idade (Interrogative suggestibility in 8 and 9 year old children)’, Análise Psicológica, vol. 1, no. XXVIII, pp. 193–206.
Deshpande, M. & Karypis, G. (2004) ‘Item-based top-N recommendation algorithms’, ACM Transactions on Information Systems, vol. 22, no. 1, pp. 143–177. Publisher Full Text
Drachsler, H. (2009) Navigation Support for Learners in Informal Learning Networks, Centre for Learning Sciences and Technologies, Open Universiteit Nederland, Heerlen, Netherlands.
Drachsler, H., Hummel, H. & Koper, R. (2008) ‘Personal recommender systems for learners in lifelong learning networks: the requirements, techniques and model’, International Journal of Learning Technology, vol. 3, no. 4, pp. 404–423. Publisher Full Text
Gemmis, M., et al., (2009) ‘Preference learning in recommender systems’, Paper presented at the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Bled, Eslovénia.
Lee, W. (2001) ‘Collaborative learning for recommender systems’ 18th International Conference on Machine Learning, Williamstown, MA, USA, pp. 314–321.
Lichtnow, D., et al., (2006) ‘O uso de técnicas de recomendação em um sistema para apoio à aprendizagem colaborativa (The use of recommendation techniques in a collaborative learning support system)’, Revista Brasileira de Informática na Educação, vol. 14, no. 3, pp. 49–59.
Manouselis, N., et al., (2011) ‘Recommender systems in technology enhanced learning’ in Recommender Systems Handbook, eds. F. Ricci, L. Rokach, B. Shapira & P. B. Kantor, Springer, US, pp. 387 415.
Mödritscher, F. (2010) ‘Towards a recommender strategy for personal learning environments’, Procedia Computer Science, vol. 1, no. 2010, pp. 2775–2782. Publisher Full Text
Mödritscher, F., et al., (2011) ‘May I suggest? Three PLE recommender strategies in comparison’, The PLE Conference 2011, Southampton, UK.
Nurmoja, M. (2005) Interrogative Suggestibility, Trait-Related and Morphofeatural Characteristics of Human Phenotype, Dissertation Thesis, University of Tartu, Estonia.
Pedro, L., et al., (2012) ‘Building a shared personal learning environment with SAPO Campus’, The PLE Conference 2012, Aveiro.
Santos, C., et al., (2013) ‘Students’ perspectives on badges in educational social media platforms: The case of SAPO Campus tutorial badges’, IEEE International Conference on Advanced Learning Technologies, Beijing, China, pp. 351–353.
Santos, O. C. & Boticario, J. G. (2010) ‘Modeling recommendations for the educational domain’, Procedia Computer Science, vol. 1, no. 2010, pp. 2793–2800. Publisher Full Text
Saz, A., et al., (2011) ‘The construction of knowledge in personal learning environments. A constructivist perspective’, The PLE Conference 2011, Southampton, UK.
Siemens, G. (2008) ‘Collective or connective intelligence’, Connectivism: A Learning Theory for Todays Learner, [online] Available at: http://connectivism.ca/blog/2008/02/