A portal of educational resources: providing evidence for matching pedagogy with technology

The TPACK (Technology, Pedagogy and Content Knowledge) model presents the three types of knowledge that are necessary to implement a successful technology-based educational activity. It highlights how the intersections between TPK (Technological Pedagogical Knowledge), PCK (Pedagogical Content Knowledge) and TCK (Technological Content Knowledge) are not a sheer sum up of their components but new types of knowledge. This paper focuses on TPK, the intersection between technology knowledge and pedagogy knowledge (cid:2) a crucial field of investigation. Actually, technology in education is not just an add-on but is literally reshaping teaching/learning paradigms. Technology modifies pedagogy and pedagogy dictates requirements to technology. In order to pursue this research, an empirical approach was taken, building a repository (back-end) and a portal (front-end) of about 300 real-life educational experiences run at school. Educational portals are not new, but they generally emphasise content. Instead, in our portal, technology and pedagogy take centre stage. Experiences are classified according to more than 30 categories (‘facets’) and more than 200 facet values, all revolving around the pedagogical implementation and the technology used. The portal (an innovative piece of technology) supports sophisticated ‘exploratory’ sessions of use, targeted at researchers (investigating the TPK intersection), teachers (looking for inspiration in their daily jobs) and decision makers (making decisions about the introduction of technology into schools).


Introduction
In the past decade, a large number of repositories of educational resources have flourished on the Internet. Some (or most) of them are open, while others require a fee. Some of them are meant for teachers, others for students and families. They cover all school grades, from pre-school up to higher education, and all possible subjects. Examples are MERLOT, CAREO, Wins-Online, Orange Grove Repository (OER, 2013), Florida's digital repository, the (Khan Academy, 2014) and (BrainPOP, 2014).
A common feature of these repositories is what may be defined as a 'contentcentric' approach. Emphasis is first and foremost on what the educational resource is about (e.g. 'volcanoes'), next on what subject it is related to (e.g. 'science') and finally on what school grade it could be applied to (e.g. '3rd grade'). This organisation is meant to answer practical needs such as 'how can I introduce volcanoes to my 3rd graders?' (by a teacher) or 'how can I learn more on volcanoes?' (by a student). But, a number of crucial needs such as 'how can I improve collaboration within my class?' or 'how can I use digital storytelling to foster media literacy?' (by teachers) or 'what impact are technologies having on education?' (by a district manager) are not addressed at all.
To answer questions like the ones above, this paper proposes a novel approach in which technology and pedagogy take centre stage. A portal, backed up by a repository, gathering almost 300 educational experiences involving the use of technology has been built. Among other features, technology factors involve types of hardware, types of software, ways of using HW (Hardware) and SW (Software) and types of activities. Pedagogy factors involve organisation (heterogeneous groupwork, homogenous group-work, role-playing, individual assignments, etc.), inclusion problems and solutions (hospitalised children, disabled students, immigrants, disaffected students, etc.) and educational benefits (cognitive benefits, enhanced motivation, improved relationships, etc.). The core idea is to encourage users to look at the experiences not because they cover a specific subject (as in the content-centric approach) but because they face specific pedagogical issues and/or make use of specific technologies. Figure 1(A and B) shows how the portal looks like.
The new paradigm has been designed and implemented in a portal, backed up by a repository that gathers 274 educational experiences from pre-school to high-school (see Figure 1). Each educational experience is tagged through a complex taxonomy encompassing more than 30 parameters ('facets') and almost 200 'facet values' ( Figure 1B). The repository and the portal were designed and implemented in the frame of L4ALL (Learning For All), an Italian national research project set up to investigate in depth the impact of technology on teaching and learning practices (Ferrari et al. 2012).
The description of an educational experience includes a detailed analysis based on TPK (Technological Pedagogical Knowledge) factors and various kinds of documents, including audio files of interviews with the teachers, transcripts, materials produced during the educational activities, multimedia presentations and videos. Three types of users are envisioned: teachers, researchers and decision makers in the education field. Preliminary evaluations have been carried out via focus groups with teachers and via experimental research with scholars, investigating seven different issues related to TPK. District managers (at a regional level in Italy) have shown great interest in the portal. The data show high levels of interest and satisfaction. The main request by users is 'for more', in terms of the number of experiences available.
The paper is organised as follows. First, the background of the TPACK (Technology, Pedagogy and Content Knowledge) model and an overview of existing repositories of educational resources is provided. Next, the repository and the portal are described, starting with requirements and ending with implementation. Then, the evaluation is introduced, leading to a final section on conclusions and future steps.  Background and the state of the art TPACK model The TPACK model/framework (otherwise known as the TPCK model) 1 stands for the 'technology, pedagogy and content knowledge' model. The model was introduced by Mishra and Koehler (2006) following the insights of Shulman (1986aShulman ( , 1986b, who in the 1980s had invented the PCK (Pedagogical Content Knowledge) model. The PCK model advocated the need to consider the intersection between pedagogy and content knowledge and train teachers accordingly. The TPACK model follows in the footsteps of the PCK model and advocates the need to consider the intersections between TPACK in order to effectively train 21st century teachers Mishra 2005, 2008;Koehler, Mishra, and Yahya 2007;Mishra and Koehler 2005).
The TPACK model has witnessed enormous success. It would be beyond the scope of this paper to account for all the various implementations, interpretations and developments it has fostered in only a decade. As of June 2013, Mendele (2013) listed 466 academic papers on the TPACK model. A good literature review can be found in Voogt et al. (2013).
Among the various trends of research that stem from the TPACK model, the one of specific interest to this paper discusses the interrelation between the TPACK components, arguing in favour, or against, the relative independence of each ingredient. As Voogt et al. (2012) point out, three different understandings of the TPACK model as a concept have been developed: TPACK as an extension of PCK (Cox and Graham 2009;Niess 2005), TPACK as a unique body of knowledge (Angeli and Valanides 2009;Archambault and Barnett 2010) and TPACK as the interplay of three different knowledge domains (Graham 2011;Koehler and Mishra' 2005, 2008, Koehler, Mishra and Yahya 2007. In the first case, TPACK is seen as the integration of subject matter knowledge with technology, teaching and learning knowledge (Niess 2005, p. 510). In other words, TPACK is considered to be an evolution of PCK (Cox and Graham 2009). The integration of these three domains is interpreted as a way to teach using technology (Voogt et al. 2012). In the second understanding, what may be defined as a transformative view of the TPACK model is adopted. For example, Angeli and Valanides (2009) argue in favour of TPACK as a distinct body of knowledge that can be developed and evaluated on its own. In the third understanding, Mishra and Koehler put forth a different interpretation. They see TPACK not as an enhancement of PCK nor as a distinct body of knowledge but rather as a progressive understanding of TK (Technological Knowledge), PK (Pedagogical Knowledge) and CK (Content Knowledge) and all their intersections, leading to seven distinct bodies of knowledge (Koehler and Mishra 2008). Cox andGraham (2009) andNiess (2011) underpin the difficulties that such a conceptualisation, with its seven intersections, brings about. The model is further complicated by the fact that context is added by Koehler and Mishra as a fundamental part of the framework, as an element requiring deep understanding on the teacher's side. Koehler and Mishra's position can be defined as an integrative view of TPACK.
With respect to the integrative view of TPACK, it can be observed that the description of an educational experience can address different aspects, including how it was devised, how it was implemented, what was important when it was conceived and what surfaced as important after completion. In addition, the interplay between the different components of the TPACK model can have several interpretations. A teacher, for example, may start with the idea of using tool 't' and then decide that for her goal the best pedagogical approach is approach 'p'. Once the educational experience is made available in a public portal (such as the one presented in this paper), it may happen that another teacher takes a different path. Investigating experiences with approach 'p', he/she may be led to take tool 't' into consideration. In other words, a good educational portal should be able to support several different conceptual pathways, possibly ones not even corresponding to the original goal of the teacher who created the experience in the first place.

Educational repositories
In the mid-1990s, relatively simple learning objects were made available informally as instructors shared syllabi, lesson plans and learning activities. Later, more complex and/or topic-specific repositories came into existence as museums, journals and magazines, educational television productions and other actors placed content on the Web and encouraged it to be used for educational purposes. Many institutions developed learning activities around the objects (thus creating larger, 'second-order' learning objects) and made them available from their websites.
Afterwards, websites became populated by learning objects from all disciplines but, instead of housing the objects on their own servers, they often linked them to the original sites where the resources were made available. Large repositories of learning objects are now available on sites such as MERLOT (Schell and Burns 2011), a leading archive for higher education that provides tools for user comments and peer reviews; CAREO (Neven and Duval 2012), a collection of Web-based multidisciplinary teaching materials; and Wisc-Online, which houses the online resources supporting Wisconsin's technical colleges.
Examples of repositories of learning objects for use in specialised training include those from the aviation industry for which the Aviation Industry Computer-based Training Committee helped in setting standards. Another repository is the (NMC, 2007) Learning Object Initiative, thanks to which many industry and military issues have been resolved. Other early examples in this area include the Educational Object Economy (Roschelle et al. 1999) and Ariadne (Forte et al. 1997). More recently, spurred by the development of the 'learning object metadata' standard of the IEEE Learning Technology Standards Committee (LTSC), numerous initiatives have been launched in academic and corporate contexts (Neven and Duval 2012;Primo, Vicari, and Bernardi 2012). Several standards have also been developed in the field of e-learning systems (ELS) as reported by Cos Garcia (2011).
Learning object repositories can be difficult to navigate, and the educational material difficult to integrate into online courses. Schoonenboom, Sligte, and Kliphuis (2009) observe that the literature on the reuse of learning materials has largely focused on the development of materials. The authors developed guidelines that support staff and/or management in cases of (un)successful reuse of existing digital materials and provided methods for teachers in higher education in such cases. They collected existing strategies from 11 literature studies and developed new guidelines from 19 (mainly Dutch) case studies. Lukasiak et al. (2005) present the Smart Learning Design Framework that supports sharing, reuse and adaptation of learning material via a metadata-driven philosophy enabling the technicalities of the system to be imperceptible to the author and consumer. The system proposes the use of pedagogically focused metadata to support and guide the author and to adapt and deliver the content to the targeted consumer. Even though many educational 5 technologies and services have been provided to enhance educational pedagogy, in contrast with what was expected, a drop in participants and subscribers to these educational technology resources among students and instructors was recorded in many different studies (Cuthrell and Lyon 2007;Jones 2009;Willems 2007). These studies have also identified that the reason for such a drop was that a 'one-size-fitsall' approach was used instead of an approach that provided educational courseware for students using all available educational technologies (Cuthrell and Lyon 2007;Matar, Khwaldeh, and Hunaiti 2007). Many adaptive ELS were created to satisfy students' needs and learning styles using different approaches, but their focus and intent was not on unified learning object repositories (Aroyo, Mizoguchi, and Tzolov 2003;Dolog et al. 2004). Andrew, Farrow, and Cooper (2012) argue that online resources for education not only offer opportunities for those with disabilities but also raise challenges regarding how to best adjust resources to accommodate accessibility. The authors carried out a study within the EU4ALL (European Unified Approach for Accessible Lifelong Learning) EU-funded accessibility project at the Open University, United Kingdom, in parallel with studies at three other European universities, developing systems and services to give personalised access to online materials that include alternatives.
To overcome these limits, unified E-learning repositories (UER) were used for simplifying the tasks of sharing learning contents and resources between different universities through a unified access point. Such systems are capable of providing management information regarding contents and users; in addition, they give instructors and course authors the ability to reuse learning contents and to assemble them into courses (Hatala et al. 2007). Different initiatives have been implemented to support the idea of UER among organisations and educational institutions. The first one is called the centralised LCMS (Learning Content Management System) approach, where a central LMS (learning management system) is used to provide the shared courses as ready-made packages that can be used by instructors (Matar, Khwaldeh, and Hunaiti 2007). Such an approach can be performed using any available LMS such as Moodle, ATutor, Blackboard and WebCT. The second one is called the networked approach (NA). The focus of NA is on the learning objects level rather than on ready-made courses, so that each instructor can use learning objects and structure them into courses (Pouyioutas and Poveda 2005). An example of such a system is the learning object network with CAPA. (LON-CAPA, 2013) provides unified e-learning through a dedicated network; in order for universities to use the system, they should structure their servers to be a part of the learning network. LIONSHARE is another UER-oriented platform that uses NA for sharing resources between learners. SPLASH is an NA-oriented system that is actually used as a distributed learning objects repository/metadata tagging tool. SPLASH was developed as a part of the Portal for Online Objects in Learning (POOL) project, which is a consortium of several educational private and public sector organisations aimed at developing an infrastructure for learning object repositories (Perkins 1995). POOL addresses the issues of building such architectures including metadata, software and hardware, and bootstrapping the system with initial content. It also makes tools available for download to help set up similar infrastructures elsewhere and to connect them to POOL.
The authors observe that the tendency of current repositories is to retain content in the form of a broad mix of text documents, videos, audio files and graphics (EDRENE 2009). It also emerges that a few repositories include non-digital materials (e.g. text books). A little less than a third of repositories surveyed have a mix between free and commercial material. What is relatively clear is that educational repositories are mainly created to share learning objects, often characterised by metadata or ready-made courses, intended as an organised set of learning resources related to a specific discipline. However, they largely fail to provide a whole, fully described and reproducible learning experience that can clarify when, where and how materials, digital or not, were used; how the learning process was organised; what educational goals were planned; which educational benefits were generated and what the role of the technology was.
In order to schematise and understand the circumstances and the outcomes around an educational experience, studies have been conducted and conceptual and meta-learning frameworks have been proposed. The study in So (2012) endeavours to understand how teachers make use of online resources to design learning environments that are motivating and cognitively engaging to their learners. The final aim is to propose a resource-based e-learning environment (RBeLEs) framework, based on five teachers' authentic experiences in planning, designing, implementing and reflecting on such learning environments. In-depth interviews were conducted to collect more information. In-depth interviews have the primary advantage of providing much more detailed information than other data collection methods such as questionnaires, and they are particularly useful when detailed information about a person's thoughts and behaviours is wanted (Boyce and Neale 2006). The five participating teachers were individually interviewed face-to-face for about 30 minutes before and after lesson implementation. All interviews were semi-structured, audiorecorded, and transcribed. In the pre-lesson interviews, the teachers were encouraged to describe how they had created the teaching designs and how they had planned to implement them in the classroom. The post-lesson interviews, in contrast, focused on understanding how the teaching designs were actually implemented, and what the pupils' motivations and cognitive engagements were in the lessons.
Frameworks for supporting knowledge capture at source (Verhaart 2002) and collaborative knowledge management systems (Maher and Kourik 2008;Nunamaker 2001) have been proposed in different fields, for example in e-Government (Zhou 2008). The need to assess quality levels in engineering educational programmes led ABET (Accreditation Board for Engineering and Technology) to issue the Engineering Criteria 2000 (EC2000) standard (Lattuca et al. 2006). EC2000 shifted the basis for accreditation from inputs, such as what is taught, to outputs, what is learned. The new criteria specify 11 learning outcomes and require programs to assess and demonstrate their students' achievement in each of those areas. In Besterfield-Sacre et al. (2000), a framework based on Bloom's taxonomy (Bloom et al. 1956) has been developed for better specifying the EC2000 outcomes. Bloom's taxonomy is based on six levels of the cognitive domain: knowledge, comprehension, application, analysis, synthesis, and evaluation. To meet EC2000 assessment criteria, information from all university constituents needs to be routinely collected and tracked longitudinally. The management of EC2000-related information through a data warehousing approach is the main focus of Ingham (2000); the data warehouse approach allowed the management of 100,000 student records covering a 10-year span, as well as 30,000 course records over a similar period of time. This has also led to the advantage of supporting the assessment process.
The OER is Florida's digital repository for instructional resources. It provides an environment for educators to search for, use, remix, share and contribute educational resources. The repository can also import resources directly from existing LMSs (e.g., Blackboard, Desire 2 Learn, Canvas). The diverse collections available for searching contain a wide range of resources, including open textbooks. Resources are free for educational use but may be protected through various copyright statements associated with each resource. The main audience of the repository is Florida public higher education faculty and staff, but Florida K-12 educators, educators outside of Florida, and guests can also contribute resources.
A growing number of repositories explicitly meant for students and families are becoming available for all levels of schooling. The Khan Academy (KHAN) is a not-for-profit organisation with the goal of bringing innovation to education by providing online materials and resources available free of charge for anyone, anywhere. Students can make use of an extensive video library, interactive challenges and assessments from any computer with access to the Web. Coaches, parents and teachers can have different levels of visibility regarding what their students are learning and doing with the Khan Academy resources. The BrainPOP repository (BrainPOP) encompasses more than 650 curricular topics designed to actively engage students from grades 3 and up. It creates animated, curriculum-based content that supports educators and students in school, at home, and on mobile devices. Resources include movies, quizzes, games, mobile apps, experiments, activity pages and other kinds of learning content covering hundreds of topics within math, science, social studies, English, technology, arts and music and health. All content is aligned to and searchable by US state standards, including Common Core State Standards. PoliLab Kids (PoliLab, 2013) is an Italian initiative that released an online repository of educational videos (lasting 2Á3 minutes each) related to various areas (science, math, technology, etc.). Each video is accompanied by a rich set of educational materials including quizzes and a teacher and parents' guide.
With the aim of driving and supporting the implementation of innovative technological experience at school, a number of technological and conceptual frameworks have been proposed (Bucciero et al. 2011;Di Blas et al. 2012;Vergallo 2012). However, they do not include a sharing platform like a repository, so they lack structuring of the material that is aimed to make it easier to share and reuse the experience. Also, it is evident that adopting well-designed collaboration process models (Barchetti et al. 2011) for supporting empirical research about the role of Information and Communication Technologies (ICT) and its influence on pedagogy-related concepts (various kinds of implementations, inclusion problems and solutions, envisioned benefits, etc.) could ease information-intensive research activities and improve results. Starting from these considerations, we defined the main requirements of the repository's collaboration and data model as will be described in the next section.
The repository and the portal The repository and the portal described in this paper were designed and implemented in the framework of the research project, L4ALL, and aimed at detecting how ICT is affecting learning processes at school. The project, taking an evidence-based approach, investigated a large number of educational experiences where ICT had played a prominent role.
Each experience generated a fair amount of material (interviews, analysis, documents, lesson plans, etc.). This sizeable body of documentation was collected with several aims in mind: to foster research by scholars both within the project itself (L4ALL) and the community of researchers at large, to inspire teachers through the evidence of works and experience by other teachers, and to support decision makers in the field of education.
In order to make the reader better understand what kinds of user experiences we mean to support, let us introduce two hypothetical scenarios of use.

Scenario 1
A scholar is investigating the relation between different kinds of group work in digital storytelling activities and the educational benefits achieved by the participants. She accesses a portal where hundreds of educational experiences are gathered. Each educational experience is accompanied by a rich set of materials (documents, presentations, videos, etc.) and is classified according to a set of 30 facets (technology used, benefits, organisation, inclusion issues, etc.) and a high number of facet values (for the facet 'benefits', cognitive benefits, relational benefits, etc.). First of all, she selects 'digital storytelling' as a technology-based activity. Then, she selects all the kinds of group work (there are four: homogenous, heterogeneous, same-skill and different skill group work) within the facet about organisation. She thus creates S1, a subset including all experiences that combine storytelling with all kinds of group work. S1 shows that digital storytelling and group work generate a number of benefits, among which relational benefits stand out as prominent. She now wonders whether this activity (digital storytelling implemented through group work) may have an impact on inclusion issues, but she wants to leave aside disability since she is not an expert in that field. She just wants to focus on disaffected students, immigrants and students with a poor background. She thus gets to the facet 'inclusion issues' and excludes 'disabilities'. She gets S2, another useful subset that she saves for further investigation.

Scenario 2
A high-school teacher in humanities wants to improve her use of ICT at school, and therefore accesses a portal like the one described above. First, the teacher browses the initial set (S0). She takes a look at the facet about technology, getting an idea of what technologies are most popular. She also takes a look at the 'organisation' facet, to see what kinds of pedagogical implementations (heterogeneous group-work, role-play, etc.) are used. Then she selects interactive whiteboards and tablets within the facet about technology (creating subset S1). She has an interactive whiteboard in her class but she seldom uses it. Moreover, she has been told that the school principal will buy a tablet for each student, so she needs to get prepared for this change. She further refines her exploration by selecting 'motivational benefits' within the facet about benefits and 'group-work' within the facet about organisation (creating subset S2). There are many disaffected students in her class, so it may be a good idea to start trying to involve them by getting the help of the performing students. Now, she notices with dismay that in S2, technical disciplines are overrepresented. Therefore, she goes back to S1 and adds the criterion that humanities be the overall discipline. She also excludes 'immigrants' as a problem within the facet about inclusion issues 9 (in the profile of S1 'immigrants' is a major issue, but there are none in her class). She gets S3, a small, interesting set. She saves the contents to investigate them at a later time.
In order to support the aims of the project exemplified by the above scenarios, two developments were needed: A repository: to collect (in an organised fashion) all the materials available for each experience, along with managerial information (dates, who the author is, etc.). Users are the staff of the project.
A portal: to provide effective access to the educational experiences. Users are scholars (both within and outside the project's team), teachers and decision makers.
Although they share a fair amount of information, the repository and the portal (given the different aims and the different types of users) required different design concepts and different implementation strategies that are described in what follows.

The repository
The repository was co-designed and implemented by the GSA-Lab (Graphics and Software Architectures Laboratory) of the University of Salento and the HOC-LAB (Hypermedia Open Center Laboratory) of Polytechnic of Milan. Its main aim was to store educational experiences and ease the collaboration among staff. The repository was the 'backstage'. In other words, external users (researchers, teachers, decision makers) did not access it nor did they have any perception of its existence.
The design of the repository was driven by two inputs, the 'static model' of the information to be stored and the dynamic aspects of the workflow (for experience collection and analysis). The latter was quite complex and distributed among various actors. If the requirements for the 'static model' were relatively easy to elicit, the analysis of the workflow was a far more difficult task. The staff of the project were altering their methods as the project proceeded; therefore, the workflow was continuously redefined. At the end of the project, the following were the main roles that surfaced: Interviewer: she carried out the interviews and produced all the materials concerning an experience (as described in the next section).
Editor: she reviewed the materials produced by the interviewer, improving the overall quality. In addition, she 'tagged' the experience using the proper facet values.
Coordinator: she orchestrated and supervised the work of interviewers and editors. Furthermore, she approved publication of the educational experiences in the public portal.

Experience data model
The objects stored in the repository were not simple files or learning contents but instead were educational experiences. From a technical perspective, an experience is a collection of resources (files and descriptions) related to the implementation of a technology-based activity at school. The experience data model defines the type, number and structure of such resources within an experience (see Figure 2).
A large variety of materials can be associated with an experience: Interview(s): audio file(s) of interviews of the teacher (before and after the experience).
Interview transcripts: the interviews are transcribed and filtered, that is, irrelevant parts are removed and more colloquial sentences are summarised in order to make the transcript more readable.
Abridged interview (optional): if the interview includes relevant or interesting sections, the interviewer can decide to extract such parts in a specific document.
Contents: any other useful resource, such as materials produced in the course of the experience (e.g. multimedia presentations).
FEE (FEatures Extraction) forms: FEE forms are the schemas that structure the relevant information hidden in a set of resources about the experience (characteristics of the whole class, the social/cultural/economic context of the school, the rationale behind the experience, etc.). In L4ALL, there were three different forms: one about expectations, one about results and another about a comparison between the two (performed by a scholar).
Each interviewer had to manage the lifecycle of the experiences assigned to him or her within the repository. A minimal set of metadata (for each of the objects) was added both for fast identification and for managing purposes. So, it was easy to find items such as what was done on a given day, how many interviews the repository held, which FEE forms were missing and what the job done by a specific interviewer was.

Experience collection process
The collection and sharing of educational experiences consisted of four subprocesses: interview, review, tagging and publishing (see Figure 3). There were two  types of interviews, one about expectations and one about results. The 'expectation interview' was taken before the educational experience began to investigate what the teacher was looking for. The 'results interview' was taken after the educational experience was completed. The audio of the interviews was recorded, then a transcript was made and the writing was tightened to eliminate what was not relevant. After that, the three FEE forms were filled in: expectation, results and a comparison between the two. Next, the editor reviewed all the materials and tagged the experience, interpreting the main findings of the FEE forms.

Implementation
Flexibility in implementation was a key requirement, since the repository actually underwent a continuous cycle of designÁimplementationÁevaluationÁredesign as the project moved on and new needs surfaced. Alfresco Enterprise Content Management (ECM) software (Alfresco software 2013) was selected. The key technical features driving this choice were the following: Alfresco is a lead ECM platform, widely adopted by both private enterprises and public corporations. Since the first release in 2005, it has evolved adding several features: Web content management, versioning, records management, workflow management, and so on.
The Alfresco Community Edition is free and open source. Alfresco's source is freely downloadable and customisable. Moreover, the Alfresco community is very active and lots of add-ons are available.
In Alfresco, the repository's interfaces are highly decoupled from the core document management. It is distributed as composed by two main Web applications, Alfresco Explorer, which provides main administration capabilities, and Alfresco Share, which provides a rich customisable collaboration environment.
Alfresco provides built-in user profiling management and configurable workflow management. Five versions of the repository were released without a significant re-working effort.
Alfresco widely supports open standards that allow a fast content transfer with other ECM environments.  Figure 4 shows the high-level architecture for the repository. The experiences request broker is an architectural element that can arrange the stored experiences and dispatch them according to three different perspectives (interfaces). In our architecture, the user views were decoupled from the experience management (insertion, updating). Figure 5 shows the detailed architecture of the repository. It includes the internal interface, the experience's data entry built on top of the Alfresco ECM and the search interface. The internal interface was used by the interviewers and was deployed in Alfresco Share as a custom extension. This was structured according to three layers: The persistence layer used the file system to store the experiences materials, while the database was needed for storing user information, user groups, content categorisation, and so on.
The business logic layer allowed the development of custom extensions to manage documents, users, groups, categories, permissions, and so on.
The integration layer let user interfaces interact with the business logic.
The search interface mainly intertwined tag-based, full Boolean, weighted and visual search strategies, to provide a rich experience search engine. The tagging experts used both the internal and the search interface. They downloaded the experiences to be tagged from the Alfresco repository, used QSR NVivo 9 (QSR International 2013), a third-party tagging software installed on their desktop environments, then exported the tags in Microsoft Excel format and uploaded xls files to the search interface through an ad-hoc view.
It was crucial to let the repository and the portal remain absolutely independent, to allow both the interfaces to evolve separately and to keep one of the subsystems online while the other was in maintenance/update mode. Developing an Alfresco extension, we defined the experiences data model as a common language adopted by the two interfaces. Moreover, we configured the interaction between the two interfaces by minimising their mutual relationship.
The Alfresco extension we developed included an experience export mechanism for transferring the data from the internal (the repository) to the external interface (the portal). The coordinator partner performed the export task from time to time, for example, when a massive data update was done. It allowed transferring of the resources' links, and of the content metadata as well, for each completed experience.

The portal
The portal was designed and implemented by HOC-LAB of Polytechnic of Milan, according to the following requirements: to support an innovative way of accessing rich information, emphasising exploration and serendipitous discovery over search, to accommodate 'niches' of users (scholars, experienced teachers, novice teachers, managers, experienced researchers, novice researchers, etc.), to emphasise pedagogy and technology (and their intersection) rather than the specific content (subject matter) to which they had been applied, to combine very synthetic information with a large variety of additional content (audios, videos, documents, etc.) without any specific constraint, to combine at-a-glance understanding with the possibility of focusing upon details, and to allow different scenarios of use: supervision, decision making, focused research investigation, broad research investigation, generic inspiration, focused inspiration, and so on.
The portal was implemented using standard Database Technology and an 'experience engine' with a novel interface. The portal was made public in December 2012. It currently hosts about 274 educational experiences, carried on in Italian schools, where the use of technology has been crucial.
In order to provide an effective exploration, a rich taxonomy of more than 30 facets and almost 200 facet values was developed. The taxonomy was revised several times since it was crucial to combine powerful exploration with understandability and usability. Thanks to an innovative HCI (Human Computer Interaction) approach (and novel supporting technology), the taxonomy is at the same time a tool for navigation and exploration as well as a tool for feedback. Once a value is selected within a facet, all the other values in all other facets 'react' to the choice. For example, if primary school is selected as school level, all the other facets (e.g. the technologies used) modify their values accordingly (so, for example, a scholar can see what technologies are used at primary school level). Table 1 shows the list of the main facets. They range from factual elements (e.g. which tools are used) to interpretative ones (e.g. the impact generated by the adoption of a technology). Table 2 shows an example of facet values for the facet 'activities'. The user's at-a-glance understanding of a dataset is facilitated by numerous visualisation strategies. For example, all the elements of a set can be visualised as a 'mosaic' where shapes and colours are meaningful, as elements over a geographical map emphasising the geographical location of the experiences (see Figure 6) or as thumbnails (to support visually driven selection). Also, within the facets, different visualisations are possible, including absolute numbers, percentages and word clouds. Boolean logic, such as conjunction 'and', disjunction 'or', 'exclusive or', and negation 'not', support the use of the portal (see Figures 1 and 7).
A 'shopping cart' allowed the user to save the results of an exploration. Through proper commands, investigations could be saved, stored and shared, or 'exported' as PDF files.
If a specific experience was of interest, the user was provided first with an abstract and then with all the available materials, including documents, audio files and presentations.

Evaluation
The evaluation of the L4ALL Portal is a difficult task for several reasons, including the variety of user communities, variety of aims, variety of tasks and variety of scenarios. In the following sections, we describe two specific qualitative evaluations: focus groups with teachers and monitored experimentation with researchers. Other larger quantitative evaluations are planned with teachers for 2014. School level at which the educational experience took place, for example, pre-school, primary school. Rationale: the school level has a crucial impact on the most relevant factors in education Á implementation, teacher's attitude towards inclusion, envisioned benefits, etc.

Monodisciplinary versus Multidisciplinary
Whether the project is related to one or more than one subject.

Subject
Areas of study and subjects involved. Main distinction between humanities and STEM (Science, Technology, Engineering, and Mathematics) subjects. Further values list all possible subjects (literature, art, history, chemistry, etc.). Format A format is a pre-defined technology-based educational activity that implies, though they are not strictly mandatory, specific technologies, specific implementation procedures, a workflow of activities, etc. For example, 'PoliCultura' (i.e. a specific format for multimedia storytelling deployed by Politecnico di Milano). Initiative An initiative is a broad 'umbrella' (a ministry-funded project, a course, etc.) under which different formats can be found, for example, 'Classes 2.0', a ministry-funded initiative to introduce interactive whiteboards into some selected Italian schools.

Geographic location
Where the experience took place (region, county, etc.). Rationale: these data are crucial for statistics about technology use in education, for example, for comparing southern with northern regions.

Social context
The socio-economic level of the students' families Á 'low', 'average', etc. Rationale: the learning attitudes and performances of the students are oftentimes influenced by the context.

Cultural context
The cultural and educational level of the students' families. The same rationale as for the social context.

Class performances
Level of the students' performances during normal school activities. Rationale: when results of a special activity are analysed, they should be compared to the typical performances of the students during normal school activities.

Class homogeneity
Level of homogeneity of the students in terms of performances. For example, they can be 'highly homogeneous' but in the sense that they are almost all poorly proficient. Rationale: it is very important for a researcher/teacher to know how the class is performing in order to understand or take inspiration from a specific experience. Hardware The most important pieces of equipment used during the experience Á . PC, iPad, video camera, etc. Rationale: The hardware components have an influence over the experience's implementation in all its aspects. For example, they may determine roles in the class according to the students' expertise. Software The most important pieces of SW used during the experience, for example, tools for image processing such as Photoshop and Gimp. Rationale: the SW influences the experience's implementation in all its aspects, just like the HW does: it may determine students' roles. The physical environment where the activities took place Á school, lab, home, social space, etc. Rationale: the location is a crucial aspect of the implementation. 'When' The hours during which the activities were carried out Á during school hours, at home, etc. Rationale: it is important to understand whether the regular school hours were enough. Curriculum Whether the experience stood within the curriculum or not, for example, an experience could be curricular or extracurricular. Rationale: often technology-based activities are not considered part of the curriculum. Programme The inclusion of the activities in the official 'programme' of the school. Rationale: activities that are officially part of the programme are often felt as more important. Design How well defined the design was at the start. For example, an experience could be 'poorly sketched' or 'sketched in detail'. Rationale: it is often the case that innovative educational activities cannot be fully designed beforehand.

Human resources
Who was directly involved in the experience's implementation Á more than one teacher, the school's technician, etc. Rationale: to allow evaluation of the effort needed to run the experience.

Other subjects
Additional subjects involved in the activities with different roles Á the principal, the class' council, families, etc. Rationale: to understand the complexity of the experience. Activities Main activities characterising the experience Á 'brainstorming', 'creation of digital media', etc. Rationale: it is only by knowing what activities the experience requires that the experience itself can be fully understood from a pedagogical point of view. Organisation How the class was organised Á 'heterogeneous group work', 'homogenous group work', 'role-playing', etc. Rationale: same as per activities; more specifically, an organisation is likely to shed light on 'inclusion' issues because it shows whether all students were really involved, to what extent and how.

Key features
The aspects of the implementation deemed to be particularly important for its success. For example, 'method', 'organisation', or 'visibility' (of the final results).

Inclusion problems
List of possible inclusion issues. For example, 'children with severe learning difficulties'.

Inclusion strategies
Possible strategies adopted to overcome inclusion issues. For example, 'personalised learning' or 'peer to peer education'.

Cognitive benefits
Benefits at the cognitive level, better understanding of the subject dealt with, capacity to make complex connections, etc.

Motivational benefits
Benefits related to enhanced motivation, increased self-esteem, increased motivation towards a specific subject, increased motivation towards school in general, etc.

Relational benefits
Benefits related to the sociological sphere Á improved relationships within the class, improved relationships with the teacher, negotiation skills, etc.

Focus groups
Using a portal such as the one provided by L4ALL is a novelty for teachers. Therefore, focus groups were chosen (over surveys) as a way to investigate issues not fully known in advance. We wanted teachers to face the novelty together, share their first impressions and, through discussions and task performances, reach a shared opinion about the tool they were presented with. Three focus groups were held in autumn 2012, in cooperation with three different research groups: HOC-LAB (at Polytechnic of Milan Computer Science School); a team at the University of Perugia (Department of Education) and another at the University of Bologna (Department of Education). On the whole, 32 teachers were involved. They were all working as teachers across all school grades (from pre-school to high-school).
The average age was 42. The three focus groups adopted a similar procedure: First, the moderator gave a general introduction to the portal and the goal of the study: what it was about, what kinds of materials could be found and what kinds of scenarios could be supported (allotted time: 10 minutes).
Then, the helper showed some of the portal's affordances: the taxonomy, the different views and the possibility of saving queries and materials (allotted time: 10 minutes).
The participants were divided into small groups of three teachers each. Select three facets in the taxonomy and see how the features of the educational experiences vary (e.g. understanding how different technologies affected cognitive benefits).
Select some experiences and explore to some extent the content (abstract, synopsis, document, audio files, etc.).  During task performance, the focus group moderator and the helper would monitor what was going on within the groups but without interfering.
The helper took notes about the teachers' actions, comments and attitudes. All the participants re-grouped again for the final discussion, prompted by the moderator's questions (allotted time: 20 minutes): What was your first reaction to the portal? Did it take long to make up your mind about how it works? Do you think that the portal is too difficult from an interactional point of view?
Do you think that the portal is too difficult from the point of view of content (i.e. the taxonomy, the materials provided, etc.)?
Would you prefer to have a simplified version of the portal? Do you think the portal is useful? Would you use the portal? The helper recorded the discussion and a transcript was made that was later boiled down to the essential comments only.
The outcome of the focus groups was highly encouraging: HCI features: after the initial difficulty of the apparent complexity of the interface, teachers became comfortable with it. Overall, they said that after a few minutes of usage, the interface became quite usable.
Experience description: teachers did like how the educational experiences were described. They said that it was not usual for them to describe experiences in that way, but they found a systematic analysis useful for deep understanding.
Browsing and serendipity: teachers quickly got very familiar with the browsing method and found the lack of focus on 'content' interesting.
TPK: teachers said that the complex taxonomy helped them to start realising the relationships between technology and pedagogical issues. Whether to simplify the taxonomy for the teachers leaving the more complex one for researchers only was one of the project's concerns, but the majority of teachers declared that after a few minutes they became 'intrigued' by all the facets and did not want a simpler version.
Size of the portal: many teachers pointed out that a larger number of experiences would add value to the portal.

Researchers
In order to evaluate the quality of the portal as a support for research activity, five research groups with different backgrounds (technology and pedagogy) were asked to use the portal for their research activities. Seven different investigations were carried out: ( Some of the reports have already been turned into scientific papers (e.g. Di Blas and Paolini 2013). The researchers were interviewed extensively and they outlined several advantages introduced by the portal: Content: researchers showed appreciation of the in-depth understanding of the experiences afforded by the portal's wide range of materials.
HCI: as was expected given the type of user, the mechanics of the interface were quickly understood.
Browsing and serendipity: the two most important methods of usage were formulating research hypothesis and verifying them and browsing around trying to 'stumble' into interesting data. Researchers declared that both methods were effective, and that the latter was a pleasant novelty.
TPK: interesting investigations were easily and effectively carried out. Facets were not just used for selection but also for feedback. For example, without even looking at the description of the specific experiences, it was possible to detect that group work (quite typical at primary school level) generates different benefits with respect to the experiences where individual assignments are used. Discovering 'properties of sets' (without looking at details) emerged as a major affordance of the portal and it has become a base for future enhancements (as will be discussed in the next section).

Discussion
The work described in this paper presents several new aspects: TPACK and Educational Portals: our contribution lies in the emphasis placed at the TPK intersection. While most educational portals are built around a school grade-content paradigm, we strive to communicate how technology and pedagogy influence each other. This is done in two ways: The description of each educational experience is mainly about the TPK intersection.
The taxonomy used to 'explore' the set of experiences is based upon TPK facets.
Sharing educational 'evidence': in the field of education, the most common pattern is that each research group creates its own experimental data (about a phenomenon of interest), collects it and interprets it. This common pattern can be criticised in several respects. First of all, it is an expensive way of proceeding. Young researchers can have difficulty with this and students cannot easily access empirical data. In addition, 'theoretical' researchers do not have data to work with. Third, the different 'interpretations' cannot be easily (and reliably) compared, since they are based on different data sets. In September 2012, in a workshop organised in Como, Italy, several experienced and world-renowned researchers agreed that sharing empirical data in education would be valuable for the community. Therefore, we advocate the advantages of making publically available empirical data about innovative learning experiences. The L4ALL portal is open in two ways. It can be expanded with new experiences fed by other research groups and any research group can use it for research and investigations.

Research in Learning Technology
Teachers are users: teachers can find in the L4ALL portal an unusual source of inspiration and, most important, an occasion for professional growth, while paying attention to TPK's intersection. They can realise that the relation between pedagogy and technology is multifaceted and complex (e.g. tablets will not save education nor they will destroy it), and that 'details' (that they can find in the portal) make the difference. This is what we think we have accomplished, to some extent. Therefore, we advocate the need for using educational portals not just as a 'hub' where information can be found but also as a way to encourage teachers to broaden their horizon and their way of planning their activities.
Researchers are users: we have indicated that evidence-based research can be carried on well by exploiting all the information gathered in the portal. The researchers found some expected phenomena in the portal, but they also serendipitously discovered new aspects and were stimulated to formulate new research directions. Therefore, we advocate that the empirical evidence should be explored in view of detecting new, unexpected, patterns and not just to prove a hypothesis that was formulated beforehand.
Decision makers are users: the Italian ministry of Education and several regional district managers showed great interest. In an open meeting in Rome, they declared that our effort was worthwhile and that we should continue expanding the portal. Therefore, we advocate the need for basing education policies (e.g. how to use technology at school) not just on anecdotal evidence, nor only on largely dry statistical data, but also on a fine-grained understanding of the issues and of the various implications. Most important is the fact that data about TK and PK, if considered separately, do not provide an adequate insight.
Exploratory portals and technology: we have created a novel paradigm for exploring 'rich data sets', that is, complex information organised via a complex taxonomy. Its development involved HCI design, user experience design and a strong technological development. Our approach is the development of a new generation of portals, supporting better investigation, exploration and serendipitous discovery. The approach will be based on a general repositoryportal model, such as the one described in this paper.

Future work
Several directions for future work are open to add to the research described in this paper: TPACK and educational portals: we are in the process of refining the understanding of how TK and PK influence each other. This will lead to a redefinition (expansion and refinement) of the taxonomy upon which the portal is based. In a forthcoming paper (in cooperation with one of the authors of the TPACK model), we are also investigating how TK and PK are 'distributed' among various actors and how they evolve within the same educational experience (Di Blas et al. 2014).
Sharing educational 'evidence': we are pursuing contacts with other research groups in Italy and with district authorities in order to expand the size and the coverage of the portal, sharing its use. The major problem is cost. Going from 300 experiences to 10,000, as it seems it would be necessary, is a cost that must be covered with a variety of financial resources.
Teachers are users: we were delighted by the positive reactions from the teachers. At the moment, we are actively using the portal as an overall resource for training teachers on the job. Researchers are users: five different groups of researchers are pursuing their activities with the portal. As it is in Italian, it is not easily 'exportable'. We are working in the direction of making it more international.
Decision makers are users: the initial reactions of decision makers were quite enthusiastic. The main problem, as mentioned before, is the size of the portal. An Italian regional district would prefer to have between 500 and 1,000 experiences related to its territory in order to have a reliable foundation for a decision. We are signing agreements with some of these districts, in order to make this possible.
Exploratory portals and technology: we are working on expanding and improving the paradigm and the user experience. This new vision (that we call EXPLORE) will provide significant enhancement to the ways we access today rich data sets. We are also moving towards applying the same technology to other application areas (such as e-tourism and e-food).