Research in Learning Technology
Supplement: ALT-C 2012 Conference Proceedings
Thomas Cochrane*
Centre for Teaching and Learning, Auckland University of Teaching, Auckland, NZ
(Received 27 February 2012; final version received 14 June 2012; Published: 30 August 2012)
Abstract
Having implemented and evaluated over 35 mlearning projects in a variety of contexts in higher education over the past 6 years the researcher is ready to share the untold secret: not all mlearning projects succeed! This article critiques three of the researcher's mlearning projects that can be classed as “failures” and compares them to successful projects to draw out critical implications for mlearning project design and implementation to avoid common pitfalls leading to potential project failure. This article uses the researcher's six critical success factors identified across the 35 mlearning projects to evaluate these three projects, and concludes that projects resulting in “no significant difference” in pedagogical outcomes are the result of attempts to shoehorn old pedagogies into new technologies. Lecturer professional development and sustained collaborative support are critical to fostering new pedagogies utilising the unique affordances of mobile devices.
Keywords: mlearning; pedagogy; heutagogy
*Corresponding author. Email: thomas.cochrane@aut.ac.nz
ISBN 978-91-977071-4-5 (print), 978-91-977071-5-2 (online)
2012 Association for Learning Technology. © T. Cochrane. This is an Open Access article distributed under the terms of the Creative Commons Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0) license (http://creativecommons.org/licenses/by-sa/3.0/) permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
http://dx.doi.org/10.3402/rlt.v20i0.19186
This article is based on the researcher's experience of longitudinal participatory action research in mobile learning from 2006 to 2012 (Cochrane 2011). While the researcher has published over 75 articles (conference papers, journal papers, book chapters and workshops) based on these projects over the years, most of these have focused upon the project outcomes that had successful impact upon pedagogy in a variety of contexts. However, the identification of key critical incidents and understanding of the benefits of mlearning have often come from critical reflection on the “mistakes” or failures of these projects that then led to the redesign and implementation of subsequent iterations. The goal of the research that surrounded these projects was to explore the potential of mlearning as a catalyst for enabling social constructivist pedagogy. Hence while the researcher acknowledges that there are many other pedagogical approaches beyond social constructivism it was the explicit chosen foundation for the research projects and therefore project success was defined as enabling a pedagogical shift from instructivism to social constructivism. The research was situated within the context of the establishment of a new institutional elearning strategy (Cochrane 2010a) named the “living curriculum” that was effectively based on social constructivism, and launched with a institution-wide “roadshow” entitled “Kaleidoscope” in July 2011 with several of the researcher's “successful” mlearning projects showcased:
Reeves (2005, 2009) argues that even when educational technology projects are described as successful often the results and impact of such projects reveal “no significant difference” on pedagogical outcomes when compared to more traditional teaching and learning approaches, because there has been no explicit design for pedagogical change within these projects. While the potential for innovation and pedagogical change afforded by mlearning is mooted by researchers, for example: (Cook 2010; Herrington et al. 2009; Laurillard 2007; Kukulska-Hulme 2010), the evidence is often ephemeral (Pachler, Bachmair, and Cook 2010). In their review of 112 innovative mobile learning projects published between 2002 and 2007, Frohberg, Goth and Schwabe (2009) found that only 5% of these projects focused upon social learning, less than 4% required higher level thinking, with 89% targeting novice learners, and only 10% facilitated user-generated content. In contrast to the majority of mlearning projects, the researcher has been interested in transforming pedagogy from teacher-directed towards social constructivism using mlearning as a catalyst to enable student-generated content and student-generated learning contexts. The Pedagogy-Andragogy-Heutagogy (PAH) continuum (Blaschke 2012; Luckin et al. 2010) has been used by the researcher as a critical framework to measure how much pedagogical change each mlearning project has achieved, with each project evidencing a move along the PAH continuum (or not). Thus one indicator of achieving “significant difference” has been the evidence of a change in pedagogy (from teacher-directed pedagogy to student-centred andragogy and ultimately student-directed or negotiated heutagogy) as an outcome of each project.
To date the majority of mlearning research has used traditional research methodologies with a focus upon predominantly small-scale descriptive case studies with little evaluation and reflection witnessed (Wingkvist and Ericsson 2009). A review of MLearn2007 and 2008 papers (Wingkvist and Ericsson 2009) found that an action research methodology was used by only 5% of these papers. Traxler (2011) commenting upon the impact of mlearning observes:
These achievements have usually been focused on pedagogy and technology, and have often been part of the research work of universities and institutes, separate from mainstream teaching and learning. Consequently, most of this research and development has been proof-of-concept, project-based, fixed-term and small-scale with little consideration of how to embed, sustain or scale up. (Traxler 2011)
Reeves (2005) makes a case for new educational technology research methodologies that move away from comparative control-based studies that usually result in “no significant difference” to research methodologies that leverage sustained collaboration between researchers and practitioners exploring the unique affordances of new technologies within unique contexts based on reflective enquiry. Reeves (2009) promotes design-based research as such a potential transformational research methodology. Design-based methodologies in mlearning research have been championed by Herrington et al. (2009). Similar in approach to that of a design-based research methodology advocated by Reeves, the researcher used a participatory action research methodology for all of the 35 mlearning projects from 2006 to 2012. This involved a collaborative partnership between the researcher and each of the participating lecturers in each mlearning project. Many of these partnerships have extended over several mlearning project iterations spanning several years, for example (Cochrane 2010a; Cochrane and Bateman 2010; Cochrane and Rhodes 2011).
Critical reflection upon the design, implementation and outcomes of the researcher's 35 mlearning projects have led to the identification and refinement of six critical success factors for mobile web 2.0 project implementation (Cochrane 2010b; Cochrane 2012):
This section outlines three examples of “failed” mlearning projects and critiques them using the researchers “critical success factors” as a framework to analyse why these projects failed to achieve a significant pedagogical shift towards social constructivism.
The 2008 Diploma of Landscape Design project followed two successful previous mlearning project iterations in 2006 and 2007 within the course. The previous mlearning projects were collaborative partnerships between the researcher and a second year course lecturer and focused upon an annual national flower show contest where teams of students designed and constructed themed landscape designs for the show. The students used eportfolios and cameraphones to record and share their project decisions and progress. The 2008 project involved the addition of a second lecturer from the department in the collaborative research and was based on an elective investigative field trip to Japan, for which the participants were required to be able to fund the cost of the trip themselves. This added cost factor effectively limited the participants to a small group of students who could afford the trip with the resulting age range of the participants being from 42 to 69, with an average age of 55. Not all of the students participating in the Japan trip volunteered to be involved in the mobile web 2.0 project. The Japan trip ended up with a total of 15 students and 2 course lecturers, with 6 of the students participating in the mobile web 2.0 project. All of the mobile web 2.0 participants were part-time students who had either part-time or full-time jobs related to landscape design, this resulted in a markedly different student profile to that of the previous projects where the majority of students had been in their early 20s. The 2008 student participants had very little previous experience of web 2.0 or mobile technologies (apart from traditional standard cellphone use), and significantly less experience than previous student groups, and student participants had particularly demonstrated a lack of engagement with web 2.0 tools prior to their involvement in the mobile web 2.0 projects. This had implications for the level of pedagogical and technical support required to make the project successful.
The 2007 mobile web 2.0 project identified the need to get lecturers and students up to speed with the mobile web 2.0 technologies before the course project entered critical time-consuming stages. The early integration of the use of the mobile web 2.0 tools and the development of a regular weekly community of practice (COP) were highlighted as potential ways of supporting the mobile web 2.0 projects. Therefore, the researcher and lecturers planned to establish a COP with the students, lecturers and the researcher as the technology steward in semester one in preparation for the elective course beginning in semester two 2008. However, the course participants were all part-time students and were reluctant to attend the COP sessions before the start of the course. Hence COP sessions were limited to four introductory sessions in semester one, followed by a three month break, then four more COP sessions in the month leading up to the trip to Japan in semester two. Key elements of cultivating a COP as emphasised by Wenger, McDermott and Snyder (2002) were compromised by this approach, including: a lack of sustained engagement leading to weak development of a sense of community, a lack of modelling of the expected communities practices by the lecturers leading to the students remaining on the periphery of the group and a resultant reverting to the COP to effectively become workshop sessions rather than forming the core of a developing COP. The first four COP sessions were held in the researcher's computer lab, while the second set of COP sessions were held in a very noisy shared computer lab space, neither of which were conducive to the students’ forming a sense of group space or belonging. The 2008 group did not establish a sense of community identity until they were together on the trip to Japan, where they spent significant time together. At this point there was little opportunity for the technology steward to help the group as they encountered technical issues while in Japan with unexpected limited WiFi connectivity options, and as a result several of the student participants, and one of the lecturers, struggled to integrate the mobile web 2.0 technologies into their workflows.
The sonyericsson P1i UIQ-based smartphone was chosen for the 2008 project based on feedback from the 2007 project that indicated better text entry options were need for students. However, the profile of the 2008 students did not suit the rather complex UIQ operating system, and thus the smartphone interface presented a barrier to mlearning adoption.
The project achieved the following significant outcomes.
The project failed in the following aspects.
The 2009 Bachelor of Architecture mlearning project was the first foray into the integration of mobile and web 2.0 tools within the course. The Bachelor of Architecture year-two course mlearning project drew upon the lessons learned from the previous Landscape Design, Product Design and Contemporary Music mlearning projects. Having heard about the previous mlearning projects, lecturers within the school of Architecture at Unitec had expressed interest in exploring the potential of mlearning within their courses. Thus the researcher brainstormed the possibilities with the architecture technology lecturer who then partnered with the researcher to facilitate the instigation of an architecture-based mlearning project in 2009. The Architecture technology lecturer identified the second year architecture course with 115 students as a potential target for the proposed 2009 mlearning project. The Bachelor of Architecture 2009 project used the Nokia XM5800 smartphone and the Dell Mini9 netbook. The project began by initiating a pre-project Architecture lecturer COP cultivated by a weekly meeting of the participants at one of the Campus Café's. Participants brought along their mobile devices and discussed pedagogical and technical issues around the use of mlearning tools facilitated by the researcher.
As reflections on the previous mlearning projects had established that critical success factors for implementing mlearning included the level of integration of the technology into the course and assessment, the negotiated plan was to include the use of moblogging within the second year Architecture compulsory Studio course as a new form of documenting, sharing and critiquing students’ individual and group design projects. However, the studio-coordinator lecturer responsible for setting the assessments for the course declined to be involved in the lecturer COP and essentially vetoed the integration of mlearning into the course assessment. Discussions held between the researcher, the COP participants and the studio-coordinating lecturer did not mange to bridge this impasse. The reasons cited by the coordinating lecturer were: “Architecture is not interested in process, only the final design, and therefore design journaling will not benefit the course”, and secondly “In the Studio course the face-to-face interaction is of primary importance”. While both of these assumptions were hotly debated, the coordinating lecturer refused to be persuaded. From the researcher's perspective, it appeared the root of the dispute was really about the threat of the project to the centralised control imposed upon the course by the coordinating lecturer. Within the context of the research, the ontological leap (Chi and Hausmann 2003) from lecturer-focused pedagogy to a social constructivist student-centred pedagogy facilitated by mobile web 2.0 was too much for the coordinating lecturer to bridge. The potential for mobile web 2.0 to create or enhance context-independent learning communities (Cook et al. 2007; Cook, Pachler, and Bradley 2008) was beyond the lecturer's experience and ability to conceptualise. Thus the “disruptive” nature of mlearning (Sharples 2001; Stead 2007) was viewed by the coordinating lecturer in a negative light, rather than positively as it had been found to be within previous projects. However, the lecturers who had been involved in the COP were keen to continue the project. Thus the mlearning project became a voluntary option for the second year Architecture students rather than integrated into the course assessment as had been planned, but was promoted and supported by the lecturers involved in the architecture COP (six of the nine second year lecturers). While this was a definite setback for the project, it was decided to go ahead as a proof-of-concept exploration anticipating that the student response would be positive and facilitate a re-think by the non-participating lecturers for 2010.
The project achieved the following significant outcomes.
The project failed in the following aspects.
The 2010 Bachelor of Computing mlearning project was the first attempt at the integration of mobile and web 2.0 tools within the department. The elective course was introduced and designed in the first half of 2010 to explore student mobile application development, specifically the development of iOS applications for the iPhone. The course presented a unique opportunity for the researcher to work collaboratively with the course lecturers to design and implement new pedagogical approaches within the department that leveraged the unique affordances of mobile web 2.0. The elective course was severely under-funded, with only one Macintosh iMac and two iPhones available for the 25 course students to share for iOS development. Consequently the researcher agreed to supply the course with 25 iPhones and 25 “hackintoshed” netbooks (netbooks that booted from a custom installation of OSX on USB memory sticks) for the students and three lecturers to borrow and use as their own throughout the length of the course. The researcher and course lecturers then spent the first semester collaboratively designing the new course curriculum to not only cover iOS development, but to also embed the use of mobile web 2.0 tools throughout the delivery of the course and to establish a learning community that could facilitate interaction, sharing of student-generated content and formative feedback beyond the classroom via Twitter, student blogs, wiki pages, web-based presentations (for example using http://prezi.com), and a course Moodle hub for administration. The use of interactive communications technology was particularly important for the course as the iOS development expert lecturer lived 400 km away and therefore planned remote lecture interaction with the students. Thus the specific outcomes of the co-designed course stated: This course provides us with a unique opportunity to use technology in our teaching and learning.
However, while spending the first semester collaboratively designing and developing the course, there was no time allocated to developing the lecturers’ experience or capability for using mobile web 2.0 tools themselves. Consequently, unbeknown to the researcher, the lecturers reverted to their previous course delivery methodologies rather than appropriating or embedding the use of mobile web 2.0 to explore new pedagogies. Thus the first class consisted of the researcher handing out iPhones and netbooks to the participants, followed by a the handing out of a completely revised version of the course outline that specified the course requirements as:
Very little thought had been given by the lecturers to appropriately scaffolding the students into iOS development or of the presentation or network requirements of the course. For example: the course was the first experience of Mac OS for the majority of students (and two of the course lecturers), whose previous computing experience had been solely Windows-based, there was no time allocation during classes for exploring the use of mobile web 2.0 tools for communication and collaboration, and no interaction with students via these tools beyond the classroom by the lecturers. The researcher attempted to compensate for these omissions by attending the classes, helping students with their basic OSX issues, providing short tutorials on mobile web 2.0 tools and web-based presentation tools and working with the IT department to install wifi access, laptop connectivity to the classroom projector and an audio system for class presentations. However, the synchronous communication tools used by the lecturers proved to be unreliable and ineffectual for class interaction. Discussions between the researcher and course lecturers revealed that the lecturers did not want to use asynchronous communication and collaboration tools such as Twitter or blogging as they did not have time to learn how to use them or understand the pedagogical benefits, also the final year students were expected to be self-motivated and self-directed with minimal lecturer contact beyond the classes. Student blogs and Twitter use were belatedly added to the course, but received only sporadic uptake from students. Although several student teams did eventually produce some very creative iOS applications and some voluntarily engaged with the mobile web 2.0 tools demonstrated by the researcher, overall the course was regarded as a lost opportunity by the researcher.
The project achieved the following significant outcomes.
The project failed in the following aspects.
The three projects reinforced the researchers belief that designing for pedagogical change enabled by technology is crucial for moving beyond the phenomena of no significant difference. Critical in this design is the establishment of pedagogical and technological support strategies. Creating a truly collaborative partnership between educational technology stewards or researchers and course lecturers well before implementing pedagogical change with students is one way to support lecturer professional development and buy-in. Choosing appropriate technologies for each unique student cohort is also important, and a focus upon supporting student-owned devices and providing appropriate flexible learning spaces that bridge formal and informal learning is ultimately the way forward. Achieving this is often a long-term commitment, seldom reached within a single iteration of a project implementation. While the Diploma of Landscape Design and Bachelor of Architecture projects developed into further successful iterations, the lack of commitment to change from the Bachelor of Computing lecturers did not sustain further attempts.
A sustained collaborative partnership between the researcher and discipline-based lecturers provides a staged and scaffolded approach to what is effectively apprenticing lecturers into a community of scholarship of teaching and learning and the appropriation of new educational technologies.
These three projects demonstrate that lecturer professional development and sustained collaborative support are critical to fostering new pedagogies utilising the unique affordances of mobile devices. The model adopted by the researcher across the 35 mlearning projects to achieve this was to establish a COP (Lave and Wenger 1991; Wenger 1998) of the participating course lecturers as a catalyst for exploring new teaching and learning frameworks for social constructivist pedagogy in tertiary teaching and learning (Cochrane 2010a). The goal was to provide the Lecturers with an authentic experience of the development of a learning community enabled by technology that would then be implemented and modelled by the Lecturers within the course itself. In all three of the illustrative cases the main failure was to not establish a supporting COP including all of the key lecturers involved in the projects.
The researcher therefore suggests that the implementation of “successful” mlearning projects that result in significant pedagogical change can effectively be supported by collaboration between the project leader (course lecturers) and an academic researcher/advisor taking on the role of the technology steward (Wenger, White, and Smith 2009) within a COP. Table 1 provides an outline of an indicative implementation strategy for an example mlearning project. Note that while the context of this research was enabling social constructivist pedagogy, other learning paradigms can be added or substituted according to the chosen project goals.
This approach involves a significant time and trust commitment from the participants that needs to be clearly articulated right from the start of any mlearning project, however the effort involved can be justified by the potential benefits and evidence of significant pedagogical change from successful implementations, and balanced against the huge time and resource loss of a needless project failure. Potential outcomes of COP supported mlearning projects for the participants include:
The three example case studies illustrate that when this collaborative COP was not established effectively then these outcomes were not explicitly achieved within the project and therefore this was a factor in the failure of these projects to enable any significant pedagogical change.
Often the most significant breakthroughs in pedagogical transformation resulting from participatory action research are those borne of experiences of failure and critical reflection upon the surrounding issues. However, these “failed” projects are usually not the subject of research papers, rather their successful successors are. This article confronts the reality of educational technology interventions by sharing some of the valuable lessons learnt from failed mlearning projects that have then successfully informed the design and implementation of subsequent projects. The researcher has found that often the first initial foray into mlearning project implementations within a new educational context represent large learning curves for the participants that result in project failure or demonstrate no significant pedagogical difference, but these projects can inform subsequent successful iterations. Sustained pedagogical and technological support are critical to scaffold the lecturers and students reconceptualisations of teaching and learning enabled by mobile web 2.0.
Blaschke, L. M. (2012) ‘Heutagogy and lifelong learning: a review of heutagogical practice and self-determined learning’, The International Review of Research in Open and Distance Learning, vol. 13, pp. 56–71.
Chi, M. & Hausmann, R. (2003) ‘Do radical discoveries require ontological shifts?’, in International Handbook on Innovation, eds L. Shavinina & R. Sternberg, Elsevier Science Ltd, New York, pp. 430–444.
Cochrane, T. (2010a) ‘Beyond the Yellow Brick Road: mobile web 2.0 informing a new institutional elearning strategy’, Special Issue 'The Transformational Impact of Learning technology' of ALT-J, Research in Learning Technology, vol. 18, pp. 221–231.
Cochrane, T. (2010b) ‘Exploring mobile learning success factors’, ALT-J, Research in Learning Technology, vol. 18, pp. 133–148. [Crossref]
Cochrane, T. (2011) ‘mLearning: why? What? Where? How?’ in Proceedings of the 28th ASCILITE Conference, ASCILITE 2011: Changing Demands, changing Directions, eds G. Williams, P. Statham, N. Brown & B. Cleland, The University of Tasmania, Hobart, pp. 250–262.
Cochrane, T. (2012) ‘An mlearning journey: mobile web 2.0 critical success factors’, International Journal of Handheld Computing Research (IJHCR), vol. 3, pp. 44–57. [Crossref]
Cochrane, T. & Bateman, R. (2010). ‘Reflections on 3 years of mlearning implementation (2007–2009)’, in Proceedings of the IADIS International Conference Mobile Learning 2010, 19–21 March 2010 Porto, eds I. A. Sánchez, & P. Isaias, IADIS international Association for Development of the Information Society, Portugal, pp. 181–188.
Cochrane, T., Bateman, R., Cliffin, P., Gardner, J., Henderson, I. & Holloway, S. (2009) ‘Mobilizing learning: Mobile Web 2.0 scenarios in tertiary education’, in EDULEARN09 the International Conference on Education and New Learning Technologies, eds L. G. Chova, D. M. Beleguer & I. C. Torres. (International Association of Technology, Education and Development (IATED), Barcelona.
Cochrane, T. & Rhodes, D. (2011) ‘iArchi[tech]ture: heutagogical approaches to education facilitated by mlearning integration,’ In Proceedings of the International Conference on Information and Communication Technologies in Education ICICTE 2011, eds K. Fernstrom and C. Tsolakidis. University of the Fraser Valley and the University of the Aegean, Rodos Palace Hotel, Rhodes.
Cook, J. (2010) ‘Mobile phones as mediating tools within augmented contexts for development’, International Journal of Mobile and Blended Learning, vol. 2, pp. 1–12. [Crossref]
Cook, J. et al. (2007) ‘Generating learner contexts with mobile devices’, in Mobile Learning: Towards a Research Agenda, ed. N. Pachler, WLE Centre, Institute of Education, London, pp. 55–74.
Cook, J., Pachler, N. & Bradley, C. (2008) ‘Bridging the gap? Mobile phones at the interface between informal and formal learning’, Journal of the Research Center for Educational Technology (RCET), vol. 4, pp. 3–18.
Frohberg, D., Goth, C. & Schwabe, G. (2009) ‘Mobile learning projects: a critical analysis of the state of the art’, Journal of Computer Assisted Learning, vol. 25, pp. 307–331. [Crossref]
Herrington, J., et al. eds (2009) New Technologies, New Pedagogies: mobile Learning in Higher Education, Faculty of Education, University of Wollongong, Wollongong.
Kukulska-Hulme, A. (2010) ‘Mobile learning as a catalyst for change’, Open Learning: The Journal of Open and Distance Learning, vol. 25, pp. 181–185. [Crossref]
Laurillard, D. (2007) ‘Pedagogcal forms of mobile learning: framing research questions’, in Mobile Learning: Towards a Research Agenda, ed. N. Pachler, WLE Centre, Institute of Education, London.
Lave, J. & Wenger, E. (1991) Situated Learning: Legitimate Peripheral Participation, Cambridge University Press, Cambridge.
Luckin, R., et al. (2010) ‘Learner-generated contexts: a framework to support the effective use of technology for learning’, in Web 2.0-Based E-Learning: Applying Social Informatics for Tertiary Teaching, eds M. Lee & C. Mcloughlin, IGI Global, Hershey, PA.
Pachler, N., Bachmair, B. & Cook, J. (2010) Mobile Learning: Structures, Agency, Practices, Springer, London.
Reeves, T. (2005) ‘No significant differences revisited: a historical perspective on the research informing contemporary online learning’, in Online Learning: Personal Reflections on the Transformation of Education, ed. G. Kearsley, Educational Technology Publications, Englewood Cliffs, NJ, pp. 299–308.
Reeves, T. (2009) ‘The application of “design research” to e-learning’, The First International Conference for e-Learning and Distance Learning. Ministry of Education, Riyadh.
Sharples, M. (2001) ‘Disruptive devices: mobile technology for conversational learning’, International Journal of Continuing Education and Lifelong Learning, vol. 12, pp. 504–520. [Crossref]
Stead, G. (2007) Mobile Learning: The Devices may be Small, but Their Impact isn't. M-Learning Symposium, WLE Centre, Institute of Education, London.
Traxler, J. (2011) ‘Introduction’, in Making Mobile Learning Work: Case Studies of Practice, eds J. Traxler & J. Wishart, ESCalate, University of Bristol, Graduate School of Education, Bristol, pp. 4–12.
Wenger, E. (1998) Communities of Practice: Learning, Meaning, and Identity, Cambridge University Press, Cambridge.
Wenger, E., Mcdermott, R. & Snyder, W. (2002) Cultivating Communities of Practice: A Guide to Managing Knowledge, Harvard Business School Press, Boston.
Wenger, E., White, N. & Smith, J. (2009) Digital Habitats: Stewarding Technology for Communities, CPsquare, Portland, Oregon.
Wingkvist, A. & Ericsson, M. (2009) ‘Current practice in mobile learning: a survey of research method and purpose’, in MLearn 2009: 8th World Conference on Mobile and Contextual Learning, eds D. Metcalf, A. Hamilton & C. Graffeo, The University of Central Florida, Florida.