Figure 1. The original continuums (page 1).
Richard Osborne, Elisabeth Dunne and Paul Farrand
University of Exeter, UK
Abstract
Current pressures in higher education around student employability are driving new initiatives for change. Assessment is also a topic of debate, as it is a key driver of student behaviour, yet often falls behind other metrics in national surveys. In addition, increasing focus on digital literacies is catalysing new appreciations of what emerging digital culture might mean for both students and staff. These three highly topical challenges were jointly explored by the University of Exeter’s Collaborate project, which aimed to create employability-focused assessments enhanced by technology. By combining existing research on assessment with grounded data derived from local stakeholders, the project has developed a model for assessment design which embeds employability directly into the curriculum. Digital technologies have been aligned with this model using a “top trump” metaphor, where key affordances of technologies are highlighted in the context of the model. This paper explores the design-based research approach taken to develop this model and associated “top trumps”, along with results from the first practical iteration. Results suggest that the model is effective in supporting the design of an “authentic” assessment and that a targeted affordances approach can support the alignment of specific technologies with a particular pedagogic design.
Keywords: employability; assessment; authentic; affordance; evaluation
Published: 6 September 2013
RLT 2013. © 2013 R. Osborne et al. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 3.0 Unported (CC BY 3.0) Licence (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited.
Citation: Research in Learning Technology 2013, 21: 21986 - http://dx.doi.org/10.3402/rlt.v21i0.21986
This paper documents the preliminary findings from the University of Exeter’s Collaborate project, which aims to bring together staff, students and employers to create employability-focused assessments enhanced by technology.
Student employability is currently something of a hot topic in Higher Education, and perhaps for good reason given the importance placed on employability by those thinking of attending university. Recent surveys have highlighted it as among the foremost reasons for considering a university education (Pollard et al. 2008), while over 70% of students stated that improving job opportunities was the most important reason to go to university (CBI/NUS 2011). One possible way for universities to help improve student employability is to “close the gap” between higher education and business (Wilson 2012). Exploring ways to embed the development of skills within students’ learning experience has been highlighted as having particular potential:
Universities should reflect on the opportunities that are provided for students to develop employability skills through the formal learning methodologies used within the university.
(Wilson 2012, p. 10)
This sentiment is at the heart of the Collaborate project, embedding employability into the curriculum by focusing on assessment as a vehicle for change.
Assessment is widely acknowledged as a key driver of student behaviour. As Gibbs and Simpson (2004) summarises, what students attend to, how much work they do and how they go about their studying are completely dominated by how they perceive the demands of assessment. Assessment can therefore be an ideal target for educational innovations. Traditional forms of assessment such as exams and essays, however, are not well suited for embedding employability as they tend to be used infrequently in professional contexts. Other forms of assessment exist which may be better suited to this task, in particular authentic assessment.
The concept of an authentic assessment appears to originate in the US K-12 school system (Terwilliger 1997), as a reaction against the use of multiple choice questions as the core or only way of assessing student performance. The use of the word authentic has proved challenging, however, as Terwilliger summarises: “the term inappropriately implies that some assessment approaches are superior to others because they employ tasks that are more ‘genuine’ or ‘real’” (1997, p. 27). Although authentic assessment appears to be increasingly used in further and higher education, the term is still not widely understood (Falchikov 2005), partly perhaps due to multiple perspectives and definitions (e.g. Gulikers, Bastiaens, and Kirschner 2004; Mueller 2005; Wiggins 1993). Indeed, surveys by Whitelock and Cross (2011) at the Open University suggest that only 25% of the academics surveyed had heard of the terms “authentic learning” and “authentic assessment”.
In order to bring more clarity to this area, Whitelock and Cross (2011) analysed literature from key writers in the field to summarise the common features of an authentic assessment, which included:
Whitelock and Cross (2011, pp. 3–4)
At Exeter, this summary was combined with research from Herrington (2006), Karet and Hubbell (2003) and local stakeholders, in order to develop a new “dimensions” model for the design of authentic assessments.
The value of constructive alignment (Biggs 1996) between intended learning outcomes, teaching methods and assessment is widely acknowledged (Gulikers, Bastiaens, and Kirschner 2004), yet this same constructive alignment is rarely seen in the application of technologies to learning scenarios. Technology use is widespread in higher education, yet it is often “treated as ‘natural’ or given” using a common sense approach (Oliver 2013). Oliver contends that this may in part be due to a lack of theory to explain technology in an educational context, and suggests that one possible approach is to use the theory of affordances (Gibson 1979). Although Oliver argues that affordances may in fact be too problematic to be of practical use, others suggest that affordances can be a useful framework for considering how technologies might be used to support learning and teaching (e.g. Bower 2008; Conole and Dyke 2004; Hammond 2010).
Attempts to align technologies directly with pedagogy may be impractical, as individuals tend to find ways of using technology in unplanned ways to suit their needs. The notion of interpretive flexibility, as described by Pinch and Bijker (1984), suggests that technological artefacts are interpreted by the society that uses them, and tend to be used in flexible ways. Indeed Squires and McDougall (1994) note that in educational contexts technologies can be most effective when teachers and students use them in ways that the original designers had not intended. The theory of affordances suggests ways of addressing this issue of flexibility and has two other characteristics that may allow technologies to be aligned with pedagogy using an affordances approach:
Although the theory of affordances may have potential for thinking about how technologies might be aligned to a specific pedagogic model, it does not help in suggesting which technologies might be useful. Indeed, the issue of which technologies are appropriate in education is something of a moving target. As the digital world matures, traditional models of technology use in education, with a central IT department responsible for the delivery of learning technologies, are being challenged by a “Bring Your Own Device/Service” approach, a cultural shift where “off the shelf” technologies are used in increasingly hybrid ways to support studying. There is evidence that students feel central technology provision has fallen behind what they can provide themselves and that “the type of technology used in HE is increasingly outdated” (NUS 2010). For this reason, together with the overall focus on employability, the Collaborate project has focused on how contemporary “off-the-shelf” technologies might be aligned with the new “dimensions” model.
A Design-Based Research (DBR) methodology was adopted to explore how assessments might be adapted to include features associated with authentic assessments, supported by contemporary technologies. DBR is an emerging methodology, with no strict definition, though researchers have identified a number of principles. Reeves (2006), for example, outlines three principles:
… addressing complex problems in real contexts in collaboration with practitioners; integrating known and hypothetical design principles with technological advances to render plausible solutions to these complex problems; and conducting rigorous and reflective inquiry to test and refine innovative learning environments as well as to define new design principles.
(Reeves 2006, p. 58)
Anderson and Shattuck (2012) suggested the following five characteristics were hallmarks of a DBR approach:
The Collaborate project matches well with these principles/characteristics of DBR, in that it is:
The challenge for the project was to create a model to support the redesign of assessments, embedding the development of skills and competencies into the module. Various methods were explored, beginning with a set of continuums based on a model of “tensions and challenges” relating to assessment (Price et al. 2011). This initial design evolved through testing with collaborative partners, and eventually words were chosen to summarise the key messages from both literature and local research concerning authentic assessment, which then defined new continuums (see Figures 1 & 2). A key feature of these continuums was that they were designed to be used and filled in rather than simply read.
Figure 1.
The original continuums (page 1).
Figure 2.
The original continuums (page 2).
Discussions with academic colleagues showed that the use of these linear continuums with two extremes tended to lead to positive and negative perspectives, isolating and reinforcing existing beliefs and polarising opinion. Any attempt to cast current academic practice near one end and authentic practice nearer the other suggested conflict between the two positions. A new model was sought which would reduce this tendency, with a more “value neutral” position that would allow for the exploration of assessments without negative connotations.
The project team decided to adopt a radar chart for the model, which has no right and wrong ends and therefore does not suggest positive and negative positions. The dimensions shown in Figures 1 & 2 were collapsed from seven to six, with the problem and data dimensions combined, and arranged in a hexagon shape. It was also decided to move away from the term authentic assessment, coining instead the phrase “work-integrated assessment”, due to concerns over the meaning of the word authentic. We defined work-integrated assessment as an assessment where the tasks and conditions are more closely aligned to those experienced within employment. The resulting six dimensions of work-integrated assessment are listed and described in Table 1, and their hexagonal arrangement is shown in Figure 3.
Figure 3.
A blank version of the dimensions model.
The dimensions model was printed on A3 sheets, and used by collaborative design teams of academic staff, together with employers, students and other staff as appropriate, to help change existing assessments. The model is both a thinking tool, using the dimensions to stimulate discussion, and a redesign tool, annotated in order to visualise and agree collective thinking. It is used in three stages, each of which is marked on the chart:
The analysis and re-design stages are completed at the same time, the evaluation stage after the module has finished. Figure 4 is a completed model showing the analysis and design stages.
Figure 4.
A completed version of the dimensions model.
The purpose of the current project is not only to create a model to help design work-integrated assessments, but also to find a way to integrate contemporary technologies into this model. A method was needed which would identify the affordances of “off the shelf” technologies in relation to the model’s six dimensions.
The “Top Trumps” card game, popular in many parts of the world, seemed to offer a format ideally suited to our needs. “Top Trumps” show a series of related objects, listing their strengths and weaknesses against a series of fixed measures. By adopting this “top trump” metaphor, and setting the measures as our six dimensions, ratings could then be assigned for each technology summarising what it “provides or furnishes” (Gibson 1979) from an affordances perspective for each dimension.
A large number of potential technologies were identified for Collaborate, using data from previous technology training sessions and local expertise, and these were then narrowed down to a collection of approximately 60 of the most prominent and well-known technologies, such as Blogger, Facebook and Screenr. Small focus groups of staff then allocated up to five “stars” for each of the dimensions as listed in Table 1, by asking questions such as “what does this provide or furnish for …”, with different questions written to prompt thinking for each of the six dimensions. Blogger, for example, scored highly for Time and Review, as it uses a diary-based approach and has extensive commenting and feedback facilities. It scored lower for Collaboration and Problem/Data, however, as it has limited team-working facilities and does not inherently provide any data or real-world problems. By using this technique the affordances of each of the technologies was rated, in the context of the dimensions model, creating a set of technology “top trumps” for work-integrated assessment. Examples of these “top trumps” are shown in Figures 5 & 6.
Figure 5.
Top trump cards (front).
Figure 6.
Top trump cards (back).
The overall model or top trump system is designed to recommend not just one potential technology to support a pedagogic design, but multiple technologies that might be appropriate. In accordance with the features of affordances outlined earlier, this design feature should allow individual students and/or academics to select specific technologies from a choice based on personal needs and experience.
Fourteen iterations have currently been planned with the model, in such disparate disciplines as Spanish, Engineering, Geography, Law, Education, Mathematics and Politics. This paper focuses on the first completed iteration, with 25 students taking Professional Issues and Development (PSY1301), a first year module of a BSc Applied Psychology (Clinical) programme.
The original assessment for the module consisted of a 1½-hour written examination (75%) and a 20-minute oral presentation (25%) that was undertaken in pairs with each student contributing 10 minutes. The overall position of these assessments was decided by a collaborative design team and sketched out on the dimensions model, which can be seen at the heart of Figure 4 as the smallest polygon.
The next step was to discuss what changes to the assessment might be made in order to move along each of the dimensions. These changes were discussed by the same team, which made the following design decisions:
The anticipated consequences of this design can be seen in the larger polygon in Figure 4.
Technologies were then identified that might support this assessment. Of the six dimensions being considered, it was felt that those of structure and collaboration were critical to the overall task, and potentially the most challenging for first year students. For this reason, technologies were sought which matched a star rating of five for both the structure and the collaboration dimensions. Using this process, “top trumps” for Do, Trello and TeamMatch were recommended, as shown in Figures 5 & 6.
Once technologies had been identified that had the potential to support the pedagogic design, the students were given 10-minute demonstrations to show how these technologies might be useful in supporting their assessment.
Once the iteration had been completed, evaluations were carried out with the students, academics and employers.
A student session was organised where students completed three evaluation exercises:
Figure 7.
“Mark on the line” evaluation data.
The personal perspectives of academics, employers and students involved with the assessment were evaluated using semi-structured interviews.
The evaluation data as shown in Figures 7, 8 & 9 suggest that the overall “authentic” assessment approach taken by the project was effective and that the inclusion of employers in the assessment process was beneficial. Although some students commented that the module design was unexpected, data show that they enjoyed the module nevertheless, and they appear to have developed their skills as had been intended through the design. Detailed discussion focuses on the key aspects of the design.
Figure 8.
“What’s the most important thing I have learnt” postcard evaluation data.
The dimensions model has now been tested by a wide range of academic staff, from senior members with extensive educational experience to relative newcomers to the field, and has been well received by all. It seems to require little guidance in its use, and in the iteration described it served the designed purpose in supporting assessment change, as skills and competencies do appear to have been built into the fabric of the module. It has been particularly effective in prompting thinking in unexpected ways, such as along the dimensions of audience and structure.
The success of the model may in part be due to the fact that it “invites engagement”, that is, it is in part blank and designed to be actively completed, as opposed to being read passively. Those who use the model therefore have ownership of it; it becomes part of their thinking rather than that of the original designers. Users are effectively constructing their own meaning from the model, and the model itself serves only as a scaffold for their own thinking.
The purpose of the technology “top trumps” was to provide a way of aligning “off the shelf” technologies with the dimensions model. By allocating one to five stars against each technology to quantify their affordance in relation to the six dimensions, specific technologies could then be recommended that should have a positive impact on an assessment designed using the model. In this iteration, technologies were recommended to support the structure and the collaboration dimensions, specifically to support students in developing skills in collaboration, team working and project management.
Analyses of data gathered from the evaluation sessions with students (as shown in Figures 7, 8 & 9) all showed that the students had developed their collaboration skills. Analyses of the postcards data and the skills drag and drop exercise suggested that the students had developed time management skills, and their ability to work autonomously. Whilst we cannot claim a causal relationship between the technologies used and these results, there is a strong correlation between the designed objectives and the evaluated outcomes in terms of developing skills and competencies. We propose that this is at least in part due to the alignment of appropriate technologies with the pedagogic design.
Figure 9.
Drag and drop skills evaluation (completed dimensions model).
Interviews with academic staff and students appeared to corroborate this conclusion. The academic lead commented that some students “adapted technologies that they were familiar with to suit this purpose”, for example two groups decided to use Facebook to organise work rather than the suggested tools. This type of innovative use of technology is predicted by the concept of interpretive flexibility, but suggests that the “top trumps” do need refining further. Crowdsourcing, that is, using student input to moderate star values over an extended period of time, is being explored as a possible long-term solution. Facebook was not considered as providing enough Structure to warrant five stars for this dimension, though does score five stars for Collaboration. The Trello software was only adopted by a small number of students, but it was deemed to be “absolutely excellent” in supporting their team working and time management, and has now been adopted by some students for all their studies. The TeamMatch software was useful in highlighting to the students their own strengths and weaknesses, though it was not used in the first iteration to form teams. No students reported using Do.
The use of affordances as the underlying theory to align technologies with a pedagogic approach may be questionable, and there is certainly much debate over the pros and cons of the term from an educational perspective (Hammond 2010). Nevertheless the project seems to have had some success using the theory of affordances to highlight the strengths of technologies when considered in a specific pedagogic context.
A possible reason for this success is the project’s focus on Gibson’s (1979) original definition of affordance, rather than later evolutions most often ascribed to Norman (1988). In particular we have focused on affordances as functional, and not as “action possibilities”, a term which now seems synonymous with affordances (although there is no mention of this term in Gibson’s original work). By functional affordance we mean what something “provides or furnishes”, in line with Gibson’s original definition, rather than simply how it can be interacted with, which was more the concern of Norman. We have also avoided broad taxonomy approaches, such as that taken by Conole and Dyke (2004), or detailed classification systems, such as that proposed by Bower (2008), instead focusing on analysing affordances of technologies in relation to a specific context (i.e., the dimensions model), in a more “targeted” approach.
Understanding affordances as “action possibilities” is arguably limiting its usefulness in educational contexts, as it fixates technologies as tools, “things” to be prodded, pushed or pulled. We argue that a functional perspective brings time into the affordance concept, and allows technologies to be explored not simply as tools but also as specific “places” in their own right, with functions that happen in and across time. Blogger, for example, provides not simply a tool to write, but a place capable of providing for self-reflection and peer review; Evernote is not simply a note taking tool, but a place for centralising and contrasting disparate experiences. By conceptualising affordances in this way, we suggest that they may yet be useful in helping to develop pedagogic strategies similar to what we have presented for integrating technology effectively.
This paper has described the creation of a new model for “authentic” assessment design, which integrates off-the-shelf technologies using an affordances approach. Results from the first iteration using this model suggest that it is effective in supporting the design of an “authentic” assessment and that the targeted affordances approach can support the alignment of technologies with a pedagogic design. Further iterations with students studying Spanish, Geography and Law are also showing strong support for the overall approach, and more iterations are planned across the 2013/14 academic year to refine and develop the concepts presented.
We thank Stuart Redhead and Charlotte Anderson, Project Officer and Project Assistant respectively for the Collaborate project, for their invaluable assistance, particularly with the design of the model and the associated technology “top trumps”.
Anderson, T. & Shattuck, J. (2012) ‘Design-based research’, Educational Researcher, vol. 41, no. 1, pp. 16–25. Publisher Full Text
Biggs, J. (1996) ‘Enhancing teaching through constructive alignment’, Higher Education, vol. 32, no. 3, pp. 347–364. Publisher Full Text
Bower, M. (2008) ‘Affordance analysis – matching learning tasks with learning technologies’, Educational Media International, vol. 45, no. 1, pp. 3–15. Publisher Full Text
CBI/NUS. (2011) Working Towards Your Future – Making the Most of Your Time in Higher Education, [online] Available at: http://www.cbi.org.uk/media-centre/news-articles/2011/03/working-towards-your-future/
Conole, G. & Dyke, M. (2004) ‘What are the affordances of information and communication technologies?’, Research in Learning Technology, vol. 12, no 2, pp. 113–124 Publisher Full Text
Falchikov, N. (2005) Improving Assessment through Student Involvement: Practical Solutions for Aiding Learning in Higher and Further Education, Routledge, Falmer.
Gibbs, G. & Simpson, C. (2004) ‘Conditions under which assessment supports students’ learning’, Learning and Teaching in Higher Education, vol. 1, no. 1, pp. 3–31.
Gibson, J. J. (1979) The Ecological Approach to Visual Perception, Houghton Mifflin Harcourt (HMH), Boston.
Gibson, J. J., Reed, E. S. & Jones, R. (1982) Reasons for Realism: Selected Essays of James J. Gibson, L. Erlbaum, New Jersey.
Gulikers, J. T. M., Bastiaens, T. J. & Kirschner, P. A. (2004) ‘A five-dimensional framework for authentic assessment’, Educational Technology Research and Development, vol. 52, no. 3, pp. 67–86. Publisher Full Text
Hammond, M. (2010) ‘What is an affordance and can it help us understand the use of ICT in education?’, Education and Information Technologies, vol. 15, no. 3, pp. 205–217. Publisher Full Text
Herrington, J. (2006) ‘Authentic e-learning in higher education: design principles for authentic learning environments and tasks’, Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2006, eds T. Reeves & S. Yamashita, AACE, Honolulu, HI, pp. 3164–3173.
Karet, N. & Hubbell, E. R. (2003) ‘Authentic assessment’, [online] Available at: http://www.elizrosshubbell.com/portfolio/authenticassessment.pdf
Mueller, J. (2005) ‘The authentic assessment toolbox: enhancing student learning through online faculty development’, Journal of Online Learning and Teaching, vol. 1, no. 1, pp. 1–7.
Norman, D. A. (1988) The Psychology Of Everyday Things, Basic Books, New York.
NUS. (2010) ‘Student perspectives on technology – demand, perceptions and training needs’, [online] Available at: http://www.hefce.ac.uk/data/year/2010/studentperspectivesontechnologydemandperceptionsandtrainingneeds/
Oliver, M. (2013) ‘Learning technology: theorising the tools we study’, British Journal of Educational Technology, vol. 44, no. 1, pp. 31–43. Publisher Full Text
Pinch, T. J. & Bijker, W. E. (1984) ‘The social construction of facts and artefacts: or how the sociology of science and the sociology of technology might benefit each other’, Social Studies of Science, vol. 14, no. 3, pp. 399–441. Publisher Full Text
Pollard, E., et al. (2008) ‘University is not just for young people: working adults’ perceptions of and orientation to higher education’, Institute for Employment Studies DIUS Research Report, vol. 8, no. 6.
Price, M., et al. (2011) ‘If I was going there, I wouldn’t start from here: a critical commentary on current assessment practice’, Assessment & Evaluation in Higher Education, vol. 36, no. 4, pp. 479–492.
Reeves, T. C. (2006) ‘Design research from a technology perspective’, in Educational Design Research, eds J. den Akker, K. Gravemeijer, S. McKenney & N. Nieveen, Taylor & Francis, New York, pp. 52–66.
Squires, D. & MacDougall, A. (1994) Choosing and Using Educational Software: A Teacher’s Guide, Falmer Press, London.
Terwilliger, J. (1997) ‘Semantics, Psychometrics, and Assessment Reform: A Close Look at “authentic” Assessments’, Educational Researcher, vol. 26, no. 8, pp. 24–27.
Whitelock, D. & Cross, S. (2011) ‘Academics understanding of authentic assessment’, 2011 International Computer Assisted Assessment (CAA) Conference, Research into e-Assessment, [online] Available at: http://oro.open.ac.uk/29079/
Wiggins, G. (1993) ‘Assessment: authenticity, context, and validity’, Phi Delta Kappan, vol. 75, no. 3, pp. 200–208.
Wilson, T. (2012) A Review of Business – University Collaboration, Department for Business Innovation and Skills, London.