A window into learning: Case studies of online group communication and collaboration

The two case studies presented explore the potential offered by in-depth qualitative analysis of students’ online discussion to enhance our understanding of how students learn. Both cases are used to illustrate how the monitoring and moderation of online student group communication can open up a ‘window into learning’, providing us with new insights into complex problem-solving and thinking processes. This concept is demonstrated with examples of students’ online interaction using the group collaboration tools offered by the Blackboard v.6 and WebCT Vista 3 Virtual Learning Environment (VLE) software platforms. These extracts offer examples of students’ ‘thinking aloud’ while problem-solving, showing how and why they arrived at particular outcomes and the underlying thought processes involved. It is argued that these insights into students’ learning processes can in turn offer us the opportunity to adapt our own teaching practice in order to achieve a better pedagogical ‘fit’ with the learning needs of our students. This may be, for example, through a more precise and targeted or more timely intervention on the part of a tutor to correct a misunderstanding that has become apparent through monitoring of student-to-student discussion online. It is also suggested that looking through this ‘window’ enables us to concentrate our assessment more closely on the process of task completion, rather than focussing solely on the end product .

We refer to this concept as the e 3 model to capture the essence of the three aims.
The theoretical framework that guided our use of the VLE was that of the socialconstructivist (or collaborative-constructivist) perspective, placing the role of communication and interaction at the heart of education. This perspective favours the Vygotskian approach of enabling the construction of meaning and knowledge through shared dialogue and discourse and the confirmation of understanding through mutual sharing and testing of ideas in a collaborative environment (Mason, 1996;Harland, 2003). In particular, we aimed to create a learning environment that empowered students and encouraged them to become more independent and autonomous with regard to their own learning -in effect, using technology to create a virtual 'Autonomous Learning Zone' (Hughes, 2003). In its ideal form, this supportive environment would offer a virtual form of Lipman's 'community of enquiry ' (1991) where …students listen to one another with respect, build on one another's ideas, challenge one another to supply reasons for otherwise unsupported opinions, assist each other in drawing inferences from what has been said, and seek to identify one another's assumptions.
Analysis of interactions on the student discussion forums suggested that the students did, indeed, form such supportive 'communities of enquiry'. However, we believe that an additional benefit from the use and analysis of these forums has been the opportunity opened up to us to enhance our own practice through a better understanding of students' problem-solving processes and from an insight into their own 'reflection-in-action' (Schon, 1982). This is of crucial importance in a context where, it has been suggested, it is not uncommon for academics engaged in teaching students to focus on session content during teaching rather than on student learning (Fry et al, 2003). It is surely not unreasonable to assume that the more we can know about student learning processes, the better we are likely to be able to teach. We hope that the current study will provide some ideas as to how we can harness the power of collaborative communication tools to improve this knowledge.

Methodology
The research approach is exploratory -although, from a practice point of view, we were interested in extending and evaluating the opportunities offered by use of the VLE, the quality of the insights into the problem-solving and student learning processes that were gained were not initially anticipated. The research strategy involved the use of two case studies, the contexts for which were selected in part on the grounds of their suitability as a test-site for theory (Yin, 1994) and in part on a pragmatic basis in terms of access to data. Although it is recognised that such an approach does not provide a strong basis for generalisation, we subscribe to Denscombe's view of its utility for small-scale research where "…there may be insights to be gained from looking at the individual case that can have wider implications…The aim is to illuminate the general by looking at the particular" (Denscombe, 2003, p.30). This approach also enabled us to investigate a 'naturally occurring phenomenon' (Yin, 1994) within its own setting, with minimal risk of the occurrence of artificially induced findings.
The method adopted for data collection and analysis involved monitoring the subgroup discussion threads and analysing the content to identify the 'thinking' process that was taking place at that time. This was achieved by manually scanning the texts for natural language 'thinking indicators', to illuminate the underlying problem-solving process in which the students were engaged. These 'thinking indicators' all illustrate an aspect of the benefits from using this window into student learning to steer our own practice and interaction with our students.
By 'thinking indicators' we mean online 'speech' that suggests an underlying cognitive problem-solving process is taking place. This is represented by words or phrases such as 'I think…' or 'on reflection…', along with requests for explanation or confirmation of thoughts suggested by terms such as 'do you think…?' or 'how do you feel about…?' Although these words and phrases may not provide conclusive evidence of a cognitive process, they can act as a 'hook' that highlights the potential existence of a learning window. The use of a proprietary software package for the analysis was rejected, as we did not consider that it would be suitable in this particular context given the inevitable linguistic subtleties involved.
The text of all of the discussion content from each group in the two student cohorts was scanned. Those excerpts reproduced in this paper represent only those that we considered to be particularly illustrative of the concept of the Window into Learning. These indicate notions such as the identification of an appropriate point to steer the group away from going down a 'blind alley'; the benefit to students of a 'safe house' in which to test their ideas; and the ability to gain an insight into changing students thinking. In the excerpts presented here, underlining has been added by the authors to emphasise specific concepts, and commentary follows each extract. The insights provided through this exercise were supplemented by analysis of comments relating to the use of the discussion boards found within the text of City University students' written reflections on the process of completing the group task.

Case Study One Context -a problem-based experiential learning approach
The first study took place within the Department of Computing and IT at The case study is based on one such innovative approach by a Senior Teaching Fellow in the Faculty of Technology. This innovation adopted a Problem-Based Experiential Learning approach (the 'Design Studio'), intended to facilitate collaboration and deep learning among MSc Information Technology students.
The course involved a mix of part-time and full-time students studying the design, modelling and implementation of complex software artefacts. The use of the 'Design Studio' approach was premised on the belief that the development of critical thinking and problem-solving capabilities can facilitate a higher cognitive level of thought process. It was also intended to offer a means of illustrating and capturing best practice as demonstrated in the software industry, which operates in a field characterised by rapid, in-depth change and adaptation in the methodologies used to produce modern software.
Using the 'Design Studio' approach, students are faced with a large, complex software design problem, centred on a real-life activity and which must be taken to completion, even if this is only as a prototype artefact. In this instance, the specific task that students were required to accomplish was to produce software for an Airline Reservation system. Students work in groups of approximately four as independent teams reporting back each week in an iterative staged process.
These processes are assessed at each stage through presentation of their progress and subsequent review of submitted work. It was hoped that use of the Blackboard group communication tools would assist these group processes by encouraging students to support each other in a virtual community based on communication and collaboration -they could use interaction and feedback from others to help them determine the accuracy and application of their own ideas, thus providing an 'empowering mutual learning experience' (Palloff and Pratt, 1999, p.18) that would engender a Gestalt 1 effect in the output of the group. The module tutor's intervention in the group communication was restricted in the main to technical guidance when required or requested.
The Problem-Based Learning approach of the Design Studio represents an iterative process that incorporates conceptual stages that can be used sequentially, or more powerfully, in parallel. These stages comprise: • Triggering event or problem • Exploration -divergence, information exchange, suggestions, brainstorming, intuitive leaps • Integration -convergence, synthesis, solutions • Resolution -apply, test, defend (Garrison and Anderson, 2003) By using the Blackboard group communication tools it was anticipated that these stages could take place within an environment that offered students a suitable space for reflection, and, most importantly, a 'safe house' in which to air opinions, confident that their contributions were restricted to a closed egalitarian community.

Case Study Two Context -a group-based research and presentation task
The second case study draws on the experience of using group discussion boards at City University, with WebCT as the software platform. The use of WebCT at the University was pioneered within the Department of Information Science and since the start of the academic year 2003-2004 the VLE has been rolled out across the University. Currently about 400 modules University-wide use WebCT to provide online content, support and communication, in a combination of blended, flexible and distant learning modes. This particular case study illustrates use of the group online collaboration tools to complete the research and preparation for a face-to-face group presentation task by full-and part-time MSc Information Science and MSc Library and Information Studies students in a blended learning context.
In this case, students taking a module on Libraries and Publishing in the Information Society were asked to work in groups of four, using the group 'signup' sheets offered on WebCT to self-select their own groups. They were challenged to analyse the current state of the information sector and factors impacting on information flows and access to information within a country of their choice, taking account of considerations such as political and economic context; legislation; demographics; geography; historical and cultural background; and technological infrastructure. They used the group discussion boards to communicate and to co-ordinate their work. Tutor intervention in these discussions was in line with that in the previous case study. In addition to presenting their findings to the whole group, the students were required to produce a written reflection on their learning from carrying out the task.
The pedagogical intentions behind this particular assignment were shaped by the importance of students developing a high level of professional skills in areas such as independent research, team working, critical analysis, time management and presentation skills. This need has been much emphasised within the discipline area by employers and professional bodies alike (CILIP, 2004;Gorman, 2004;McAllister and Alexander, 2003, 285-6;Martin, 1989) and is embedded within the disciplinary benchmark statement (Quality Assurance Agency, 2000). The assignment was also intended to encourage a reflective value-critical approach, both to students' own professional practice and to the wider professional context in which they will be working. The use of group work also offered a means of transcending some of the fragmentation that students themselves had identified as a result of the inevitable loss of a 'course identity' within the context of a modular programme.

Ethical issues in the research strategy
It should be noted that the permission of students to analyse and use their contributions in this way has been obtained, and that all contributions are anonymised. With regard to the written reflections, the assignments of any students who did not explicitly give permission to have their work used in this way were excluded from the analysis.

Results
The results for the two case studies have been grouped thematically and presented in an integrated format to emphasise the portrayal of common concepts and themes. Within categories, the examples from the MSc Information Technology students are presented first, continuing with those from the MSc Information Science and MSc Library and Information Studies students. The examples given are quoted verbatim (apart from anonymisation), to give an accurate insight into the actual student dialogue and are followed by our comments. Only the text of discussion contributions is included; headers have been omitted in order to ensure full anonymity. For the same reason, any names used in the body of the text have been changed.

Reservations Software System
This echoes my own thoughts. Given that we are using a "Design Studio" process to develop the application (OOImp) and that we have considered web based development … together with DE's Databases and we know how to design a system … perhaps we should put this all together to suggest a totally web based system? i.e. front office apps such as the public web interface as well as back office solutions that would be suggested by a distributed system supported by a database (??) This example indicates a 'thinking aloud' comment by the student, reflecting on their approach to solving the specific design problem. In this instance, the tutor was able to steer the group away from wasting time on pursuing a web-based application, which was not the intended outcome for this assignment. The response to this posting from another group member illustrates the evolving problem solution process in action: I would suggest that the first step would be to reclarify the purpose statement to include Graham's alterations, then to elaborate on the features to discover what are the features that should be on offer to the three groups of system users (if one adopts the above) i.e. system managers -set-up type operations; RIC users -day to day transactions from RIC's perspective; and finally RIC's clients -online reservations and enquiries.
This hints at a first cut, use case diagram.
This new "confidential thread" seems to work.
In addition to the insight offered into the problem-solving process, the final comment in this excerpt suggests that the 'safe house' concept of the sub-group ("confidential thread") is beneficial.
…In light of our First stage and comments I have altered my thinking slightly, it occurs to me that possibly a high percentage of charter airlines (as per our remit) would probably have some form of WWW reservation/advertising. Whilst the advertising part is outside the remit the reservation part is not. Should we therefore now give more consideration to a WWW based interface? Or is it too early at this stage?
In this excerpt, we can see the dynamic nature of the thought processes as the student reflects on contributions from other group members. It provides an insight into the student's current thinking with regard to the problem solution, and, in this instance, again afforded an opportunity for the tutor to intervene with regard to the appropriateness of the proposed web-based solution. Here, the student seems to be developing the concept of a topic (ReservationTransaction) for him or herself and the text gives an insight into their thought process. Also the link into other subjects could be picked up by their colleagues to form associations that help enrich the learning experience of the students.

Student thinking processes and tutor intervention -analysing the global Information Society
I've been thinking about the education bit -such a vast subject. To narrow it down I thought to take it to a more basic level.
To do that I asked myself what was most fundamental to all information. The obvious answer is of course reading and writingliteracy, literate, illiterate … So my focus is Literacy from an Australian perspective.
In a very different context from the previous examples, this 'thinking aloud' excerpt nonetheless gives a similar insight into the thought processes that led to a particular decision or outcome. The group had chosen to research the Information Society in Australia and it had been decided that this group member would investigate 'education'. The discussion board helps the tutor -and other group members -to understand why the student chose a particular focus for analysis.
As the idea was to discuss the ppt itself, we're doing well. I do agree a minimum of slides is important. I think not more than 6 -7 in total… I've also been thinking that everything on the ppt should have a comparison or analysis, not merely be a pointer to what we will talk about next. We need to get the audience thinking about the subjectperhaps we could put questions up then supply the answer etc....or leading statements.
An interesting -and unexpected -outcome of this particular assignment was the insight gained into students' pedagogic awareness, and the attention they gave to this aspect of the task. Here it can be seen that they are thinking about how to engage their audience actively with the presentation content. There were a number of such examples: a couple of these are offered below as further illustration.
Animation wise, best not to do too much as takes away from content, but we can add a few cool things in, like the pictures swishing in from the side or something, just to add a bit of life to it.
What do you reckon? Also, don't forget to time yourselves reading -3 mins per bit. I find it's best to write it exactly as it is spoken and to simplify any sentences that create stumbling blocks -what looks great to read isn't necessarily easy to speak. keep practising and revising the words until you find the natural flow.

Developing a 'Community of Enquiry' and Contribution Audit Trail -designing an Airline Reservations Software System
In the reply to this post, i thought about the answers to the questions and came up with these answers RIC must be running scheduled flights and even if it does not run scheduled flights at present, it is important to have the system support this for future growth.
Freight services is something i did not include when thinking of the system, but i think i would like to include that option as well, which may be useful.
About hiring the plane, it is not something unique and does need to be included and it may be that RIC also offers planes for hire to other charter companies.
I think it can be made a point to add this to the system as sub functions under main categories which we have identified.
Your comments on this will be appreciated. This is an interesting example, as this particular student was not forthcoming in the face-to-face sessions, due to a lack of confidence in spoken English, but was willing to put ideas forward via the discussion board. This gave the tutor the opportunity to monitor their unique original contribution. The student actively engages in the 'Community of Enquiry' in a virtual culture, presumably due to a sense of security provided by the relatively private nature of the medium.

Developing a 'Community of Enquiry' and Contribution Audit Trail -analysing the global Information Society
Thoughts re demographic and geography: how do you perceive the demogr bit? Shall I do a rundown of stats or do you think it's more about how the demograp affects the info flow? I was thinking the latter, but with only 3 minutes, it seems a little contrived to analyse in so short a space without making huge assumptions, ie: info flow increased with younger, computer literate age groups and amongst non-indigenous peoples. This seems more theory based than analysisis this right? re geog bit, I'm thinking that it's more about how location (urban, rural, outback) effects info flow for the communities than detailing the overall geog of Australia -do you think this is right?
There seems to be several ways we can go with this, and it's important that we're all thinking the same way to make the overall presentation show cohesion, especially when it comes to linking the factors.
Would appreciate feedback so know I'm on right track and if you guys have quns re your bits too, let everyone else know.
In this excerpt, it can be seen that the student is consistently seeking feedback from her peers on her own ideas and suggestions. She is keen to engage them and draw them in to put forward their own ideas, as can be seen in the final sentence.
From a tutor's point of view, this affords a good insight into some of the dynamics of this group, as well as giving a context for the direction they subsequently took in their work.
I have found a good resource that I think would be of use to 3 of us www.abs.gov.au this is the Australian Bureau of Statistics and lists various stats relating to demographics, social changes, education (sorry, no law stuff!).
It benefits from being an official government site, this gives it authority/veracity and also shows how the country sees itself. It isn't biased but concentrates on those things important to the country (does that make sense?). This excerpt has been included because it is a good example (amongst many that could have been used) to illustrate students using the group boards to share knowledge and resources. In addition, it gives the tutor an indication of individual contributions, as well as an insight into the evaluation process that the student has undertaken in selecting this particular resource -a process that would not previously have been transparent to the tutor. And once again, we see the student using the discussion board as a means seeking reassurance of their own understanding. Not all minority groups are disadvantaged and ethnicity is not an issue as it includes the 90% caucasian.
Surely pigeonholing a minority by explaining that the reason things happen to them is because of their race and not their circumstances, perpetuates the notion that it is acceptable to do so and hides the social responsibility that Australia has towards them… This was part of a lengthy discussion between two members of one group, who disagreed on the role and importance of ethnicity as a barrier to Internet use.
Without access to the discussion board, the tutor would not have been aware of this debate, nor of the high-level of mature intellectual argument that was used by participants. The visible 'end product' would simply have been the omission in their presentation of ethnicity in their discussion of barriers to access, which was the final outcome of their deliberations.

Student reflections
Asking students to reflect on the group presentation process as part of their assessed task for the Libraries and Publishing in the Information Society module provoked unexpectedly deep and open reflections on the group processes. It also provided the tutor with a qualitatively different insight into the students' experience of using the group discussion boards than could be obtained from analysis of discussion content alone. For example, a student who had appeared to engage happily and successfully with the presentation task and with their group's discussion board made the following comments: I think the presentation would have been more polished if we had actually been able to meet up. It is a huge trial of patience to be restricted by the limitations of email and discussion boards, as posting an idea, opinion, or question does not guarantee an immediate response. Waiting for answers ultimately wasted time that would have been better spent fine-tuning the presentation. However, this is probably just reflective of my ardent attention to detail and perfectionist attitude, the end result, after all, achieved its objectives.
Overall, I would say that I have found this whole process utterly absorbing, not just for the research undertaken, but for the introspective examination it has provoked. I have discovered that despite my natural inclination towards detail and exactness, I am able to relinquish some control. Looking back, I can appreciate the team effort put into our presentation and am very proud of the outcome.
The value for the tutor of the discussion board as a contribution audit trail was reinforced by comments in the students' reflections: where issues of noncontribution were discussed, it was found that this state of affairs was reflected in the level of contributions made to the discussion boards.
…by making use of the dedicated area on the discussion boards, we could post our initial thoughts and provide easy access to them for the whole group. The boards became very useful for this work, in that we could 'float' potential ideas between ourselves and also let each other know where we had got to in our individual research. Where group members are so geographically removed from each other and may only be in university once or twice a week, the discussion boards are a welcome virtual meeting place. However, they are limited by the fact that they only work where all members are willing to contribute and engage with the assignment -if a group member chooses not to access the boards (as happened with our team) then unfortunately there is little the others can do to maintain the balance.
In this instance, a 'triangulation process' involving monitoring of contributions to the discussion board, monitoring of individual students' access to the VLE, analysis of the students' own reflections (including those of the 'non-player'), and discussion between the tutor and the whole group, confirmed that one student had not played a full role in the preparation of the group's presentation. It was therefore possible to use this evidence as a basis for moderating the marks awarded to the individual group members to reflect this imbalance.
The action plan posted on the discussion board…helped the group focus more specifically on each allotted task and was constructive in piecing the presentation together. Communication via the discussion boards was useful too, as the postings helped thought processes by making them explicit.
This is a particularly interesting excerpt, with the student spontaneously noting that posting on the discussion board 'helped thought processes by making them explicit'. This relates back to the 'thinking aloud' notion, with the concomitant benefit that other group members and the tutor were also able to learn from these thought processes.

A window into learning
While the established benefits for students of using discussion boards had been anticipated, there were additional opportunities presented to tutors as a result of the groups' use of the VLE. One of the most interesting of these opportunities is the one that we have termed 'a window into learning'. This refers to the ability that the use of group communication tools offered to audit at a much finer level the problem-solving process in which students engaged -as noted previously, it provided an insight into how they solved problems, rather than focussing solely on the end product of the process (Kolb, 1984). It became more pragmatic to devote time and effort to the individual's learning of the subject, rather than looking at the competence of their output as an indicator of their learning. For subjects that are about problem-solving, analysis and design, the manner of the problemsolving can be quite ethereal; the VLE afforded an opportunity to take a "snapshot" of this rich and dynamic process. It also became possible to gain an enhanced understanding of the group dynamics that contributed towards the final outcome.
Furthermore, observation of this process enabled tutors to intervene to adapt the teaching and learning environment in pseudo-real time. Rather than waiting for the next contact session, or worse still, a terminal assessment point, it became possible to provide students with more effective and timely feedback and guidance.
At both institutions, many of the students played an active part in the general discussion board, as well as providing information for the module resource area.
However, at both institutions it was when the mini-groups were established for private discussion, group communications and file transfer that students' use of the VLE increased significantly. In addition, there was an increase in the 'quality' of the interaction, leading to the 'window into learning' approach being identified and further developed. Overall, the efficiency and effectiveness of both modules appeared to be greatly improved through the use of the VLE. For tutors, by far the most important benefit was the rich source of data offered through this window, from which we were able to glean in-depth information into how the students were engaging in the problem-solving process.

Limitations of the research
The authors recognise the limitations imposed by the scale and the size of the samples for the research. This study involved two cohorts of around twenty students each, from two separate -and very different -institutions and subject areas. Although the findings from each context supported the notion of the Window into Learning, it is recognised that it would be useful to test this concept more fully in a wider range of environments and that a more exhaustive study would be warranted. Nevertheless, we believe that this research has shown that the potential benefits to be achieved exemplify the value of this form of content analysis of discussion threads.
With regard to reflections on what might be done differently in future, the following are of prime importance: o Capture the early enthusiasm for participation in the interaction by using the concept of 'private mini-groups' earlier in the teaching and learning process; o Make more thorough notes of the interactions to inform enhancements and modifications to subsequent teaching sessions; o Consider rewarding student participation and the quality of their contributions via the award of credits As far as students are concerned, we must work hard to ensure that students feel secure in the 'safe house' of the learning environment, and that they gain 'achievement reward' to encourage them to participate in the full use of the VLE.
Students hesitant in using the technology, or lacking confidence in posting their views on the discussion boards, will need extra support, encouragement and guidance.

Future directions
The potential for further evidence-based research into the use of online collaborative communication tools to enhance student learning opens up new possibilities for investigation into areas such as learning networks, learning objects and learning agents systems. Coupled with this is the possibility to automate the text scanning process in order to extract the required information more efficiently. Exploring the possibility of using Artificial Intelligence techniques within the 'restricted' context of the discussions might prove a beneficial avenue to follow. Evaluation of the usefulness of existing content analysis software such as Nvivo, and the like, for this kind of analysis of forum content would also be beneficial. Such work might utilise the Problem-based Learning cycle four-stages approach, as described earlier, for an analytical framework. The development of a scale of quantifiable measures, or benchmarks, to evaluate the impact of the use of technology in enhancing the learning process also deserves attention.

Conclusions
The development of the concept of 'a window into learning' has been an iterative approach that identified a beneficial outcome for students and academic staff of the collaborative use of a VLE. We believe that this concept makes a contribution to our understanding of the potential of online discussion forums. The evidence to date suggests to us that this 'window into learning' offers an important insight into student problem-solving processes, giving us the potential to bring about a richer, qualitatively enhanced, autonomous learning environment for the student and enabling us to achieve the aims of efficiency, enhancement and extension represented by the e 3 model.
Taking the wider perspective, if the full potential offered by the use of the VLE is to be realised, the role of the tutor will need to adapt. In particular, there will be a need to focus sufficient time on assimilating the qualitative information offered by the window on student interactions. It is envisaged that the enhanced information on the students' understanding of the topic will allow for more customised seminar and tutorial work to be set, together with more appropriate assessment tasks. However, this aspect will need further investigation to provide more reliable approaches that can help to maximise the role of the tutor without over burdening them.
We believe the Window into Learning is an exciting concept that we have only just begun to explore. The potential for further investigation into how we -and ultimately, our students -can benefit from the insights gained from opening up this window is considerable. 1 Used here in the synergistic sense of the whole being more than the sum of the parts.