An open learner model for trainee pilots

This paper investigates the potential for simple open learner models for highly motivated, independent learners, using the example of trainee pilots. In particular we consider whether such users access their learner model to help them identify their current knowledge level, areas of difficulty and specific misconceptions, to help them plan their immediate learning activities; and whether they find a longer-term planning aid useful. The Flight Club open learner model was deployed in a UK flight school over four weeks. Results suggest that motivated users such as trainee pilots will use a system with a simple open learner model, and are interested in consulting their learner model information both to facilitate planning over time, and to understand their current knowledge state. We discuss the extent to which our findings may be relevant to learners in other contexts. Keywords: Intelligent Tutoring Systems; learner model; open learner model; trainee pilots DOI: 10.1080/09687760802318436


Introduction
Intelligent Tutoring Systems (ITSs) offer individualised learning assistance.For example, such assistance may include providing exercises of appropriate difficulty and content, and selecting suitable learning material according to the learner's current learning needs.
Traditionally an ITS contains three models.The domain model is a representation of the domain (e.g.concepts, relationships between concepts, prerequisite relationships).The domain model allows the system to generate and understand the content it is teaching, and evaluate the accuracy of the user's input.The learner model is a model of the individual's current understanding of the domain, constructed based on interactions between the user and system, such as material viewed, time spent on material and, in particular, responses to questions or attempts at problem-solving.A learner model is therefore a representation of the learner's knowledge (including skills, competences and problematic issues), and is dynamically updated -i.e. it is a model of current knowledge, not an overall performance score or summary.The pedagogical model is a model of teaching knowledge or strategies.Teaching strategies allow an ITS to individualise its teaching: the system interprets the user's knowledge from their learner model, comparing this to the domain model, and then makes a variety of teaching or guidance decisions such as choice of topic, level of detail of information or explanations to present, method of presentation of information or explanations and generation or selection of appropriate exercises.ITSs are being developed for a range of domains, from mathematics (Melis, Haywood, and Smith 2006) to astronaut training (Dubois, Nkambou, and Hohmeyer 2006) to music (Lahart 2005).
In this paper we focus on the learner model, and the specific case of open learner models (OLMs).There is an increasing trend towards opening the learner model to the learner -i.e.externalising the model so that it is both visible and interpretable by the user.Examples include OLMs for computing subjects such as programming (Brusilovsky and Sosnovsky 2005) and databases (Mitrovic and Martin 2007); technical terminology (Dimitrova, Brna, and Self 2002); and foreign language learning (Bull and Nghiem 2002).OLMs can be simple in format -e.g.skill meters giving a high level overview of knowledge (Mitrovic and Martin 2007); or complex -e.g.conceptual graphs (Dimitrova, Brna, and Self 2002) and hierarchical tree structures (Kay 1997).By externalising the model a number of benefits can be achieved, including use of the OLM to raise learner awareness of their current knowledge state, to help the student plan and reflect upon their learning, and to allow them greater control and responsibility for their learning through identification of their learning needs.OLMs can therefore help users to set personal goals, plan ways to achieve these goals, and monitor their progress towards these goals (Kay 1997).
Positive results have been found for users inspecting even a simple learner model using the OLMlets OLM deployed in a range of university courses in electronic, electrical and computer engineering (Bull, Quigley, and Mabbott 2006).The OLMlets study was with users mostly taking compulsory courses.In such cases students are likely to have varying levels of motivation.Given the positive results amongst such students, we wish to investigate the potential for simple OLMs to support learning amongst different types of learner.
Here we consider the case of highly motivated independent learners, using the example of trainee pilots studying for their Private Pilot's Licence.While we acknowledge that not all trainee pilots will complete a course, and private pilot trainees may be less motivated than those on more advanced pilot courses (Frederick-Recascino and Hall 2003), it seems likely that trainees who choose to continue at a flight school will be at least as motivated to succeed as students in other learning contexts.(As will be seen in the next section, trainees are likely to be intrinsically motivated learners -see Ryan and Deci 2000 -whereas students may be externally motivated, for example if they perceive their degree as a necessary step towards their chosen career.)Our first question, then, concerns whether results for use of an OLM with typical university engineering students are also applicable to intrinsically motivated independent learners.
A second line of investigation concerns the method of presentation of the learner model.Use of OLMlets revealed that, of the five simple OLM views available, 'skill meters' -part shaded bars to represent knowledge levels of the various topics -were the most frequently used (Bull and Mabbott 2006).Skill meters are also the most common form of simple OLM.Nevertheless, in OLMlets, even though skill meters were the most frequently used OLM view, not all students used the skill meters and some used these in combination with other presentations of their learner model.It was therefore recommended that alternative presentations should be considered in the design of OLMs to accommodate individual preferences.In this paper a new alternative format of learner model presentation is introduced for use alongside skill meters in order to extend previous work on presentation preferences for OLMs.In our second question we seek, therefore, to discover whether a different OLM presentation may be found useful by some students.
We also investigate users' use of their OLM as a support for planning over time.Skill meters and equivalent learner model views allow learners to monitor their changing knowledge and plan their learning in the short term as they show a 'snapshot' of knowledge.A third model view is also available in our system, which is solely concerned with aiding longer-term planning -hypothesised to be particularly useful to the learning context of the trainee pilots who are working independently.Our third question therefore asks whether independent learners use the more common 'snapshot' OLM or a longer-term planning aid (or both).
Research in open learner modelling is still in the early stages.There are therefore very many questions remaining to be investigated, relating to a broad range of issues including learning gains, learner model presentation preferences, what students use OLMs for, what information they attend to, how OLMs may support collaborative interaction, why some students do not access their OLM, use of OLMs over time, use of OLMs in different courses and the utility of OLMs for different types of user.(See Bull, Dimitrova, and McCalla (2007) for further discussion of research questions.)Using the 'Flight Club' OLM, this paper considers just some of the early questions.We investigate whether previous work on the use of simple OLMs is also relevant to motivated, independent learners -i.e.we aim to find out whether this type of OLM approach might also be of benefit to the kind of learners who have a high drive to succeed, and who may already have successful learning strategies in place.
Our results with trainee pilots may generalise to other motivated learners, such as intrinsically motivated students who find themselves with little in common with extrinsically motivated students who may dominate their course.The results may also be relevant to distance learners who, like trainee pilots, tend to be older and more mature (Tucker n.d.).
In summary, we consider (i) whether our trainees inspect their learner model; (ii) which learner model presentations are found useful; and (iii) use of the open learner model as a short-term versus long-term planning aid.

'Flight Club' users and the learning context
Flight Club is targeted at trainee pilots working towards their Private Pilot's Licence (PPL).This licence comprises a course of practical training and a series of seven ground examinations, set in accordance with the Joint Aviation Regulations (JAR) syllabus (Campbell 2005).Flight Club is designed to aid the user in learning the material for each of the ground examinations by helping them become more aware of their knowledge state, and providing information to help them set personal targets.
Ideally, of course, in an educational setting the complete learning context would be considered, incorporating an understanding of how technology can support learning taking account of psychological processes of learning and wider environmental features, then developing appropriate instructional strategies to encourage learning (Alavi and Leidner 2001).In our setting this would involve a synergy of understanding about the learner's theoretical knowledge and any difficulties through learner modelling, and the interaction between these data and what is being undertaken in the practical flight hours.However, PPL trainees receive no formal instruction for ground examinations, with the emphasis being on independent learning; and there is no direct correspondence between what trainees might be doing during their practical training, and the stage they have reached in their ground examinations.An OLM approach of raising learner awareness of their knowledge and potential difficulties therefore appears to be an especially useful way to support learning in this context.Successful use of an OLM in this situation would suggest the potential for OLMs to support independent learning more generally.

The 'Flight Club' system
Trainees may consult information based on the JAR syllabus available in Flight Club, or consult the standard materials -either while using the environment or after having identified their needs through questioning by the system.Questions are presented in multiplechoice format.There are two main reasons for this: (i) multiple choice questions can be designed to elicit known common difficulties; and (ii) the JAR PPL examinations comprise multiple choice questions, so Flight Club questions match the format of the examination towards which users are working.Flight Club uses the responses to construct a model of the learner's knowledge level, areas of difficulty and specific misconceptions.Knowledge level is identified by referring to the system's representation of the expert domain knowledge; misconceptions are identified by comparing the learner's input to a misconceptions library; and general difficulties are identified by learner 'knowledge' that neither matches information in the domain model nor the misconceptions library -i.e.these are things that the learner does not yet know, but about which they do not have a specific misconception.
Flight Club is not a full ITS -it is primarily concerned with questioning the learner, modelling their knowledge and presenting the learner model to the learner.The adaptivity of the system is therefore focused around open learner modelling rather than, for example, adapting the content or explanations to the user.Kay (1997) argues that giving learners responsibility for their learning can enhance the effectiveness of learning.This also concurs with the aim of promoting learner autonomy and formative assessment through use of an OLM (Bull, Quigley, and Mabbott 2006).The aim of Flight Club is therefore to help independent learners identify what they need to do, either using external materials or (nonadaptive) materials in the system.Thus Flight Club facilitates identification of learning needs by showing learners representations of their understanding in their learner model, but at the same time allowing the learner to retain control and responsibility over the decisions regarding their learning -appropriate for advanced, motivated adult learners.

The 'Flight Club' open learner model
Because Flight Club users are learning independently, with no teaching support, a simple learner model presentation was chosen for the initial evaluation.The OLM can be viewed in three forms: skill meters, aircraft positioning and runway schedule.
Skill meters represent user knowledge as a subset of the system's expert knowledge, with green (shaded medium grey in Figure 1) to portray the proportion of the user's knowledge in relation to full knowledge.For example, Figure 1 shows examples of both when a user currently understands 50% of a topic as inferred from their most recent responses (Air Law), and also when they have a weaker understanding of the subject (Meteorology).The lighter grey area represents unknown or currently problematic issues.Red (shaded dark in Figure 1) indicates the existence of one or more misconceptions in a topic.The skill meter becomes 'overridden', turning red (Human Performance), in order to focus the user's efforts onto resolving the problem.This is because trainees are required to prepare for ground examinations whilst concurrently conducting practical training -a misconception could be dangerous if applied in a real situation.accordance with previous results with university students who consulted descriptions of their misconceptions when these were provided (Bull, Quigley, and Mabbott 2006).The correct concepts and information are available for consultation in Flight Club's learning materials.The second way of viewing the learner model is through the aircraft positioning presentation (Figure 3).This view includes the same information as the skill meters, but portrayed in a different manner.Similar to the skill meters, the user is made aware of the proportion of their knowledge as a subset of the topic.This is achieved through a pictured aircraft taking off from a runway.As knowledge increases, additional aircraft appear, with an aircraft in the top left corner representing expert knowledge (i.e.where the knowledge of the user is equal to the system's representation of the topic).In contrast to the skill meter, here knowledge of only one topic is displayed on the screen.
The existence of misconceptions is indicated by the dial in Figure 4, based on a cockpit instrument (bottom left of the screen in Figure 3).When a misconception is identified, the needle moves into the red area at the bottom of the dial.As with the skill meters, the identification of a misconception causes the representation of knowledge level to be overridden with all aircraft disappearing from the screen, to direct the user's attention towards resolving the misconception.Learners view their misconception description in the same form as for the skill meters (Figure 2).The two learner model views described above are primarily concerned with helping learners identify their current knowledge and difficulties, thereby facilitating self-monitoring and immediate planning through allowing them to compare their knowledge level against the system's expert knowledge, and making them aware of any misconceptions.The third view, runway schedule (Figure 5), aims to help users plan their learning over a longer period.Users provide information on the date of their examination to allow the system to calculate a number of benchmarks as appropriate for the particular examination, at dates up until the examination.Runways start unfilled, and as the system's representation of the user's knowledge increases, the runway slowly fills (unfilled portion(s) represent a gap in user knowledge).If the user does not fill the runway to the benchmark point by the date shown, they are alerted that they may have difficulty passing their examination.Should the learner's progress be faster or slower than planned, the schedule and benchmarks can be recalculated so that a learner's different constraints at various times during their study can be accounted for.

Evaluation
This section describes the initial evaluation of Flight Club deployed in a UK flight school.Our purpose was to build upon the results of a simple OLM used by university students   studying in traditional face-to-face courses (Bull, Quigley, and Mabbott 2006), investigating the potential for a simple OLM for highly motivated, independent learners.As Flight Club does not provide adaptive teaching and given that a main aim is to prompt studying away from the system, we do not attempt to measure learning gains during use of Flight Club.Instead, we investigate the issues raised previously: whether trainees choose to use a simple OLM; the learner model views they use; the purpose for which they use the views; and the perceived utility of the OLM as a learning aid.

Participants, materials and methods
Flight Club was installed on a computer at the participating flight school, for a four-week period.All current trainees were invited to try the system, though usage was voluntary.All 16 PPL trainees used Flight Club.As the JAR PLL assessment consists of seven examinations taken at different dates, the participants used the system in accordance with the examination they were working towards.Use was unconstrained and unsupervised, and at the time and frequency of the user's choosing.
All interactions were logged, and the learner models were available for analysis.Participants completed a questionnaire, addressing issues relating to the features of each learner model view.Responses were on a five-point scale indicating the extent of agreement with a statement of the utility of a feature.

Results
The logs reveal that the majority of users viewed their OLM using more than one presentation.A breakdown of usage levels and learner model presentations accessed by each user is shown in Table 1.All users viewed each presentation while becoming acquainted with the system.These first viewings are not included -i.e. the results relate to viewings that took place during learning interactions.
Ten of the 16 trainees used Flight Club on a single occasion, with six using it multiple times.Individuals answered between 29 and 109 questions (mean 65.75; median 73.5).The skill meters and aircraft positioning views are equivalent in content.Five users used both; nine used only one (with a near-equal split).Amongst those who used each of the views, usage of the aircraft positioning view tended to be higher (mean 4.2; median 4) than use of the skill meters (mean 3.33; median 3).Eleven users used the runway schedule (mean number of accesses 5.36; median 5), and all but one of the six trainees who used the system multiple times accessed this longer-term planning aid.
Table 2 details the identification and viewing of misconceptions, and access to learning materials, as recorded in the system logs.
Five users had misconceptions at some point.Each of these accessed the misconception descriptions.Only three of the 16 users consulted the learning materials.
Tables 3, 4 and 5 show the number of users who found each of the presentations useful in providing a specific benefit associated with the OLM, obtained from the questionnaire data.
Most users found the skill meters and aircraft positioning views helpful for identifying their difficulties; most also found the skill meters useful for identifying misconceptions, while only five found the aircraft positioning view helpful for this purpose.The aircraft positioning view was also found useful for identifying or quantifying knowledge level.Only two people found the skill meters useful for this purpose.A minority found the skill meters and aircraft positioning useful as a planning aid.In contrast, all found the runway schedule useful as a means to aid the planning of their learning.(The runway schedule was not designed to address the other features, and accordingly was not found to be helpful for those purposes, by most users.)Fifteen of the 16 users stated that they found the runway schedule useful in general; 13 found the aircraft positioning view useful; and two users considered the skill meters to be generally helpful.

Discussion
In this section we discuss the results with reference to our three main questions from the introduction: (1) Will trainee pilots use an OLM? (2) Which learner model presentations will be found most useful?(3) Will trainees use the OLM as a short-term or longer-term planning aid?
We then consider the relevance of our findings with this population, for other learners.

Will trainee pilots use an OLM?
Our first question relating to whether trainees will use an OLM necessarily includes use of the environment as a whole, since the OLM is the mechanism through which users obtain feedback on their understanding, modelled according to their interaction.This can be addressed by looking at the usage statistics.All current PPL trainees were invited to try Flight Club, but given the context of learning and the target users, usage was voluntary.Nevertheless, all used the system, and it is unlikely that the usage levels would have been maintained if trainees had not perceived some benefit to their exam preparation.Depending on their stage of learning and preferred learning patterns, learners may find it more or less useful to return to Flight Club over the four-week period.We therefore do not consider frequency of login as necessarily indicative of the general utility.Ten users used the system on a single occasion, with the remainder using it between two and four times.Those who used it multiple times will likely have returned because it was helpful, and because they felt there was further need to identify what to study for their exam.Given the extent of interaction also in the single sessions, it is likely that many of these users also found it useful to help them identify where to focus their study.
Only three participants viewed the learning materials.Considering the general usage levels, this suggests that most were using the OLM as intended -to identify where they needed to invest further effort; and most were choosing to follow this up outside the system.Nevertheless, the fact that some users did use the materials suggests the utility of including these for users who wish to consult information immediately upon encountering a problem or a gap in their knowledge.
We would not expect an OLM such as that presented here to be a preferred source of information or practice for all trainees.Our aim is to offer an environment for those who find it useful, to help them identify their learning needs and plan their learning accordingly.It appears that some learners working in a highly independent learning context will indeed use a simple OLM to support their learning, and this may make the provision of an OLM worthwhile.
Which learner model presentations will be found useful?One of the aims of this study was to build upon previous work (Bull and Mabbott 2006) where the most common form of simple OLM, the skill meter, was used more frequently than four alternatives with equivalent information.Our second question addresses this.Using Flight Club we investigated whether users preferred to review their knowledge during their interaction using the skill meters or an equivalent alternative, aircraft positioning.In terms of overall usefulness, only two users agreed with the statement that the skill meters were useful, whereas 13 agreed that the aircraft positioning presentation was useful.This is supported by the logs which identified more viewings of the aircraft positioning view.Finding that the skill meters were not used more than the aircraft positioning presentation is in contrast to the results of Bull and Mabbott (2006), but coheres with the findings of Mabbott and Bull (2004), where participants had preferences for a variety of learner model views.In the study of Mabbott and Bull, complex learner model presentations were provided (e.g.concept map, prerequisites structure, hierarchical trees).Our results now suggest the same might be true in some contexts in which learner model information is presented in a very simple format.Certainly it suggests that skill meters may not always be the preferred simple learner model presentation.This result may be due to the specific alternative view chosen for Flight Club (aircraft positioning), the fact that it focused on a single topic, or the different type of user.Whatever the reason, at this stage our contrasting results with OLMlets and Flight Club strengthen the recommendation of providing alternative learner model views even when simple presentations are used.This is further supported by the fact that five users in this study consulted both views.Furthermore, although the two views were designed to carry the same information, it seems that users were using them for different purposes.Both were considered useful for identifying difficulties.However, only the skill meters were generally considered useful as a means to raise awareness of the existence of misconceptions; and only the aircraft positioning view was generally found useful to identify level of knowledge.The overall preference for the aircraft positioning view, despite the fact that the skill meters were considered better for highlighting the existence of misconceptions, may be because only five users had misconceptions.With more finegrained learner modelling, or in a subject with typical entrenched misconceptions, or with learners of lower ability and/or motivation, use of the skill meters might be higher relative to other views, as found by Bull and Mabbott (2006).Therefore, as suggested above, further research should be undertaken as learners may be more likely to use an OLM if it is presented in a format that they prefer.
Will trainee pilots use an OLM as a short-term or longer-term planning aid?Our third question regarding use of the OLM as a short-term versus longer-term planning aid can be addressed by comparing use of the skill meters and aircraft positioning views, with the runway schedule.In the runway schedule, focus is primarily on aiding users in planning and monitoring learning over time, rather than raising awareness of their current knowledge.As with the other views, the runway schedule was also used by the majority of participants.While finding the skill meters and aircraft positioning views useful for identifying knowledge, difficulties and misconceptions, users were less concerned with short-term planning.It would be interesting to find out what users were doing with the information about their knowledge state if it was not to aid immediate planning -were they simply not thinking explicitly about planning, or was the decision of what to do so straightforward for them upon viewing their learner model, that they did not consider it to be planning?Whatever the answer, usage suggests that trainees do appear to perceive some immediate benefit to using the OLM.However, it seems clearer that trainees find it helpful to use an OLM for planning longer term -a less common application of open learner modelling.

Future directions
Given that all trainees voluntarily used Flight Club independently, it would be worth investigating further questions with this type of user.The sample of 16 was sufficient to identify interest in and use of the OLM -differences in use in a small sample can illustrate a need to take individual differences into account.Nevertheless, a larger sample would allow us to confirm our results and investigate the purposes for which users were accessing the different views more fully.We could investigate the extent of the use of the learner model view designed to aid planning over time, and whether this reflects the needs of this group who are studying independently for exams.Would similar results be found for distance learners who, despite the additional academic support recommended for distance learners (see e.g.Clarke 2000;Miller and Husmann 1996), will still be undertaking much independent work?Perhaps more interestingly, might this type of learning support also be helpful for those taking part in face-to-face courses, to guide their private study?Work with OLMlets has already suggested this to be the case for university students (Bull, Quigley, and Mabbott 2006).From the Flight Club results it would appear that an OLM approach is also suitable for particularly motivated learners, and so could perhaps be used by such students in other contexts (such as university courses) to help push them further in their study (e.g. to help them extend their learning beyond that of the rest of the group).Might this approach also support students who are in danger of 'getting lost' as others progress?Access to representations of their understanding might help such students identify their needs prior to an assessment.Of course, the underlying problem of a general lack of motivation for independent study is not addressed by this, but an OLM could help learners to get back on track when they most urgently need to, by helping them to prioritise their efforts.
Many issues remain for investigation.Here we have suggested the utility of a simple OLM for motivated learners for both immediate and longer-term planning.We have also found that our users do not have the same preference for learner model views as indicated in previous work.This demands further research in this area.

Conclusions
This paper has presented Flight Club, a simple OLM to aid immediate and longer-term planning by motivated, independent learners: trainee pilots studying for their Private Pilot's Licence.All trainees used the system, finding the learner model views useful for different purposes: identifying knowledge levels, difficulties and misconceptions, and planning over time.Users had different preferences regarding the model views designed to raise their awareness of their understanding, suggesting the utility of allowing a choice of format for accessing learner model contents.
Given usage amongst learners who had no additional support for learning, we suggest that OLMs could be a helpful aid in similarly 'unfavourable' learning conditions.We also consider that OLMs might be usefully integrated into more supportive environments where learners are learning independently, such as distance learning courses.Furthermore, if OLMs can be helpful for these situations, they may also be able to help the more traditional motivated student to push themselves further, when studying amongst a group who are generally less motivated.

Figure 1 .
Figure 1.Example of Flight Club's skill meters.When a misconception has been identified, a button appears allowing the user to view the misconception.A brief description of the misconception is displayed (Figure2), in

Figure 3 .
Figure 3.The aircraft positioning learner model view.

Figure 4 .
Figure 4.The aircraft positioning view presenting a misconception.

Figure 3 .
Figure 3.The aircraft positioning learner model view.

Figure 4 .
Figure 4.The aircraft positioning view presenting a misconception.

Table 1 .
Level of use and learner model views used.

Table 2 .
Viewings of misconceptions and learning materials.

Table 5 .
Users' perceptions of utility of runway schedule.

Table 4 .
Users' perceptions of utility of aircraft positioning.

Table 3 .
Users' perceptions of utility of skill meters.