An Evaluation of Learning Resources in the Teaching of Formal Philosophical Methods

In any discipline, across a wide variety of subjects, there are numerous learning resources available to students. For many students the resources that will be most beneficial to them are quickly apparent but, because of the nature of philosophy and the philosophical method, it is not immediately clear which resources will be most valuable to students for whom the development of critical thinking skills is crucial. If we are to support these students effectively in their learning we must establish what these resources are, how we can cont inue to maintain and improve them, and how we can encourage students to make good use of them. In this paper we describe and assess our evaluation of the use made by students of the learning resources that are available to them in the context of learning logic and in developing their critical thinking skills. We also assess the use of a new resource, electronic han dsets, the purpose of which was to encourage students to respond to questions in lectures and to gain feedback about how they were progressing with the material.


Introduction
Teaching students to read and think critically is fundamental to philosophy; after all, according to the philosophy benchmarking document, it is hoped that graduates in philosophy will possess a 'precision of thought and expression in the analysis and formulation of complex and controversial problems' and, that they will develop the abilities to 'abstract, analyse and construct sound arguments and to identify logical fallacies' and 'recognise methodological errors, rhetorical devices, unexamined conventional wisdom, unnoticed assumptions, vagueness and superficiality ' {Philosophy: Subject Benchmark Statements, 2000, p. 3).Abilities of this sort come naturally in only a lucky few in any generation, and the others are led in their general direction by the teaching strategies that are currently employed in classes on critical thinking, reasoning and logic.The success of these strategies is not altogether clear and new strategies, for example, the production of Web resources, are being developed and made available to students with too little understanding of whether they will actually benefit them.
As the lecturer on the three courses -Level 1 'Introduction to Philosophy of Mind ' (200 students).Level 2 'Introduction to Logic' (100 students), and Honours-level 'Kantian epistemology based on the Prolegomena (1891)' (20 students), I have a particular interest in discovering whether resources are being employed successfully.A lot of energy and imagination has been invested in delivering lectures in a lively and engaging way, in creating handouts and tutorial worksheets, in drawing up reading packs for Level 1 and Honours, and in devising electronic course materials for Level 2. It is clearly in my interest to find out if I have been spending my time wisely, and if the resources that I encourage the students to use are in fact the resources that help them to understand the course material and to develop their philosophical muscles.
A characteristic of the Scottish University system, as exemplified in the Arts, Social Science and Science Faculties of the University of Glasgow, is to have large pre-Honours classes, and pre-Honours philosophy is a compulsory component of many Arts degree profiles.As a result, we can find ourselves teaching the full range of philosophical topics to first-and second-year classes of upwards in total of 800 students, with a wide variation in their abilities, in their levels of commitment and motivation, and in their background knowledge; all of which can, and do, produce wide variations in individual student performances.It is presumed that differently able, committed, motivated or knowledgeable students will respond to different resources and teaching strategies in distinct ways.This makes these large, mixed classes an interesting testbed for the effectiveness of a variety of teaching models.
It is also presumed that some students will have come to university with effective learning strategies and that, if these strategies transfer well, their success will not be dependent on the resources or teaching models that are employed at this level.Unfortunately, there seems no clear way of testing the veracity of this assumption, so for now it will suffice to acknowledge it as a possible influencing factor, but one that cannot be investigated any further at this time.
We take as a given that if a student is to be successful in their learning, then recognition of which resources will serve them best is going to be crucial.In relation to this we must also acknowledge that there will be a subjective element in the resources that an individual will utilize; however, we do seek to identify whether or not weaker students complete their course, and complete it successfully, by employing different resources from the students who succeed easily.In particular this will be carried out in relation to the students in a second year Introduction to Logic class.We have the examination results for those students in this class who completed a questionnaire and a comparative analysis of the grade they achieved against the learning resources they found valuable is easily possible.This will also allow us to establish if there is any robust correlation between the use of particular resources and later examination success.

The strategy
An experienced independent evahiator was employed to carry out the analysis.There were three reasons for this: the first was that a particular set of skills is required for devising complex questionnaires of this sort; the second was that, as a teacher on the courses, I might have had difficulty eliciting objective -or just plain honest -responses from the students; and the third was that it would have been too time-consuming to do alongside writing and delivering the lectures.
We made use of a model of Integrative Evaluation (Draper, Brown, Henderson and McAteer, 1996) using a resource questionnaire as our main instrument (Brown, Doughty, Draper, Henderson and McAteer, 1996).Questionnaires of this sort make it possible to measure, using student self-report, which resources students use, how much use they made of them, and how much they value each of them; thus providing us with both quantitative and qualitative information.The resources that were considered included lectures, Web resources, set texts, other texts, student-led and non-student-led tutorial discussion, electronic texts and electronic course materials, access to the lecturer, handouts, the Library, discussions with their tutor, discussions with other students, workshops, the Tomassi (Web-based) Theorem Prover (Level 2 Logic only) available online at http://www.oxford-virtual.com/Philosophy/Tomassi/,class assignments, past examination papers, specimen questions, and electronic handsets.This last item was being tried for the first time in the logic lectures to see if it helped to elicit responses from more students in the class, and also to see if there were particular types of questions that could be answered more easily by students in this way (Stuart and Brown, 2002).
Students in all three courses were presented with questionnaires asking them specific questions about what resources they had and had not used, and how useful they felt the resources had been in understanding the course material and the development of their philosophical skills.They were also asked to elaborate on their responses where relevant and to say if they intended to proceed to Honours in Philosophy.Our aim here was to see if there was a particular set of resources used by those students wishing to take the subject further.
Thus our study concentrated on the resources used in Levels land 2, feedback from Honours-level students was also gathered so that we could see how resource use changed over a prolonged period of study.We also carried out a retrospective study of the resources those Honours students recalled as being most useful in their first and second years and, as a side study, we conducted a parallel investigation of the reasons students gave for progressing to Honours in Philosophy or the reasons they give for choosing another course of study.Our aim was to try and establish whether students moved away from philosophy because of the methods of teaching, because they did not feel well supported in their study, because of the subject matter, or simply because they had always intended to take Honours in some other course.

Findings
Having looked at both the use and the usefulness of resources as rated by the students it was noticeable that there was a steady increase in the number of resources being used by students as they progressed through their courses, developing their skills and proceeding to the next year.In Level 1 students used, on average, six or more resources, in Level 2 they used eight or more and by the time they reached Honours they were using ten or more.
There seems also to be a minimum number of resources -excluding the use of electronic handsets (see penultimate section) -used by all students.In Level 1 all students used at least in Level 2 all students used at least four and in Honours all students used at least five.In all levels lectures, tutorials, handouts and course-specific reading packs or text books, were used by more than 50 per cent of the class.In Level 1 and Honours, though not in Level 2, the library was also used by more than 50 per cent of the class.The light use of the library by Level 2 students may have resulted from the seemingly self-contained nature of the Logic course.There was a set text -Paul Tomassi's Logic (1999) -that all students taking the logic component were advised to buy; electronic course materials and class assignments were made available to the students in the lectures and on the Web.From this they may have concludednot unreasonably -that it was not a course that required supplementary reading.
Students intending to take Honours philosophy made a lot more use of class assignments, the recommended text, and other relevant texts, but much less use of the Web-based course materials.However, of the 55 per cent of intending Honours students who did use the Web resources a large proportion (76 per cent) rated them highly.It might be that many of the students who were intending to take Honours in philosophy had a conception of philosophy as a subject that does not include making use of Web-based materials, but if they are being reviewed so favourably by such a high proportion of students perhaps we ought to be encouraging the others to, at least, investigate them.
The less experienced students were more conservative in their choices of which resources to use, preferring to stick with those things with which they were most familiar.But by the time they reached Honours more than 50 per cent of the students were also making use of revision tutorials, other texts, specimen and past examination papers as well as face-to-face and email discussions with their tutor.(Revision material was only available after the questionnaire had been completed by the Level 1 and 2 students, so we only have data from those students who said that they intended to make use of them.)This is probably the result of a number of factors, among them the fact that the student has a greater understanding of the system and what is available, but it also provides evidence of a maturing relationship with the subject and with their method of dealing with it.Discussions with tutors and lecturers is also much more popular in Honours; again there are clear reasons why this would be so.The tutorial groups are smaller in Honours and students can develop closer relationships with the department, their peers and their individual tutors within the department.On top of this the students' confidence will have grown.They will have successfully completed their first two pre-Honours years of study; they will have satisfied the faculty and departmental entry requirements for Honours and will have been selected to carry on at Honours level for two more years.This is a substantial achievement and students progressing to Honours will feel satisfied -justifiably -that they are getting to grips with the subject and its methods.
With all the learning resources available for students to use, it is surprising that there is still a remarkable consistency among those that the students consider to be most valuable.
What is not surprising is the fact that lectures and tutorials were ranked as the most useful resources by all students at all levels.The students enjoy having someone to communicate with directly.They report that they like having someone in the room, someone to whom they can relate and of whom they can ask questions in real time when the answer is of immediate concern to them.But there are other reasons for attending the lectures.Some of the mature students said that attendance at the lectures was crucial because trying to understand the subject by using the text alone could be an overwhelming experience.Other students added that the incremental nature of the logic course meant that missing any one lecture made it very hard to catch up.In both these cases part of the lecturer's role is as a guide and not just the conveyor of information.But, on a more mundane note, we might interpret the fact that all the students who had completed a questionnaire and reported that they had used the lectures, and that they had been 'extremely useful', were students who had been at most, if not all, of the lectures and who were there on the day the questionnaires were distributed.Indeed it would have been inconsistent if we had not received a 100 per cent figure for lecture use by these students and, at this stage in the course, the last formal lecture, it would have been unusual if the students in attendance had not rated the lectures as useful or even extremely useful.
The 100 per cent use and usefulness rating of tutorials in Level 2 and Honours was a little more interesting, especially in relation to the 93 per cent of Level 1 students who found tutorials useful, but it does not tell us anything that we did not already know, and that is, that the more experienced students will have realized the benefit of airing and discussing their problems within a localized tutorial setting where there are fewer people and the atmosphere to facilitate the exercising of ideas.

Level 2 Logic
A comparative study of the Level 2 logic students' resource use and their subsequent examination performance shows that there is a clear correspondence between lectureproduced (lectures, tutorials, handouts, class assignments with answers on the Web and handsets) and those that the students who were successful claimed to have found very or extremely useful.This is not only reassuring but in striking contrast with the numerous cases where colleagues have lavished time on creating material that has met with only a lukewarm reception.In Level 2 logic, as in Level 1 'Philosophy of Mind and Honours Kant', the resource most frequently quoted as increasing the students' interest in the subject was lectures/lecturer/lecturing style.Students report being concerned about, and influenced by, the approachability and enthusiasm of the lecturer.
It also seems true to say that those students who do well seem to be much more confident with the resources they are using.This may not necessarily be because better study skills and resource selection go with achieving higher marks, but because, in this particular subject, students can more easily and accurately perceive how well they have understood the material and students having trouble in getting to grips with the subject flail around in search of anything that seems even remotely helpful.Mature students reported that they read around and try things out, whereas the younger students are a lot more conservative and focus only on those things that have been recommended.All students made use of lectures, tutorials and paper handouts; a little over half the class made use of the set text, and most of the students used the handsets in class.One very noticeable difference was in the use of class assignments.Assignments with proofs to carry out and truth-tables to check were given out in class for the students to take home and complete.The answers were posted on the Web one week later, with more difficult problems then discussed in the lectures.The students who achieved marks in the 15-20 range, that is, more able students, reported that they had made good use of these assignments and had found them valuable.The weaker students made much less use of them.Similarly a higher proportion of intending Honours students (65 per cent) used and valued class assignments than those who were not planning to proceed to Honours (43 per cent).
It is evident that a smaller proportion of good students used a large number of resources.Indeed it is striking that almost 50 per cent of the students who scored from one to ten out of twenty used eleven or more resources, and of those who scored from eleven to twenty out of twenty only 10 per cent used so many.The average number of resources used by students in the low, middle and high scoring bands, is eight, seven, and six, respectively, with low scoring students occasionally trying out up to fifteen different means of getting to grips with the subject matter.
Attendance at the last formative lecture -where the questionnaire was completed -was strongly correlated with achieving high examination marks (see Table 2), although a few students did demonstrate that it was possible to get high exam marks without attending that lecture.Those students who were not present at that lecture achieved marks of between one and nineteen out of twenty in the examination, whilst students who were present achieved marks of between six and twenty out of twenty with five students achieving full twenty point scores.

Use of handsets -a new resource
One of the most surprising and interesting findings was that a majority of the students enjoyed using the handsets to respond to questions in lectures.The handsets had not been available in previous years, but instead had been incorporated into the class at the last minute when I was encouraged to use them by a colleague who was already successfully employing them in a number of other disciplines (Draper, Cargill and Cutts, 2002).I thought that they would enable me to provoke an entire class into responding to my questions rather than the usual few conscientious and vocal people who sit towards the front, but I had worried that the students might think of them as a novelty and become quickly bored with their intrusive use in the class; my anxiety was unfounded.
Electronic handsets are like TV remote control devices and every student is given one before the lecture begins.Students are asked multiple-choice questions that they must think about briefly and respond to fairly quickly, and each student transmits the number corresponding to their chosen answer.The answers are then collected on a laptop which displays, via the room's projection system, a bar chart representing the distribution of the responses.One of the essential features of the use of this equipment is that both the students and the lecturer get to know the distribution of responses and, in confidence, how their own response relates to that distribution.The element of anonymity -students pick out randomly numbered handsets and numbers cannot be traced back to individual students -encourages everyone to contribute and, unlike in face-to-face discussion, each individual can express the choice they incline to rather than the choice they would feel able to explain and justify to others.They provide the student with the ideal circumstances under which they can try out their responses to questions without any fear of embarrassment if their answer turns out not to be right.
The sorts of questions that can be asked vary quite considerably, but typical examples had only two possible answers, which were easier, and more importantly, quicker to ask.For example, 'If the options for the next stage in this proof are MPP or MTT, which one would you choose?Press 1 for MPP.Press 2 for MTT' I also asked more complex questions about how well the class felt they were understanding material and, although the responses were subjective, they were informative enough to provoke me to redirect my teaching.
The handset is a new and unfamiliar resource for the students, which changed the nature of the lecture sessions.Student can be passive recipients in a traditional lecture; being provided with knowledge from the tutor.Alternatively lectures can be made more interactive, by the lecturer attempting to engage the attention of the students by asking questions.However, usually only a few students speak, and so the state of understanding of the rest of the class remains opaque.Handsets facilitate, an interaction between students and lecturer that keeps the students thinking and concentrating on the material throughout the lecture.To encourage student-student interaction students were occasionally asked to answer a question without thinking about it for too long; they were then asked to discuss it with their neighbour and answer it again.There was a significant shift towards a greater number giving the right answer on the second attempt; it seems that the students who had a better grasp of the subject were able to convince their neighbour to change their mind resulting in consequential vote changes.
Towards the end of the course we asked students when they were given a problem to work out in a lecture, were they more likely to work out the answer based on a range of options (Figure 4).One way of interpreting these results is that the students enjoyed the interactive nature of the lectures.In this instance their responses have provided us with an endorsement of the use of handsets to engage students and hold their attention.
There are at least two very important things that emerge from the use of this new resource.
The first is that if the students are to answer the questions in ways that will be beneficial to them, that is, if they are not just to guess, then they have to reflect more on what they have learnt and how they are learning.A number of students reported that this was indeed the case, but also that when they could see how well they were doing they felt much more confident about what, up to then, they had only thought they knew.The conclusion here must be that this is a valuable way to carry out formative assessment.The second is that my teaching was being directed more by what the students needed, or at least, what they said they needed, rather than what I might have thought they needed; thus, I am less likely to try to second-guess or make unwarranted assumptions about their progress.The most striking instance of this was when I asked a question about how difficult students were finding the rules we had covered to date; to my astonishment many of the students found Disjunctive Syllogism (DS) much more difficult to comprehend than V-Elimination (VE).It seems such an unusual reversal in my own perception of the difficulty of the two rules that I can only imagine that disjunctive syllogism somehow appeared just deceptively simple and they were reluctant to believe that it could be so easy.

Progression to Honours
In our attempt to establish the reasons why students pursued their study of philosophy beyond their second year we discovered that students claim that philosophy is fun and that they enjoy the challenge that it offers.However, they also admit that Honours philosophy can be their only option when they fail to get in to their other Honours subject.So in some cases the element of choice is very nearly illusory.In fact, only 47 per cent of the students said that they had intended to do Honours philosophy before they came to university or in their first two years; for the rest -except for those who get better marks in it than their other subject, but who claim not to know how -Honours philosophy was little more than a fait accompli.Students also claimed that progression to Honours is often influenced by individual members of staff and, even, by how well the department is perceived to run.
Students give many reasons for not proceeding to Honours-level study in philosophy.They claim to find it more difficult than they had expected and, because of the difficulty, they find it dull and impractical.They say that they do not enjoy logic, they feel the emphasis on logic is too great, and they fear that there will be much more logic in Honours.Some of the students claimed to prefer subjects which had a clear knowledge base; subjects in which they could be given facts, learn them, and reproduce them -indicating a worrisome desire not to think.However, a comment that really does hit home is that students see philosophy as a 'filler' subject, and not one from which there is a clear route to a 'proper' career.
One of the most interesting discoveries we made was that in Level 1 43 per cent of the students reported that they were intending to read philosophy at Honours.In Level 2 and Honours the figure was 47 per cent who reported that they had known before entering university, or had decided during their first two years, that they wanted to read philosophy at Honours.The most surprising thing about these figures is how little fluctuation there is over the three years.It also indicates -as mentioned above -that for, at least, some proportion of the 53.4 per cent currently in Honours philosophy, philosophy is not their first choice.We need to put much more effort into developing and enhancing the important resources in our Level 1 and 2 classes, so that for many more students Honours philosophy will be their first choice.

Conclusion
We have judged the effectiveness of a resource in two ways, by whether students have used a resource and whether they have perceived it to be useful and, in the case of the Level 2 Logic class, by the results the students achieved in the examination.We must bear in mind that the first of these judgements is subjective, based on each individual student's assessment; however, we have seen that there is a remarkable consistency in what they say they benefit from.The second might be considered to be more seriously problematic because there may be students who use all of the 'right' resources but who do not necessarily employ good examination strategies.Unfortunately, there seems to be little way of telling when this might have happened in any individual case.
An investigation into teaching methods and resources of this sort is also beneficial for the lecturer.I regret that it has taken this study to drive home to me the great value of an accessible example in explaining something that might be obscure or difficult for the students.I was also surprised by having to return to explain a rule that I had assumed was straightforward, but thankfully I had a mechanism in place that made it possible for the students to tell me that something was going wrong.I have also had the opportunity to see whether the resources that I developed were useful for the students and, it has turned out, to my delight, that they are.
There are a couple of ways in which this study has been very good for the students.The first is that all of the students involved are now aware of the wealth of learning resources that are available to them.The second is that it has shown that handsets can be used both to engage students and to monitor class understanding.

i
figure /: Use of resources L2: Logic: Number of Learning Resources used by Students who gained 6-14 marks (lower 1/3 of students) and 18-20 marks (upper 1/3 of students)

Figure 2 :
Figure 2: Number of resources used

Figure 3 :
Figure 3: Marks vs attendance at last lecture