why computer science students find formal reasoning frightening

Journal of Compufer Assisted Learning (1994) 10,240-250

Why computer science student5 find formal reasoning frightening P. Fung', 1. O'Shea", D Goldson", S. Reeves" & R. Bornat"

Institute of Educational Technology, Open University ** Queen Mary & Westfield College, University of London**

Abstract This paper focuses on difficulties which first-year undergraduate computer science students experience in learning formal reasoning techniques. The main findings from an empirical investigation are reported. These indicated that, in the student population studied, many of those difficulties stemmed from a combination of the mathematical content of the courses and the students' unfamiliarity with formal notations and language. The paper concludes by discussing a selection of computer-based tools which have been selected to address these difficulties.

Keywords: Computer-based tools; Formal reasoning; Functional programming; Learning; Novice programmers.

Introduction

The current trend towards improving the reliability of computer software has led to interest in adopting a more rigorous approach to computer programming. Both in industry and universities, formal reasoning methods are increasingly seen as a means of reducing the likelihood of errors in computer software products by establishing the correctness of programs at an early stage in the design process. A growing number of universities are integrating this approach to programming into their undergraduate computer science courses. However, introducing formal methods into university computer sciences courses has given rise to the need for a closer look at relevant teaching and learning processes. In relation to this, a three year collaborative project (1991-94) is being undertaken which focuses on the difficulties first-year undergraduate students experience in learning formal methods and ways of helping to make their learning more effective. The overall aims of the project are to investigate the hypothesis that students' problems are mathematics related and to evaluate the effectiveness of computer-based tools in helping students overcome those problems.

Accepted: 31 January 1994

Correspondence: Dr P Fung, I.E.T. Open University, Milton Keynes, MW 6AA, UK

240 emaik p.fung62open.ac.uk

Why computer science students find formal reasoning frightening 241

As part of the project a series of empirical studies were undertaken. These were carried out within a university computer science department and focused on the cognitive difficulties which students experience in learning formal reasoning techniques. The results of each of these studies are reported in detail in (Fung & OShea, 1992; Fung & OShea,'1993). The principal finding from those results confirmed the view that although other factors come into play, many difficulties did appear to stem from the mathematical content of the courses. In this paper, we give a brief overview of the empirical investigations and discuss this finding. Following this discussion we conclude by describing the computer-based tools which have been selected for inclusion in the cumculum. Detailed descriptions of their selection and their technical development can be found in (Goldson & Reeves, 1993; Bornat & Sufrin, 1994). Within the context of the three year project this evaluation represents the second phase of the investigation. The outcomes of this second phase will become available after the tools have been in use over a complete academic year and will be reported in a subsequent paper.

Empirical studies

Initially a retrospective study was conducted in which approximately 150 second- and third-year students participated. Questionnaires were circulated asking students about their learning experiences and attitudes in relation to four courses taken in their first year, among them a first programming course in which formal reasoning techniques had formed a significant part of the work. As part of a follow-up process 12 students were interviewed individually. Feedback from both the questionnaires and interviews indicated that the majority of the students concerned had found the first programming course particularly difficult. For some it had been their first programming course and for all, their first introduction to formal reasoning methods. An interesting finding of this study was the diversity of expectations which students have of computer science, the difference in attitudes which students adopt towards programming and the sometimes almost totally opposing views they hold of its difficulty, relevance and interest. Interviewees talked of the courses, in which they had learnt formal reasoning methods, as having come as a 'shock' or 'surprise', of not having been what they expected, of having seemed 'irrelevant' at the time. In the same semester a questionnaire was sent to members of staff in the computer science department at Queen Mary & Westfield College who were or had been involved in first-year undergraduate teaching. Asked for their views on why students experienced difficulties in learning formal methods, staff indicated two areas of difficulty: lack of motivation and insufficient or underdeveloped mathematics skills.

In the following autumn the progress of the first-year computer science intake was monitored over that academic year. Intake questionnaires were distributed to approximately 115 students. It comprised three sections. The first elicited background academic details of the students while the second

242 P. Fung et al.

asked for personal views relating to computer science. In a third section students were asked to solve a selection of non-verbal reasoning problems. Data obtained from the intake questionnaires formed an information 'bank providing a snap shot of each student, the type of school they attended, their academic achievements, their attitudes to computing, their expectations of the course and their motivations for choosing a computer science degree course.

A first programming course (FPl), using the functional language MIRANDA as a vehicle to teach formal methods, was observed over the autumn semester and a record kept of student assessment and end of year marks. Sixteen students voluntarily kept a weekly record of their progress on the course. They were asked to attempt to identify the source of any difficulties they experienced and to indicate what steps they had taken to overcome them. Another eight students on the course were interviewed in- depth in order to help identify as precisely as possible at which points in the learning process, and why, students find the principles being taught difficult to understand and to apply. Initially, interviewees were given the opportunity to talk about their progress on the programming course and subsequently they were asked to log-on to a computer and step through a lab exercise which they had enjoyed doing and/or one that had caused them most problems. Below we give an overview of the main topics which arose from diary feedback and from interview data:

difficulties in coming to grips with formal language; need for more examples when new concepts were introduced; programming examples given were too maths based, not interesting or relevant enough; the course text book chosen (Bird & Wadler, 1988) presumed some prior knowledge of maths; programming groups were too large; not enough allowances were made for those without a computing background; the MIRANDA environment was unhelpful; too many new concepts were introduced too quickly; difficulties in understanding recursion; difficulties in understanding type-inferencing lab exercises too lengthy and insufficiently discussed in lectures.

On average, where help in understanding lecture material was needed, students initially resorted to reading over notes and the course book. For help with lab exercises they initially asked staff who were present during the allocated lab times. In both situations discussion with peers was seen as extremely useful, while only a smaller number of students contacted lecturers and tutors directly.

'Introduction to Logic' (ITL), an introductory logic for computer science courses, was observed during the second semester. In addition to lectures, students were given worksheets to be completed on a weekly basis for discussion at tutorials. Progress on the course was departmentally assessed


by three marked pieces of coursework and as with the programming course, marks from these formed a percentage of the final end-of-year marks for ITL. Twenty subjects were interviewed and data was obtained from an additional nine subjects via email. During face-to-face interviews students were asked to comment on their own view of their progress and invited to make any comments about the course which they felt relevant. They were given the opportunity to talk through one of the exercise sheets which they had completed, explaining what it was, if anything, that had caused them difficulty. Below is a brief summary of the main points:

difficulties with 'formal' notations;

difficulties in manipulating formulae;

In summarising the outcomes of the year's studies, problems encountered by students appeared to fall into three categories, the first of which we termed 'organisational'. This category included problems associated with transition from school to university, social problems of peer pressure, competing demands of coursework deadlines, family crises and financial difficulties. While they certainly have a bearing on student progress and must be considered in the practical organisation of courses, these problems tell us little about the process of learning formal reasoning techniques.

The second category of problems, which we termed 'attitudinal', first became evident as a result of the retrospective studies carried out at the beginning of the project. Such problems arise from illdefined or misplaced expectations of what a computer science degree course will offer, or from choosing to study for a computer science degree through motivations which overlook the reality of having to tackle a challenging and at times difficult subject. In the retrospective study, looking back at their first year's studies, students frequently commented on the di.fficulty they initially had in seeing the relevance of learning to reason about programs. They commented on the contrast the formal reasoning approach to programming had presented to their accepted way of thinking about programming. They explained that they had come to university wanting to become 'ace' programmers and had found that instead they had to do a lot of difficult thinking about programs. While this category of problems is one of attitude rather than one of learning, it can effectively block learning, since the student concerned is only concentrating on 'jumping hurdles' rather than trying to understand ' the underlying principles of the formal reasoning approach to programming.

It is the third category, that of 'skill-related' problems, which bears the most direct relationship to the process of learning formal reasoning techniques. In the light of the hypothesis that students' difficulties were

difficulties in manipulating formal language;

tedium of certain exercises, making them difficult to complete; accepting the possibility of different analyses, not necessarily only one correct answer; not understanding why certain rules are applied in a certain order (natural deduction); not knowing when to make assumptions (natural deduction).

244 P. Funget al.

mathematics related, problems in this category proved to be the most interesting. Many students perceived that a particular topic was difficult because they lacked a specific skill which they believed necessary to the successful understanding of the topic or to the completion of related work. A lack of a mathematical background and/or a lack of programming experience were the most frequently cited sources of difficulty. This qualitative data was supported by quantitative data obtained from departmental assessment. The marks which students score in their end-of- year computer science exams are combined with coursework marks to make up their end-of-year results. The average end-of-year results for students on the two courses were 55% for the programming course and 60% for the Logic course. Using the database we had built over the year to look in particular at two factors, the level of programming experience prior to university and whether or not students had studied maths at an advanced level (A-level), we considered the overall year results within a framework of eight groupings (see Table 1).

Table 1. End-of-year results, grouped according to background details

Programming experience With math Withaut maths n av.% av. % n av. % av.%

none 15 58 65 2 41 36 little 26 59 60 6 4 4 4 6 moderate 16 6 3 6 6 18 52 49 considerable 10 59 63 7 54 65

When we look at the average marks of each of these eight groups, we see that there appears to be an interesting correlation between these two 'starting points' (students' prior programming experience and maths background) and students' progress (Fig. 1).

f

60

% 50 3 40

$ 30 Q 20 c! : 10 i

0 I I I I I I I I I

none little middle range considerable

programming experience Fig. 1. FPl results showing maths-programming starting point relationship

There is no statistically significant correlation between prior level of programming and end-of-year marks. Neither is there a significant

W h y computer science students find formal reasoning frightening 245

correlation between the end-of-year marks, and the combination of prior level of programming and prior study of A-level mathematics.

However, the wide variation in numbers within these groups made it inappropriate to apply standard statistical tests to measure the correlation between the prior level of programming and end-of-year marks. For the same reason it is not possible to say that there is a statistically significant correlation between the end-of-year marks and the combination of prior level of programming and prior study of A-level mathematics.

Nevertheless and despite a lack of statistical evidence, the relevant average scores in the end-of-year exams (see Table 1) strongly suggest that the prior experience of studying A-level mathematics is an advantage. In the following section we discuss the findings outlined here. We consider ways in which students can be helped to overcome both skill-related and attitudinal difficulties, and outline tools that might help them to do so.

Discussion

In considering what it is that gives students who have studied A-level mathematics the advantage which our findings show, qualitative data give useful indicators. Students who had not studied A-level maths spoke at interview of feeling that they lacked a familiarity with formal notations, an ability to break down a problem into a series of smaller problems to be solved and an ability to manipulate formulae. The first of these skills was mentioned most often, perhaps because it is the first hurdle that students without maths or considerable programming experience have to cross, that of getting used to interpreting unfamiliar and often baffling notation. The second, being able to see the specification of a problem as a collection of component sub-problems, or being able to abstract the salient points of a problem and the relationships between them, is a pre-requisite to being able to reason successfully about a program. The third skill, regarding a program definition as an equation statement and being able to recognise and manipulate the components of that equation in order to prove the correctness of the program, is quintessential to the formal reasoning process.

It is evident that there is a correlation between the skills which these students feel that they lack and those which must be employed in formal reasoning. It is also true that the above skills are ones which students studying A-level mathematics would have had an opportunity to acquire, regardless of whether or not they actually did so.

In relation to attitudinal problems, data showed that incoming first-year students do not always have realistic expectations of what comprises a computer science course. In the first semester their introduction to formal reasoning about programs, through the functional language MIRANDA, can come as a considerable shock. Not only can they find it very difficult to understand and to use formal reasoning, but they can find it difficult to understand why they should be doing so. The initial retrospective study had shown that many students became aware of the relevance of formal reasoning techniques as their computer science studies progressed. At the

246 P. Fung et al.

outset however, they found it hard to appreciate why they were being 'put through, for reasons which they did not understand, what some saw only as 'mental gymnastics'. A proportion of the students are learning to reason about programs at a stage when for many their ability to write or understand programs is relatively limited. At a stage when they are only in a position to write comparatively small and transparent programs, it is not easy for them to appreciate the role of formal reasoning.

Nevertheless, given that the goal is to introduce this approach to programming at as early a stage as possible in the development of their computer science learning, it is important to consider some means of accelerating their awareness of the relevance of formal reasoning techniques. Interview data showed that for some students the belief that the current work is a prerequisite for being able to cope with future work is sufficient motivation in itself if this point is put forward clearly and consistently by those in a position to do so. This does not, though, hold true for all students and there is a need to look for more immediately motivating factors. For those students with little programming experience there is a need to facilitate their progress in developing programming skills. At the same time there is a need to make the abstract ideas of formal reasoning more concrete in relation to those skills.

The second hypothesis of the research project, to be tested in the second phase, is that providing students with computer-based textual and graphical tools will tackle many of the causes of difficulties which students experience in learning formal reasoning techniques by offering help both to students experiencing attitudinal problems and to students experiencing skill-related problems. The idea of using on-line tools in computer-based learning activities is not new and the benefits of doing so are widely accepted (Bma et al., 1987). Research in the area of formal reasoning has already begun, with developments such as Cooke & Stone's form-tool system (Cooke & Stone, 1992). Reasons for believing that a 'tool' approach could help in this domain are put forward in detail by Goldson (1993). Put briefly, those reasons stem from their belief that appropriate tools would give students the opportunity to investigate and to manipulate relatively complex concepts at a stage where students' own limited expertise would make this difficult in a traditional context. By representing concepts graphically and /or textually on screen in such a manner that students can explore the constraints and possibilities relatively easily, a basis may be laid for a deeper understanding of those concepts. In addition, by making it simple for students, through using computer-based tools, to display and control some of the more mechanistic and 'book-keeping' aspects of some formal reasoning processes, they can become familiar with those processes and more confident in carrying them out. In certain respects, the analogy could be that of a calculator. Users can 'play' with a calculator, exploring its operations, finding it fun to use and in doing so, strengthen their knowledge of those operations. Equally, a calculator can be used as an 'enabler' to allow users to perform complex calculations which would otherwise be timeconsuming and tedious, so reducing the likelihood of attitudinal problems.


Computer-based tools are to be introduced into both the first-year introductory logic course and initial functional programming course. A number of computer-based tools designed to help students learn methods of formal reasoning are available commercially and these were initially investigated and evaluated during the first phase of the project (Goldson, 1993). Of those evaluated, TARSKZ'S WORLD (Banvise & Etchemendy, 1990) was chosen for inclusion in the introductory logic curriculum. Research work in the field of program visualisation (Eisenstadt & Brayshaw, 1988) and graphical reasoning (Cox & Brna, 1993; Stenning & Oberlander, 1994) is currently exploring which types of visual and graphical information are useful in which contexts.

TARSKZ'S WORLD, designed to teach first-order logic, uses a mixture of text and graphical representations. Figure 2 below shows the three main components of this program, the world module, the sentence module and the keyboard. The world module displays a graphical representation of objects and the relationships which hold between them, such as Large, Larger, Between, LeftOf and so on. The sentences module displays the same relationships expressed in formal notation. Users determine the truth or falsity of the sentences in relation to the world.

I Carroll's World I Keuboard

I ............................................................................................................................. " ....................................... I..." .............. I Yes No 11 WFF? 0 0

T I'qe 7 c.2

2. SmallerIb, c) + [RightOf(b, c) A Between(a, b, b)]

3. Betweenk, b, a) + IBetween(b, a, c) V c = b l ........................................... ............. ............ ...................... ....................... ............

................................................................................................................................................. [Ytrif.!lI 1 x 1 I 14. Between(a, b, c) V Betweedb, a, c) V Betweenk, b, a)

.............................................................................................................................. ............................. ..... ................ I I " I..." "

5. TBetweedb, a, c) I Fig. 2. TARSKl'S WORLD: a world module, a sentence module and input keyboard

Sentences can be checked dynamically to verify that they are well-formed, correct and true. A 'game' can be played in which the user's decision on the 'truth' or 'falsity' of a statement is checked interactively by the system. A number of 'worlds' and their associated sentences are incorporated, but users

248 P. Fung et al.

are free to type in sentences of their own and construct worlds which prove or disprove them. Advantages of this software are that logical formulae are easy to construct and check. For the student, it is a relatively easy matter to try again when mistakes are made. The graphical representation allows the user not only to visualise the relationships between objects, but also the opportunity to manipulate these and test the 'truth of the relationship statements easily and directly. The system provides a way of demonstrating how an argument can be shown to be invalid by building a 'world which refutes it. The 'truth game facility helps the user to clarify the steps taken in proving a statement true in a way that is both fun and informative.

I i

...... . ...............................................

Bound BgS

= member' x ys, otherwise

Fig. 3. MIRACALC, a tool for reasoning about programs

No commercial tool was available for use in conjunction with the programming language MIRANDA, so MIRACALC was developed 'in-house' by researchers in the computer science department. Its design and development is fully documented in Bornat and Sufrin (1994). As shown in Fig. 3, the system offers a variety of options from a pull-down menu, all of which are directed at giving the user opportunity to experiment in reasoning about different aspects of a program. In practice, as well as addressing the problems of reasoning about program structures, MIRACALC also relates to the basic skills which students feel they lack in programming. Being able to experiment with typing, with reducing terms, and with exploring the structure of a program are activities that offer help in practical terms to students who are struggling to come to terms with a number of new programming concepts within a fairly short time span.

A third tool was also developed 'in-house'. Just Another Proof Editor (JAPE) is to be introduced into the introductory logic course (Bornat & Sufrin, 1994). This tool is expected to address a number of the difficulties which students reported as having experienced in learning to construct


proofs using natural deduction methods. These centre on the difficulty of manipulating formulae, of relating abstract concepts to actual situations, of not knowing when to take which steps in constructing proofs. By choosing steps from a menu selection, JAPE allows the student to construct, on screen, a proof for the truth or falsity of a given statement (see Fig. 4). Increased familiarity with the deduction rules being used should be gained through the ease with which proofs can be constructed and manipulated. In strengthening the connection between the abstract learning of deduction rules and the process of putting them into practice, the use of this tool should be instrumental in developing students' experience and confidence in deciding 'which step' to take next in proof construction.

1 6 File Edit Lectures Extras Conjectures I

Fig. 4. Fitch box representation of a proof inJAPE

Conclusion

At the end of the first phase of this research, the principal outcomes of a series of empirical studies confirm the hypothesis that many of the problems which students encounter in learning formal reasoning techniques are related to their mathematical background. In addition, they indicate attitudinal problems which students may experience in learning formal reasoning methods. In this paper we have discussed these outcomes and have given an overview of the tools which it is hypothesised will address some of the problems discussed. The body of the second phase of this research will be the evaluation of the use of these on-line tools. Studies will consider to what extent they help students develop basic computer skills, encourage them to explore and manipulate program structures and become familiar with formal language and notations. In conclusion, it must be pointed out that it is not expected that the introduction of these computer-

250 P. Fung et al.

based tools will solve all, or even most, of the problems related to teaching and learning formal reasoning skills or, indeed, that it is the only option open to those concerned with teaching in this domain.

Acknowledgements

The research which formed the basis of the discussion in this paper (The Calculator Project) was funded by the Joint ESRC, MRC, SERC Committee, Cognitive Science/HCI initiative (Ref. No. 9019558).

References

Barwise, J. & Etchemendy, J. (1990) The Language of First Order Logic. Center for the

Bird, R. & Wadler, P. (1988) Introduction to Functional Programming. Prentice Hall,

Bornat, R. & Sufnn, B. (1994) JAPE:Just Another Proof Editor. (in preparation) Bma, P., Bundy, A , Pain, H. & Lynch, L. (1987) Programming Tools for PROLOG

Environments. In Advances in Artificial Intelligence - Proceedings of AISB Conference (eds. J. Hallam & C. Mellish) pp. 251-264. John Wiley & Sons, Chichester.

Cooke, J. & Stone, R. (1992) The form-tool system. In Proceedings of the Dewlopments in the Teaching of Computer Science Conference (eds. D. Bateman & T. Hopkins) pp. 184-192. Computing Laboratory, University of Kent at Canterbury, Canterbury, UK.

Cox, R. & Brna, P. (1993) Reasoning with External Representations. In Proceedings of the World Conference on ArtiJlcial Intelligence in Education, Edinburgh. pp. 185-192. Association for the Advancement of Computing in Education, Charlottesville, USA.

Eisenstadt, M. & Brayshaw, M. (1988) The Transparent Prolog Machine (TPM) Journal of Logic Programming, 5,4,277-342.

Fung, P. & OShea, T. (1992) Computer m'ence students' perceptions of learning formal reasoning methods. CITE Report no.160. Open University, Milton Keynes.

Fung, P. & OShea, T. (1993) Computer Science Students' Perceptions of Learning Formal Reasoning Methods. lntemational Journal of Mathematical Education in Science and Technology, 24,5,749-759.

Goldson, D. (1993) Using Programs to Teach Logic and Programming to Computer Scientists. Technical Report 632. Computer Science Dept., Queen Mary & Westfield College, University of London.

Goldson, D. & Reeves, S. (1993) A Symbolic Calculator for Functional Programs. Technical Report 629. Computer Science Dept., Queen Mary & Westfield College, University of London.

Stenning, K. & Oberlander, J. (1994) A cognitive theory of graphical and linguistic reasoning: logic and implementation. (To appear in Cognitive Science.)

Study of Language and Information, Stanford.

London.

why computer science students find formal reasoning frightening

Documents