Generating Sensible Feedback for Gap-Filling Exercises with Verbal Units in the Chemnitz Internet Grammar

Christian EckhardtENGLISH LANGUAGE AND LINGUISTICS,

CHEMNITZ UNIVERSITY OF TECHNOLOGY, D-09107 CHEMNITZ1

I. INTRODUCTION

The exercise component of the Chemnitz Internet Grammar (CING) project, as described by Wilson and Schmied (this volume), will be discussed in more deatail here. The CALL software market is dominated by old-style drill and practice software. In most cases textbooks, audio and video material has simply been transfered to CD-ROMs. "In reviewing these products one realises that, in terms of technology, enourmous progress has been made; but from an educational point of view, no progress has been made at all." (Wolff 1998: 69)

One of the central ideas of the CING is to promote learner autonomy and to facilitate inductive learning. Inductive and autonomous learning both require some guidance of the learner. Learners need an overview of what they have done so far and where their problems lie, only then can they take qualified decisions on what to do next. Guidance can either be immediate feedback given in the exercise component or advice and

1 I want to thank my colleagues Josef Schmied, Angela Hahn and Naomi Hallan for their support and advice. The Chemnitz Internet Grammar project is located at: http://www.tu-chemnitz.de/InternetGrammar/

information given at any other time. The system's knowledge about the learner should comprise more than facts such as "s/he has made x% mistakes in the exercise on present simple and continuous". Ideally it should categorise incorrect answers according the presumed cause of the deviation and give feedback accordingly.

Two approaches to the improvement of feedback will be shown: a) error anticipation for every single item as practiced with authoring systems and b) error analysis performed by knowledge-based tutoring systems developed in Artificial Intelligence (AI) research.2 Soria (1997: 44) has termed a compromise he applied to the inflection of Spanish verbs 'Expert CALL':

In between these two approaches there may be a role for applications based on linguistic knowledge which is not as complete or as formalised as the afore-mentioned theories [of AI] propose, but wich is reliable and suffecient for teaching and learning.

In the last section a similar compormise will be proposed for verbal units3 in gap-filling exercises.

II. THE PROBLEM

An easy way to produce a large number of exercises for CALL is to develop a kind of authoring system. Such systems allow teachers to create learning units on the computer without knowledge of any programming language. Anticipated feedback and hints for expected answers can be provided. Brücher (1994:126) thinks that authoring systems are very useful because they are easy to use, inexpensive and can be adapted to the needs of the class or course. The following example from Testmaster 1.0 (cf. Brücher 1994:120) demonstrates how authoring systems work:2 For an overview of Intelligent Tutoring Systems developed for language learning see Swartz and Yazdani (1992) and Holland et al. (1995).3 The term "verbal unit" has been choosen to point out that unlike "verb phrase" is does not comprise objects and adverbials. "Verbal unit" here denotes only the inflected form of the verb (i.e. auxiliaries, modals and the main verb).

(1) 01: In how many ways can you express the names of the former two Superpowers?

02: One has stars and stripes on its flag, the other had a hammer and sickle.

03: (The) [United States of America/United States/States/USA]04: (The) [USSR/Soviet Union/Union of Socialist Soviet Republics]05: [America/Russia]#There are more elaborate names than this.06: (The) CCCP#OK, but you are using English letters as Russian

ones.

Line 01 contains the question, line 02 a hint. In lines 03 to 05 the correct answers are given: the article is optional and is therefore put in brackets. The square brackets indicate that there is a list of solutions. The different solutions are separated by slashes. In lines 05 and 06 the answers are provided with comments. The comments are separated by the "#" symbol. If the learner answers the question (01) with "America" the program will tell him or her "There are more elaborate answers than this." and accept the answer as correct.

Testmaster is a relatively flexible program because it is able to accept several answers as correct and comments on other anticipated answers. An advantage of this type of program is that teachers can adapt the exercises to the needs and the capabilities of their learners. On the other hand teachers have to spend a lot of time devising the exercises. The task becomes even more labour-intensive when a teacher wants to maximise the use of the program by anticipating and commenting as many answers as possible. Handke (1988:27) demonstrates the problem of the anticipation of answers with a relatively simple gap-fill exercise:

(2) John (1) _______ to London yesterday. He (2) ______ there a long time ago.

Now he (3) _______ in America at the University of Berkeley. The (4) _______ of his studies is to explore expert systems.

Possible correct solutions are:(3) gap 1: went, drove, flew, got, walked etc.

gap 2: had (not) been, had stayed, had lived etc.gap 3: is living, is studying, is staying, is working etc.gap 4: (primary, main, basic) purpose, aim, idea etc.

Authoring systems become even more problematic when answers that consist of more than just keywords are required, because the number of possible answers, right or wrong, increases enormously, so that anticipation becomes impossible. Higgins (1995: 44-49) lists 555 correct paraphrases of the sentence "Jim hated school because the teachers were unkind." to illustrate this problem. Taking possible learner errors into account, the number of possible sentences increases even more.4

Even when the verb to be used in a gap-filling exercise is given, a large number of different incorrect answers are possible, e.g.:

Students may be unaware of irregular verb forms They may get person and number wrong. They may apply the wrong tense. They may have word order problems. (*He walked has.) They may have problems with simple and continuous aspect. They may type nonsense.

Not all answers that do not exactly match the anticipated correct answer are errors in the sense of being based on a misconception of the grammatical rules concerned. They may be referred to as mistakes. The most obvious example of a mistake is a mis-typed word. Ignorance of irregular verbs may also lead to mistakes that are outside the focus of the current exercise. The same applies to spelling mistakes (especially e-deletion and y-to-i change).

Most programs available today get round this problem by accepting only a single answer as the correct solution and ruling out all other solutions. This appears to be an acceptable solution, but the programs usually also restrict their feedback to 'correct' or 'wrong' with no further help given. The "Grammatiktrainer Englisch" by Digital Publishing (Munich) is an exception. If an error (or deviation) has occured the program comments on every single word it detects as being wrong, as shown in the table below:4 Though certainly aware of this fact, Borraccino (1998) has implemented a tutoring system that allows students to answer in complete sentences. The system then tries to match the answer to solutions stored in a database. If the answer given by the student cannot be found in the data-base it will be transfered to a human tutor who then checks the answer "manually", passes it to the student and adds it to the data base.

Error Feedbacka) Have you *seed Roger? The Participle is wrong, it is an

irregular verb.b) I visited the Charlisles and *have gone shopping a bit.

This action belongs to the past. Change the tense.

c) She *called me four times since we met on the weekend.

This is not the present perfect.

Table 1: Feedback provided by the "Grammatiktrainer English"

Unfortunately the program for example cannot deal with the following mistake properly:

(4) I visited the Charlisles and *had gone shopping a bit.

Here the progam reports that it could not deal with the word "had" and comments on "gone" with "This action belongs to the past, change the tense", which is the same comment as with b) in the table above. Hence the program interprets the answer as present perfect because it contains a participle, since the second word in the input is the same as in the present perfect. This shows that the program can only deal with anticipated answers. In other sections of the "Grammatiktrainer" other kinds of mistakes have been anticipated. In the "past and past perfect" section the program recognises "had gone" as a past perfect form. Though the program's feedback is not always appropriate and does not cover some obvious cases, it is the only program available on the German market that gives detailed feedback at all.

Another disadvantage is that the number of exercises provided for each topic is very limited although there are more than 600 exercises on the CD-ROM. This may also be due to the fact that the exercises are stored in fixed frames, i.e. they are not generated from a database.

III. ARTIFICIAL INTELLIGENCE (AI) APPROACHES

Intelligent Computer-Assisted Instruction (ICAI) uses methods and tools from artificial intelligence (AI) for the development of learning systems; these are also called Intelligent Tutoring Systems (ITS). They are intended to approach the performance of an idealised human tutor. Leutner (1992: 59) lists four criteria that an ITS should fulfil:

1.An opportunity for the learner to exchange roles. The program should not only question the leaner but also allow the learner to put questions. It should be able to answer questions whose answer has not been explicitely stored.

2.Domain knowledge and teaching knowledge should be separated. This would allow the program to react to learners' needs flexibly. The program should follow a general teaching strategy and be able to fill it out with the required contents in contrast to "non-intelligent" programs where the whole presentation has been prepared and fixed in advance.

3.It should establish the level of the learner‘s knowledge and categorise their working habits.

4.Based on the data gained in 3.) the program should be able to adapt to the knowledge state of the learner and present the contents in the way most suitable for the learner.

These criteria are met by only few ITS. To fulfil these demands on the system a minimum of data necessary can be deduced: Self (1990) lists the different kinds of data he considers necessary for successful adaption of the program to learners' needs:

1.Expert knowledge of the domain to be taught2.Knowledge of teaching strategies3.A model of the learner, reflecting his or her learning style,

domain knowledge and behaviour.

The structure of an ITS can best be explained as consisting of four components, as shown in figure 1. The communication module is responsible for the presentation of the domain knowledge and the exercises. It also interprets the learner's input and passes it on to the student module. Most ITSs are text-based, i.e. the learner’s input has to be typed. There are a few systems that allow spoken language as input and a few that use spoken language as output (e.g. Spaai 1995). In order to allow input that is more complex

than just keywords or predefined phrases, the communication module has to parse sentences and make their meaning accessible to the computer (cf. O'Shea / Self 1983: 92). Therefore an ITS needs a syntactical parser and a semantic component. The parser should also be able to tolerate minor mistakes. Creating output is much easier because the computer can make use of predefined messages or message components.

The student module compares the learner's answer to the solution arrived at by the expert system (expert module) and draws conclusions about the knowledge and competence of the learner. It stores its conclusions in the student model. With an analysis of the learners knowledge at hand the teaching module decides on what to do next, whether to provide new exercises of the same type or to proceed to the next topic.

The expert module contains the knowledge about the domain to be taught. It should be able to solve all questions that the system can ask the user or answer all questions the user asks the system. In AI research a distinction is made between procedural and

Figure 1: The components of an ITS.

declarative knowledge. Transfered to a language teaching system, declarative kowledge consists of facts such as words and their syntactical, morphological, phonological and semantic properties while procedural knowledge is the knowledge of how to combine these words into more complex words or sentences and to interpret their meaning.

Rule systems can be designed to attempt to model human thought processes or just to try to arrive at the same output. Since there is no satifactory model of human thought, algorithms that work independent of any model of human thought processes are usually more effecient. The best example of such algorithms are chess programs. It had long been thought that chess was a task that only intelligent humans could perform well. Scientists had assumed that a computer that could play chess well could also perform most other human tasks. Thus the construction of an "intelligent" chess computer had been a primary aim in AI research. Attempts, such as Winograd's SHRDLU, to model easier tasks humans perform every day, like moving about objects or finding one's way in unknown territory have shown that this assumption is not valid. This is basically so because the algorithms underlying most "intelligent" systems do not model human thought but try to reach human performance in a way more compatible with computers.

In contrast to these so-called "black-box" models, others, called "glass-box" models, try to simulate human thought processes. This is not only problematic because human thought can hardly be observed and there is no sufficient theory but also because humans have types of data available to them, that computers can hardly access: a computer cannot look into a learner’s face. Clancey (1983), one of the pioneers of intelligent tutoring, came to the conclusion that though glass-box models are usually less efficient, they are more suitable for tutoring, because the learner can be guided along the same path the computer has used to come to its solution. Burton and Brown (1983: 82) have suggested using a black-box model to come to the correct solution an a glass-box model for the tutoring, since "backward reasoning" is much easier than problem solving from scratch.

An Example

Most ITS are prototypes designed to teach the programming language LISP or the basics of algebra, only few systems are concerned with the teaching of natural languages. One of them is the "English Tutor" (ET), which is intended to help Italian students with English tenses. It will be described here in order to give the reader an impression of how such a system works.

ET asks the students to do gap-filling exercises and then the expert system solves the tasks itself. Then ET compares its solution to that of the learner. The expert system, which will not be dealt with in more detail here, is based on a model of temporal logic by Matthiessen (1984) that enables it to determine the tense to be used.

If it detects an error it tries to figure out the cause of the error: (5) ET: Exercise-1: "He always (carry) an umbrella when he was in England."

Student: was carryingThree causes for this error are possible:1. The student does not know the rules for the use of simple past

and past continuous,2. s/he does not know simple past and therefore uses past

continuous or3. the answer was a mistake, i.e. an unsystematic error.To find the cause the system tries to verify or falsify alternative theses. Here it tries to find out if the learner knows the simple past:

(6) ET: Exercise-2 : "I (share) a room with him when we were students." Student: shared ET: Exericise-3 : "I (make) a cake when the light went out." Student: made ET: Exercise-4: "Last winter on holiday her mother often (tell) her that she (spend) too much money." Student: was telling / was spending

The answer to exericse 3 was wrong. It seems that the student has problems with the simple past. With exercise 4 the program tests its hypothesis once more. Since the error has occured again, a mistake (hypothesis 3) can be excluded. Having verifyed one of its hypotheseses proved, the program gives the following feedback (Fum et al. 1988: 515):

(7) ET: Answers to exercises 1, 3 and 4 were wrong. Your knowledge about the use of the simple past and the past continuous seems to be the following:

You utilized the simple present according to the rule "When a clause describes a single action or a state that takes place in the past, use the simple past."

This rule does not take into account the further condition that: "When a clause describes a single action or a state that takes place simultaneously to the reference time of the clause, then the tense to be used is past continuous."

The solution to exercise-1 is: Tense: simple pastConjugated verb: carried

because:the action took place in the pastthe action is habitual...

The program takes the result of this sequence of tasks as a basis for the choice of subsequent tasks. It will make the student practise the distinction between simple past and past continuous and try to correct the student's misconception about the use of these two tenses.

IV. THE SOLUTION APPLIED TO TENSE AND ASPECT IN THE CING

The tutoring component of the CING will make use of some technologies that have been developed in AI research. While ITSs are based on the idea of using an expert system to find the solution to the task. ITSs are expected to be able to guide the learner to the correct solution in a way that helps him or her to

grasp the problem. With this approach most attention has to be paid to the expert system’s capacity to solve problems and to the analogy of the expesystem's algorithm to human reasoning.

With the CING a completely different attempt that was inspired by Burton and Brown’s idea of backward-reasoning: the CING will do without an expert system to solve the tasks, instead the solution will be provided with the task:

(8) (a) I (have been living # to live) here since 1991.(b) He (has never been # to be & never) to England.(c) Columbus (sought # to seek) for a passage to India.

From these entries all information required for sensible feedback will be derived. Compared to the information necessary for "Testmaster" shown in (1), a minimum of information is provided with each task. It will be restricted to the correct solution and the text to be presented to the learner (the text after the "#" symbol).

In a first step the learner's input will be compared to the correct solution. If both are not identical, the program will derive the required tense from the correct answer and the tense used by the learner will be deduced from his or her input. This can be done with a lexicon that contains no more than modals and auxiliaries in all their inflectional variants. Additionally "words"5 will be checked for three morphological affixes ("-s", "-ed" and "-ing"). Words that contain these affixes are assumed to be full verbs. This assumption is valid because the program does not deal with full sentences but with verb phrases in a narrow sense, i.e. that part of a verb phrase that is made up of modals, auxiliaries, adverbs and verbs. Nouns, like thing, king etc. that would cause problems, are excluded by this definition.

Here the advantage of the proposed method becomes obvious: The analysis of the learners' answers and the correct solution can be restricted to a minimum, no full-scale understanding of sentences is necessary and the lexicon can be reduced to functional words. The only problem that remains with this solution are irregular verbs. Either irregular forms have to be stored with each and every task or they have to be deduced 5 Words here are defined as items in a string that are separated by blanks, but not in any linguistic sense.

otherwise. It appears to be unacceptable to extend the lexicon with a list of all irregular verbs, because due to low band-width on the internet the lexicon has to be kept as small as possible. From a comparison of the infinitive given with the task and the provided solution, irregular verbs can be detected. From the information contained in task (8c) it is possible to derive that the inflection of the verb is irregular:

1.sought search2.sought search + (e)s3.sought search + (e)d4.sought search + ing

Since the answer has to be correct, the rightmost word in the solution has to be the verb, hence it has to be an irregular form.

An example

An example will be given for the analysis of errors in the CING. Provided the student gives the following answer to task (8a)

(9) *has been livedthe program will first compare the answer to the solution. As they are not similar here, it has to find out what the features of the target phrase are. For that purpose it uses the lexicon, containing auxiliaries and modals as mentioned above, is used. Here is a sample of its entries:

belongs to lexeme

number & person

morphological verb form

be - - Infinitive

am be 1st Sg. Present

is be 3rd Sg. Present

are be 2nd Sg.1st, 2nd or 3rd Pl.

Present

was be 1st or 3rd

Sg. 2nd or 3rd

Pl.

Past

were be 2nd Sg.3rd Pl.

Past

been be - Particip

leTable 2: The lexeme "be" in the lexicon

Table 2 contains the possible forms of the lexeme be and their implications for the characteristics of the verb phrase. Combined with an analysis of the morphological variation of the main verb, i.e. by checking whether it is an unmarked, a past or a continuous form, tense, aspect, number and person can be determined. An analysis of the subject may contribute to the precise identification the number and person of the given phrase. If no pronoun is found 3rd person is required. The result of the analysis will be the following:

Words have

been living

Lexeme

have be liv

Category

aux aux verb

Person ? ? ?Number

? ? ?

Tense present

participle

-

Aspect ? ? continuous

Table 3: Analysis of the correct solution

The solution is the auxiliary have in the present tense, followed by the participle of be and a continuous form of a verb with the root liv. In other words: the solution of task (9a) is a present perfect continuous of the verb live.6 This way tenses can easily be defined in the same terms grammar text-books do it.

The learner's answer (9) can be analysed in the same way, as shown in the table below. Both, the auxiliary have and the participle of be are correct. Only the verb is inflected wrongly.

6 This example shows that spelling rules (such as e-deletion, e-insertion, y-to-ie change) have to be taken into account.

This error can stem from a mixture of the forms of the present perfect simple and the present perfect continuous:

Words have

been lived

Lexeme

have be liv?

Category

aux aux verb

Person ? ? ?Number

? ? ?

Tense present

participle

past orparticiple

Aspect - - -

Table 4: Analysis of the incorrect solution.

By using this kind of analysis feedback can be provided on the basis of generalised definitions that apply for all tasks:

Condition Feedbackhave(present)

+ be(participle)

+ verb(past or participle)

You may have confused the form of the present perfect simple and present perfect continuous. Either the verb has to be in the continuous or the participle must not be here.

Table 5: A generalised definition of feedback.

V. CONCLUSION

The approach described here is allows on the analysis of tense forms in gap-filling exercises, i.e. natural language processing in a very narrow context. The analysis is limited to the identification of auxiliaries and the few morphological variants of English verbs, which is only possible due to the restrictions inherent to gap-

filling exercises. Here much attention is paid to the formal correctness of tense forms though this is not likely to be a major problem with the users of the CING as it is aimed at advanced students of English.

It might be argued that the same goal could much easier be achieved by the provision of dual-choice exercises, where the tense forms would be clear by definition and appropriate feedback could be given according to the selection of pre-defined answers. Apart from the assumption that learners may remember their answers better when they have typed them in, there is another advantage to this approach for the CING: As the aim of the CING is to observe learner behaviour in an interactive learning environment (i.e. on the internet), the learners' success will have to be measured in order to find out about the successfulness of learner strategies. With exercises performed in traditional lessons it often seems to be the case that learners perform well in exercises as long as the problem involved is in the focus of the exercise, while problems remains difficult in other contexts. In an interactive learning environment it will be possible to mix items from previous exercises with those of the current exercise in order to check whether students are also successful with items outside the focus of the current exercise. This is an obvious advantage of gap-filling over multiple- (or dual-) choice exercises.

References

Brücher, K.H. (1994). Zur Leistungsfähigkeit und Verwendbarkeit von Autorenprogrammen in CALL. In: J.Fechner (ed.), Neue Wege im computergestützten Fremdsprachunterricht. Langenscheidt, Berlin, pp. 135-163.

Burton, R.R. and J.S. Brown. (1982). Investigation of Computer Coaching for Informal Learning Activities. In: D. Sleeman and J.S. Brown (eds), Intelligent Tutoring Systems. Academic Press, London, pp. 157-183.

Fum, D, P. Giangrandi and C. Tasso. (1988). The ET Project: Artificial Intelligence in Second Language Teaching. In: Proceedings of the European Conference on Computers in Education, Lousane, pp. 511-516.

Handke, J. (1988). Computer-Assisted Language Learning und künstliche Intelligenz. Die neueren Sprachen 88: 134-147.

Higgins, J. (1995). Computers and English Language Learning. Intellect, Oxford.

Holland, V.M., J.D. Kaplan and M.R. Sams. (1995). Intelligent Language Tutors Theory Shaping Technology. Lawrence Erlbaum, Mahwah, NJ.

Leutner, D. (1992). Adaptive Lernsysteme. PVU, Weinheim.

O'Shea, T. and J. Self. (1983). Learning and Teaching with Computers. Artificial Intelligence in Education. Harvester Press, Brighton.

Self, J. (1990). Theoretical Foundations for Intelligent Tutoring Systems. Journal of Artificial Intelligence in Education 1(4): 3-14

Soria, J. (1997). Expert CALL: Data-Based vs. Knowledge-Based Interaction and Feedback. ReCALL 9 (2): 43-50.

Spaai, G.W.G. (1995). Feedback in Computer-Assisted Instruction: Complexity and Corrective Effeciency. In: R.J. Beun, M. Baker and M. Reiner (eds), Dialogue and Instruction. Modelling Interaction in Intelligent Tutoring Systems. Springer, Berlin, pp. 167-178.

Swartz, M.L. and M. Yazdani. (eds). (1992). Intelligent tutoring systems for foreign language learning: The bridge to international communication. Springer, New York.

Wolff, D. (1998). Computers and New Technologies in Language Learning. The European English Messenger 7(2): 69-71.