Green - Watching for Washback

Download Green - Watching for Washback

Post on 16-Oct-2014




5 download

Embed Size (px)


<p>LANGUAGE ASSESSMENT QUARTERLY, 3(4), 333368 Copyright 2006, Lawrence Erlbaum Associates, Inc.</p> <p>Watching for Washback: Observing the Influence of the International English Language Testing System Academic Writing Test in the ClassroomAnthony GreenUniversity of Cambridge ESOL Examinations</p> <p>Previous studies of washback (the influence of a test on teaching and learning) have provided insights into the complexity of educational systems and test use, especially in relation to the role of the teacher, but have given insufficient attention to the relationship between observed practices and test design features. In this article a washback model is proposed that incorporates both test design and participant characteristics. The model is used to predict behaviour on preparation courses directed toward the Academic Writing component of the International English Language Testing System (IELTS) test. 197 learners and 20 teachers were observed over 51 classroom hours. These encompassed 22 IELTS preparation classes and, for comparison, 13 classes of English for academic purposes (EAP). Evidence was found for substantial areas of common practice between IELTS and other forms of EAP but also for some narrowing of focus in IELTS preparation classes that could be traced to test design features.</p> <p>Before offering a place to an international student, most universities in English-speaking countries will require evidence of the students language ability. As increasing numbers of students choose to travel to access global educational opportunities, there has been rapid growth in the use of language tests for this purpose. In the United Kingdom the most widely recognised test of English for academic purposes (EAP) is the International English Language Testing System (IELTS). Between 1995 and 2005 the number of candidates rose from under 50,000 to over half a million per year (International English Language TestingCorrespondence should be addressed to Anthony Green, Validation Unit, University of Cambridge ESOL, 1 Hills Road, Cambridge, CB1 2EU, UK. E-mail:</p> <p>334</p> <p>GREEN</p> <p>System, 2005). The rapid expansion of the test has brought with it increased demand for test preparation books and courses. IELTS is used as a means of determining whether candidates should be accepted into English-medium courses and whether they will require further language support. However, concern has been expressed that preparation for tests like IELTS, because of the limitations on what can realistically and equitably be tested in a few hours, may not develop the full range of skills required for successful university study, particularly in the area of academic writing (Deakin, 1997; Read &amp; Hirsh, 2005). J. Turner (2004), for example, argues that what the IELTS test or the TOEFL test delivers underspecifies the complexity of language issues in the academic context (p. 98). Her concern is that education in academic literacy is being supplanted by training in test taking. But what influence does the IELTS writing test really have on teaching, and how different are writing classes in IELTS preparation courses from other forms of EAP? To investigate these questions, this article compares the practices observed in writing classes of two types: IELTS preparation classes directed at success on the test and, as a suitable point of comparison, presessional EAP writing classes provided by universities to prepare learners for academic study.</p> <p>LITERATURE REVIEW Washback studies, investigating the effects of tests on the teaching and learning directed toward them, have often involved direct observation of the behaviour of teachers and learners in the classroom. The inclusion of an observational element in such studies has been recommended as a means of contextualising, corroborating, or correcting data from surveys and interviews (Alderson &amp; Wall, 1993; C. Turner, 2001; Wall, 1996; Watanabe, 2004). Table 1 summarises the methods and findings of recent case study investigations of washback in language education that have included an observational element. These studies covered a wide range of educational contexts, with observation either focussing on a small number of participants observed intensively over a sustained period (Alderson &amp; Hamp-Lyons, 1996; Read &amp; Hayes, 2003) or on a lighter sampling of classes to allow for observation of larger numbers of teachers and a broader perspective (Hawkey, 2006; Wall, 2005). With the exception of Burrows (1998, 2004) and Hawkey (2006), all included comparisons between different types of class. Wall (1996, 2005) and Cheng (2005) focused on changes over time as a new test is introduced. This approach also informed work relating to the recent update of the Test of English as a Foreign Language (TOEFL; Hamp-Lyons, 2005; Wall &amp; Hork, 2004). Alderson and Hamp-Lyons (1996) and Watanabe (1996, 2004) compared the practices of teach-</p> <p>TABLE 1 Studies of Washback in Language Education That Include Observation and Interview Data Observations Purpose-designed instrument, 2 teachers 16 classes: 8 general English, 8 TOEFL preparation over 1 week 3 focus group interviews with 312 students; 9 teachers in group and individual sessions N and Frequency Interviews Key Findings</p> <p>Study</p> <p>Exam</p> <p>Institutions</p> <p>Alderson and Hamp-Lyons (1996)</p> <p>Test of English as a Foreign Language (TOEFL)</p> <p>Specialised language institute in United States</p> <p>Brown (1998)</p> <p>International English Language Testing System (IELTS)</p> <p>University-based language institute</p> <p>Records of instructional focus and materials, 2 teachers</p> <p>All classes designated for writing over 10 weeks; 70 hours in IELTS and 30 hours in non-IELTS EAP course</p> <p>Not reported</p> <p>TOEFL washback may be generated more by teachers, materials, writers, and administrators than by test. Amount and type of washback vary according to test status, relation of test to nontest practice, degree to which teachers and material writers think about appropriate methods and their willingness to innovate. IELTS preparation instruction closely focussed on the test with regular timed practice of IELTS tasks. IELTS preparation was more successful than EAP course in improving IELTS scores from entry to exit, but small samples (9 IELTS, 5 EAP) limit interpretation.</p> <p>335</p> <p>(continued)</p> <p>TABLE 1 (Continued) Institutions COLT, 4 teachers Two 4-hour lessons per teacher Observations N and Frequency Interviews Key Findings</p> <p>336Adaptation of COLT, 3 teachers 30 teacher interviews including those observed (conducted prior to the observation phase) Follow-up 31 70-min classes interviews with c. 6 months observed before exam over teachers 2 years Cambridge IIS instrument, 10 teachers 10 IELTS preparation classes Teachers vary in responses to innovations in assessment in line with individual differences and experience. With Cheng (2003) and Wall (2005), links washback to theories of change. Following innovation, teachers engaged in exam-like activities e.g., role-plays, and used (conservative) exam prep materials in class, but beliefs and attitudes to learning did not change. Differences in how teachers cope with change. Extensive support required in implementation of innovations. Teacher preference for 120 students, 21 task-based, often teachers, and 15 interrelated macroskills receiving activities, involving institution microskills relevant to administrators IELTS. Focus on IELTS in focus groups but willingness to use a range of materials and methods.</p> <p>Study</p> <p>Exam</p> <p>Burrows (1998; 2004)</p> <p>Certificates in Spoken and Written English (CSWE)</p> <p>Adult English Migrant English Programme (AMEP) in Australia</p> <p>Cheng (2005)</p> <p>Hong Kong Certificate Examinations in English (HKCEE)</p> <p>3 HK secondary schools (main study)</p> <p>Hawkey (2006)</p> <p>IELTS</p> <p>10 language schools in UK, Japan, Cambodia</p> <p>Read and Hayes (2003); Hayes and Read (2004)</p> <p>IELTS</p> <p>2 language schools in New Zealand</p> <p>COLT and Cambridge IIS instrument, 2 teachers</p> <p>Wall (2005)</p> <p>Sri Lankan O level</p> <p>c. 50 secondary schools in 11 areas of Sri Lanka</p> <p>Purpose-designed observation checklist</p> <p>22 hours of 32-hour 23 teacher interviews; course and 28 weekly hours of 320interviews with hour (8-month) the teachers course observed 5 lessons observed 64 teachers in focus groups; over 2 years follow-up (3964 different interviews with classes observed each class per round) at observation varying periods ahead of exam</p> <p>Watanabe (1996; 2004)</p> <p>Various Japanese university entrance examinations</p> <p>3 high schools in Japan</p> <p>Adaptation of COLT, 5 teachers</p> <p>964 min of exam preparation and 833 min of regular classes over 6 months</p> <p>Teacher follow-up interviews with each observation</p> <p>Differential teacher practices at least partially linked to institutional context. Greater pressure to teach to the test in private language schools. Teachers used textbook content but failed to follow suggested methods, e.g., reading for gist. Proportion of classes was dedicated to exam preparation. Neglect of speaking skills traceable to exam content. Teachers vary in their approaches to exam preparationinformed by attitudes towards the exam. School culture an important factor. Material designed for exam preparation may sometimes be used for other purposes.</p> <p>337</p> <p>338</p> <p>GREEN</p> <p>ers in test preparation and non-test-preparation classes, while Read and Hayes (2003; Hayes &amp; Read, 2004) compared two approaches to test preparation. Three of the observational washback studies listed in Table 1 investigated IELTS preparation classrooms. Brown (1998) compared practices in two courses provided by the same institution: IELTS preparation and non-IELTS EAP. Read and Hayes (2003) compared two IELTS preparation courses at different institutions: one an intensive preparation course, the other combining IELTS preparation with other forms of EAP. The ongoing IELTS Impact Study (IIS; Hawkey, 2006) also includes observational data of IELTS preparation classes. These studies all found IELTS to affect behaviour. Brown (1998) found that students in the 10-week IELTS preparation course used IELTS preparation textbooks, completed one Task 1 and one Task 2 essay each week (and no other writing), performed three timed practice examinations, were informed about IELTS scoring criteria, received feedback on the accuracy of their written work, and were instructed in strategies for writing under timed conditions. In contrast, students in the EAP course worked on a 1,000-word project, did no timed writing, were instructed in strategies for writing in academic contexts, and were encouraged to develop research skills. Read and Hayes (2003) combined a broad survey with targeted classroom observation of two teachers to provide mutual corroboration of findings. The briefer and more intensive of the two IELTS classes they observed was more narrowly concerned with the test and included more test practice under timed conditions. In common with other washback studies, Hawkey (2006) found variation between the 10 teachers he observed in how they conducted their classes, notably in the number of opportunities they provided for learners to communicate together in English. Teachers were willing to employ a variety of teaching methods and to use material both within and beyond the textbook. However, both the institutions providing the courses and the students, who were motivated to succeed on the test, appeared to constrain teachers to focus their instruction on IELTS. In class, teachers showed a preference for task-based activities, targeting microskills they believed to be relevant to the test. Data from observational studies have informed insights into the complexity of educational systems and test use, especially in relation to the role of the teacher. However, a shortcoming identified by Bachman (2005) is the lack of a coherent evidential link between test design characteristics and the practices observed or reported in the classroom. In studies that involve predicting the effects of a test on instruction or learning, appeal is more often made to the views of educational authorities (Cheng, 2005; Ferman, 2004; Qi, 2004), teachers (Banerjee, 1996), or widely held public perceptions (Watanabe, 1996) than directly to the design of the test instrument (as evidenced by available test materials or test specifications). How might the design of the IELTS Academic Writing component be expected to influence instruction?</p> <p>WATCHING FOR WASHBACK</p> <p>339</p> <p>Watanabe (2004) suggested two sources of evidence that may be used to relate observed practices to the influence of a test. One is evidence that test design features are reflected in teaching or learning. The other is the absence of such features in teaching or learning not directed toward the test (or directed toward an alternative test). Of the three IELTS-related studies, only Brown (1998), in common with Alderson and Hamp-Lyonss (1996) study of TOEFL preparation, incorporated a comparison with courses that were not directed toward the test. The inclusion of just two classes in each of the Australian studies limits their generalisability, while the lack of a nontest comparison in Hawkey (2006) makes it difficult to disentangle test influence from teacher variables. This study involves observation of a larger number of teachers and learners than Brown (1998) and Read and Hayes (2003) but includes the comparison with nontest EAP classes missing from Hawkey (2006). In considering the mechanisms of washback, a growing body of theory relates test design, test use, and classroom behaviours, although as Wall (2005) argued, too little is sufficiently informed by empirical evidence. Most of this work takes the form of recommendations to test developers. Chapman and Snyder (2000) provided a framework for relating tests to educational practices, and Brown (2000) cited Hughes (1989), Heyneman and Ransom (1990), Kellaghan and Greaney (1992), Bailey (1996), and Wall (1996) in identifying features of a test that may be manipulated in efforts to improve instruction. These embrace both contexts for test use and technical qualities of the test instrument. Drawing together these two elements in washback theory, Green (2003) proposed the predictive model of test washback set out in Figure 1. The model starts from test design characteristics and related validity issues of construct representation identified with washback by Messick (1996) and encapsulated in Resnick and Resnicks (1992) formulation of overlap, or the extent of congruence between test design and skills developed by a curriculum or required in a target language use domain. Test design issues are most closely identified with the direction of washbackwhether effects are likely to be judged beneficial or damaging to teaching and learning. The model relates design issues to contexts of test use, including the extent to which participants (including material writers, teachers, learners, and course providers...</p>


View more >