going for gold - can checklists be a gold-testing tool in language testing and assessment ?
DESCRIPTION
Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?. Sauli Takala University of Jyväskylä 12th CERCLES Conference London Sept. 6-8, 2012. Overview Some current issues in definition of the construct in language education - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/1.jpg)
Sauli TakalaUniversity of Jyväskylä
12th CERCLES Conference London Sept. 6-8, 2012
![Page 2: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/2.jpg)
Overview•Some current issues in definition of the construct in language education•Task-based language learning/assessment : neglect of some relevant sources, critique of some sources•Some models of TB- assessment •Some arguments for and against a checklist approach•Assessment using checklists•Concluding remarks
![Page 3: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/3.jpg)
Approaches to definition of construct of language performance :
•ability-focused (eg,. Bachman 2002, 2006)•task-focused (eg., Norris 2005)•interaction-focused.
•See next slide for an overview of approach to construct definition by Bachman (2007; What is the construct? The Dialectic of Abilities and Contexts in Defining Constructs in Language Assessment. In Fox et al. Language Testing Reconsidered. Ottawa: U of Ottawa Press, 41-71)
![Page 4: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/4.jpg)
Approaches to defining the construct in language testing/assessment, Bachman, 2007Construct Ability/Trait Task/Content
Skills & elements;Lado, 1961; Carroll1961
Elements/aspects/levels/ integrated skills
Discrete points, integrative tasks. Taxonomy óf tasks.
Direct testing, perform. assClark 72, Jones85, Wesche
Lang proficiency on tasks, performance in real life
Tasks mirroring/duplic. real-life tasks; authentic p.
Pragmatic LT; Oller 1979
Pragm expectancy gramm.
Pragmatic tests
Communicative LT; Canale & Swain 1980 etc.
Communicative compet. General language profic.
Meaningful comm. situat.Authentic tasks
Interaction-ability: Bach-man 1990, Bach & Palm96
Communic. lang. abilityLanguage ability
Test methof facetsTask characteristics
Task.based perf. assessm1Brindley1994, MacNam96
Ability for language use
Simulation of resl-world tasks
Task-based perf.assessm2Norris1998, Brown2002
Ability to accomplish part. tasks or task types
Performance on particular tasks ort task types
Minimalist interactionalist Kramsch1986, Chapelle98
Intectional competence/ ability
Collaborative activity, character. of interaction
Strong interactionalistHe&Young 98, Young00
Interactional competence
Discursive practices
Moderate interact. Chalhoub-Deville03
Ability-in-individual-in - context
![Page 5: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/5.jpg)
Are there fundamental differences or only a matter of difference in emphasis?
Are we, in practice, usually interested in finding out to what extent there is ability in using language to do some tasks?
Is there an inevitable interaction between degree of ability and the nature of the tasks?
Do tasks represent the goals/required competences of interest?
![Page 6: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/6.jpg)
Some useful early sources - but to some extent neglected
•Linguistic philosophy (esp. later Wittgenstein, Philosophical Investigations, 1953)•Sociology and social psychology (Kurt Lewin, Bales (1950), Festinger (1951), Watzlawick (1967) – communication and interaction in small groups•Linguistic pragmatism – especially early critique of Austin (1955/1962), and Searle (1969) by German pragmalinguists (Wunderlich, 1972; Ehlich & Rehbein, 1972 – praxeograms...)
![Page 7: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/7.jpg)
ROLE/SOCIAL ROLE: also neglected?
In social psychology, a role: a set of connected behaviours, rights and obligations as conceptualised by actors in a social situation. It is an expected or free or continuously changing behaviour providing a given individual social status or social position.
Functionalist and interactionist understandings of society and all social behaviour, including communicative behaviour (cf. next slide)
•Division of labour - interaction among heterogeneous specialised positions/roles.•Roles are occupied by individuals/actors.•Social roles/behaviours are guided by social norms (expectations, degree of conventionality).•Legitimate and constructive roles – anticipation of rewards (vs. punishments)•Changed social conditions role change.
![Page 8: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/8.jpg)
![Page 9: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/9.jpg)
External situation
Internal situaton
Code: usage
Persons: goals, relation-ships
Time, place, ”noise”
Channels. media
Topic/aspect
Processing: input-output
Dynamics of situated inter-action
![Page 10: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/10.jpg)
Fischer et al., Guidelines for task-based university language testing. Graz: ECML, 2011, p. 21.
![Page 11: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/11.jpg)
Puren, C. ( 2008). Formes pratiques de cominaision entre perspectives actionnelle et approche communicative: analyse comparative des trois manuels. Langue Modernes. Cited in Fischer et al., 2011, p. 22. (note the inclusion of roles!)
![Page 12: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/12.jpg)
Testing/assessment – issues and problems
•Testing/assessment is notoriuosly demanding/ complex•Testing/assessment should fulfil many criteria – (a) there are many things that should be got right•(b) there may be some things that one may not know well enough (inevitable degree of ignorance/ even ineptitude?)•(c) there may be serious consequences.-> Can testing/assessment be made less demanding, less complex, and exclude avoidable problems/ errors?
![Page 13: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/13.jpg)
Assessment – So Many Things to Get Right
•Observing aspects of good practice <-> learning to first avoid less good practices •For instance, in learning to rate, avoiding
What about using Checklists?
Halo (horn) effect Central tendency effect Restriction of range effectSeverity or leniency
![Page 14: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/14.jpg)
• Checklists: Everyday use – shopping lists• Checklists/Inventories – have a long history in eg.
personality inventories (self-assessment; preferences).
• Checklists – frequent use esp. in health care/ nursing, aviation/pilots, building industry….
• Gawande, A. (2010). The checklist manifesto. How to get things right. London: Profile Books.
• Western Michigan Checklist Project (www.xxx)• Scriven, M. (2000/2007). The Logic and
Methodology of Checklists. Western Michigan University.
![Page 15: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/15.jpg)
Reluctance to use checklists - objections
It somehow feels beneath us to use checklists, an em-barrassment. It runs counter to the deeply held beliefs about the how truly great among us – those we aspire to be – handle situations of high stakes and complexity. The truly great are daring. They improvise. They do not have protocols and checklists. Maybe our idea of heroism need updating. (Gawande, p. 173)
Checklists – boring -> Professionalism – expectations: selflessness, skill, trustworthiness, discipline. We are not built for discipline (novelty, excitement). We have to work at discipline. (p. 182-3).
What about education?
![Page 16: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/16.jpg)
Checklists are valuable evaluation devices when carefully developed, validated, and applied. A sound evaluation checklist•clarifies the criteria that at least should be considered when evaluating something in a particular area• aids the evaluator not to forget important criteria•enhances the assessment's objectivity, credibility, and reproducibility.
Moreover, such a checklist is useful in planning, monitoring and guiding operations, and assessing their outcomes. In the evaluation vernacular, checklists are useful for both formative and summative evaluations. (Scriven, M. The Logic and Methdology of Checklists. Western Michigan University. 2000, 2007)
![Page 17: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/17.jpg)
Checklists are not created equal – more bad than good checklists?
•Bad checklists: vague, imprecise, too long, hard to use, impractical
•Good checklists: precise, efficient, to the point, easy to use, provide reminders (not exhaustive), practical. Usually 5- 9 points only.
![Page 18: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/18.jpg)
Types of Checklists•List (mnemonic) – eg. shopping list; laundry list – grouping important•Sequential checklist - order matters: strongly sequential kind, where the sequencing (of some or all checkpoints) must be followed in order to get valid results. •A weakly sequential checklist: order is of some importance, but for psychological or efficiency reasons rather than from logical or physical necessity
![Page 19: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/19.jpg)
Iterative checklist is sequential, in whole or part, but requires—or may require—multiple passes in order to reach a stable reading on each checkpoint.
Diagnostic checklist that is used—for example— by taxonomists, mechanics, and toxicologists. It typically supports a classificatory kind of conclusion—one that may be descriptive or evaluative.
Probably the most important kind of checklist for evaluation purposes is the criteria of merit checklist ( COMlist). This is what judges use when rating entries in a skating or barbeque or farm produce competition (and, of course, when rating language performance).
![Page 20: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/20.jpg)
Key Requirements for Comlists /Criteria of Merit)
Most of the following are self-explanatory and refer to the criteria or checkpoints that make up a comlist:1.The checkpoints should refer to criteria and not mere indicators.2.The list should be complete (no significant omissions). The items should be contiguous, i.e., nonoverlapping (essential if the list is used for scoring).3.The criteria should be commensurable.4.The criteria should be clear (comprehensible, applicable).5.The list should be concise (to assist its mnemonic function); i.e., it should contain no superfluous criteria.6.The criteria should be confirmable (e.g., measurable or reliably inferrable)
(Scriven, 200, 2007)
![Page 21: Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?](https://reader035.vdocuments.mx/reader035/viewer/2022062518/568143eb550346895db0703f/html5/thumbnails/21.jpg)
Positive developments
•Lack of knowledge /ignorance diminishing: reasonable guidelines of good practice.•Transparency is making some headway: useful research eg., on linking exams to the CEFR is being carried out and increasingly reported. •Training on language testing/assessment is increasingly available (EALTA, ALTE).•Thus, we have increasingly better chances to get things right,
Scriven, M. (2000/2007). The Logic and Methodology of Checklists. Western Michigan University.Gawande, A. (2010). The checklist manifesto. How to get things right. London: Profile Books.