what is a good language test? — on student contribution to the development of valid assessment: a...
TRANSCRIPT
What is a good language test?—
On student contribution to the development of valid assessment:
a European example
The Third Annual Conference of EALTA
Krakow 19-21 May 2006
Gudrun Erickson
Göteborg University, Sweden
European student 2005:
“I think that a good language test/ assessment should get students chance to show what they know and devolope their skills. It should contain of listening and reading comprehations, as well as writing and grammar part. But the most important thing is to speak in that language, so I think there should be also oral exams. Speaking is the most important, because without it, we wouldn’t communicate. Because of that students should also learn pronunciation.”
A wide framework
Learner-centeredness
Metacognition — Reflection — Awareness — Responsibility
Life-long learning
Learning to learn — Learner autonomy
Assessment for Learning
Self-assessment — Peer assessment
(European Language) Portfolio…
Why student involvement in assessment and testing?
VALIDITY REASONS
Ethical / Democratic reasons
Pedagogical reasons
Impact reasons
Obvious reasons!
“The candidates can provide test developers with very valuable insights: what they think about the test items, test methods, the clarity of instructions, the timing of the various sections, the relevance of the content in the light of their learning experiences or their purposes for learning the language, the relationship between how they perceive their learning abilities and their performance on the test in question, and so on…”
(Alderson, Clapham & Wall, 1995)
Examples of previous research on test-taker feedback
Shohamy (1982)
Cohen (1984)
Zeidner & Bensoussan (1988)
Brown (1993)
Kenyon & Stansfield (1993)
Fulcher (1996)
Halvari & Tarnanen (1997)
. . .
Two perspectives
Finding out about students—
Students as objects of research
&Finding out what students think/have to say
—
Students as active contributors to development and research
The 2005 ENLTA survey of students’ and teachers’ views on language testing and assessment
• Study of assessment policy and practice• Questionnaires to students and their teachers –
end of compulsory school• “What is a good language test? What is a bad
one? Why?” (Open-ended questions)
• Attitudes to + focus and frequency of, [formal] testing and continuous assessment (Likert scales)
Data• 10 countries
Bulgaria, Estonia, Finland, Hungary, Norway, Poland, Slovenia, Spain, Sweden, UK
• 1,373 students, 62 teachers
• Students usually around 16 yrs (14-18)
• Schools and students “reasonably average” — no claims
of representativity
• Answers almost exclusively in English, and referring to
English (EFL)
• Report on www.ealta.eu.org > Resources
RESULTS
Students’ definitions of
good language tests
Coverage and variety
Communicative usefulness (authenticity)
Learning potential
Clarity
Fairness
• “A good language test is checking all language abilities.”
• “A good language test is the one that concernes a varied tasks such as speaking, listening, reading and grammar. Also tests on vocabular are neccessary, however it should not be the main part of the exam.”
• “A good language test is a test whiff a lot of diferent parts. Not just gramma, it is to much. The best test is an essaytest, becouse there you test a lot of different things at the same time.”
• “Good assessment is that you learn something new that you don’t know before the assessments.”
• “Good test: The instructions are clear and example is given.”
• “In my opinion, a good test should make the writer feel challanged and it should mainly teach the students how to communicate with people.”
Students’ definitions of
bad language tests
Limited, “narrow” tasks
Lack of apparent usefulness; Irrelevance
Stress; Lack of time
“ If the test is narrow and has only one sort of tasks, it might happen that a person with good language skill gains a worse score than others… The test must contain of as much different tasks as possible in order to give everyone equal chanses as they encounter a great variaty of exercises and they show their entire knowledge…”
• “A bad test/assesment is learning vocabulary too detailed. Learning many vegetables that I don’t know what the look like and will never need them in real life is absurd. We should be expected to know everyday life vocabulary that we can use for communication with native english speakers.”
• “In my opinion a test and a assessment must have the questions well structured, clear instructions, clear exercises, not too long because when the student see a long test they feel stressed and don’t concentrate. The test shouldn’t be confusing so that the students can answer well and fail the test.”
• “I think the uneffective language tests are those that only test one area of language criteria and apply lots of pressure so the candidate is nervous and stressed, as they do not perform at their best.”
Where students sometimesdon’t agree
Test preparation
Level of difficulty
Content
(e.g., concerning oral tests and grammar)
• “In my opinion a good exam is one that has similar things to those worked in class, with similar exercises and vocabulary, a level that the students can pass if they have studied.”
• “I think a good test is unprepared. If we have an unprepared test I don’t feel any pressure because I haven’t study well enog. In a unprepared test you get to show what you really know about english. It’s not hard to get good grades in a prepared test, if you study hard. If I study very hard the day before a test I forget very much after a week. Good luck with the reasearch!”
• “I think a good test is an easy test, because then I don’t have to learn so much.”
• “I think that the best language test is test, that is not easy. That’s why you have learn a lot. I think that the worsest language test is test, that is too easy…”
— — — — —
• “Test is hard and listening even horrible.”
• “I like listenings and vocabulary exercises, but grammar is to me like a black magic…”
• “I am happy becauce we haven’t got any ”speaking on my own” or ”talking with others” tests. I get easily so nervous if I must talk English. I like to do normal test because then I can work on my own. Normaly we have 2-4 things. We read them and then we have a test…”
Students’ attitudesto the language in focus (1)
5-point Likert scales (5 = agree strongly)
1
2
3
4
5
I like learning thelanguage
I think I do well...
Students’ attitudes (2)
1
2
3
4
5
..chance toshow what I
know
I learn a lot I often feelnervous
..importantthings
assessed
I normallydo well
ExamsLessons
Students’ perceptions of the frequency of testing and assessment within
different areas
1
2
3
4
5
ExamsLessons
Students’ and teachers’ perceptions of frequency compared
Students TeachersGrammar Written production
Writing texts Reading compr.
Reading compr. Words & phrases
Words & phrases Grammar
— — — — — — — — — — — — — — — — — —
Pronunciation Oral interaction
Talking with others Pronunciation
Students as collaborative partnersThe Swedish example
• Extensive system of national testing and assessment
• Formative and summative materials
• Advisory function — teachers responsible for final grading
• Continuous development work in broad groups of experts
• Pre-testing of all tasks, for all materials, in large, randomly
selected groups of students in the country (n≈400)
• Systematic collection of students’ opinions — Likert
scale statements & open comments
• Qualitative and quantitative analyses of results and opinions
Examples of questions asked
• General acceptance • Level of perceived difficulty
• Aspects of time/speed; content/relevance; clarity
• Familiarity • Retrospective self-assessment
…and analyses performed• Characteristics of good tasks / bad tasks
• Attitudes and self-assessment of different subgroups: gender, ability, age…
• Relationships between attitude and performance
How results are used
To better understand how items function
To trace and eliminate possible obscurity and ambiguity
To choose texts, topics and tasks
To compose tests
To optimize sequencing of tasks
To improve information and guidelines
Students as contributors to validity (1)
Objecting to construct under-representation
“A bad test/assessment is the one which is only about grammar, because if a person knows grammar well, it doesn’t mean he/she can speak the language as well and communication is the most important thing in language study…” (European student, 2005)
“A bad test is when you only get tested in one thing and you dont get to show what you realy are good at…”(European student, 2005)
“…the test is too narrow and fails to include important dimensions or facets of the construct…” (Messick, 1989)
Students as contributors to validity (2)
Highlighting and criticizing construct irrelevant variance
“… if you don’t understand the questions, you can’t show what you know…” (European student, 2005)
“… bad test is when we just have a little time to do it on. Because then you have to stress thru the test and maybe you cannot show how much you can…” (European student, 2005)
“… A bad test is when other things than language are assessed (e.g. knowledge)…” (European student, 2005)
“…the test contains excess reliable variance that is irrelevant to the interpreted construct…” (Messick, 1989)
In conclusion
Students provide developers of tests and assessments at all levels with unique information about different aspects of materials and methods
used — information that contributes to
“…the adequacy and appropriateness of inferences and actions based on test scores and other modes of
assessment…” (Messick, 1989),
i.e.
enhances validity.
And finally…
“It’s nice to do nice
exercises, that is the thing
which teachers should
remember.
Even tests can be nice.”