what is a good language test? — on student contribution to the development of valid assessment: a...

What is a good language test?—

On student contribution to the development of valid assessment:

a European example

The Third Annual Conference of EALTA

Krakow 19-21 May 2006

Gudrun Erickson

Göteborg University, Sweden

Gudrun.Erickson@ped.gu.se

European student 2005:

“I think that a good language test/ assessment should get students chance to show what they know and devolope their skills. It should contain of listening and reading comprehations, as well as writing and grammar part. But the most important thing is to speak in that language, so I think there should be also oral exams. Speaking is the most important, because without it, we wouldn’t communicate. Because of that students should also learn pronunciation.”

A wide framework

Learner-centeredness

Metacognition — Reflection — Awareness — Responsibility

Life-long learning

Learning to learn — Learner autonomy

Assessment for Learning

Self-assessment — Peer assessment

(European Language) Portfolio…

Why student involvement in assessment and testing?

VALIDITY REASONS

Ethical / Democratic reasons

Pedagogical reasons

Impact reasons

Obvious reasons!

“The candidates can provide test developers with very valuable insights: what they think about the test items, test methods, the clarity of instructions, the timing of the various sections, the relevance of the content in the light of their learning experiences or their purposes for learning the language, the relationship between how they perceive their learning abilities and their performance on the test in question, and so on…”

(Alderson, Clapham & Wall, 1995)

Examples of previous research on test-taker feedback

Shohamy (1982)

Cohen (1984)

Zeidner & Bensoussan (1988)

Brown (1993)

Kenyon & Stansfield (1993)

Fulcher (1996)

Halvari & Tarnanen (1997)

. . .

Two perspectives

Finding out about students—

Students as objects of research

&Finding out what students think/have to say

—

Students as active contributors to development and research

The 2005 ENLTA survey of students’ and teachers’ views on language testing and assessment

• Study of assessment policy and practice• Questionnaires to students and their teachers –

end of compulsory school• “What is a good language test? What is a bad

one? Why?” (Open-ended questions)

• Attitudes to + focus and frequency of, [formal] testing and continuous assessment (Likert scales)

Data• 10 countries

Bulgaria, Estonia, Finland, Hungary, Norway, Poland, Slovenia, Spain, Sweden, UK

• 1,373 students, 62 teachers

• Students usually around 16 yrs (14-18)

• Schools and students “reasonably average” — no claims

of representativity

• Answers almost exclusively in English, and referring to

English (EFL)

• Report on www.ealta.eu.org > Resources

RESULTS

Students’ definitions of

good language tests

Coverage and variety

Communicative usefulness (authenticity)

Learning potential

Clarity

Fairness

• “A good language test is checking all language abilities.”

• “A good language test is the one that concernes a varied tasks such as speaking, listening, reading and grammar. Also tests on vocabular are neccessary, however it should not be the main part of the exam.”

• “A good language test is a test whiff a lot of diferent parts. Not just gramma, it is to much. The best test is an essaytest, becouse there you test a lot of different things at the same time.”

• “Good assessment is that you learn something new that you don’t know before the assessments.”

• “Good test: The instructions are clear and example is given.”

• “In my opinion, a good test should make the writer feel challanged and it should mainly teach the students how to communicate with people.”

Students’ definitions of

bad language tests

Limited, “narrow” tasks

Lack of apparent usefulness; Irrelevance

Stress; Lack of time

“ If the test is narrow and has only one sort of tasks, it might happen that a person with good language skill gains a worse score than others… The test must contain of as much different tasks as possible in order to give everyone equal chanses as they encounter a great variaty of exercises and they show their entire knowledge…”

• “A bad test/assesment is learning vocabulary too detailed. Learning many vegetables that I don’t know what the look like and will never need them in real life is absurd. We should be expected to know everyday life vocabulary that we can use for communication with native english speakers.”

• “In my opinion a test and a assessment must have the questions well structured, clear instructions, clear exercises, not too long because when the student see a long test they feel stressed and don’t concentrate. The test shouldn’t be confusing so that the students can answer well and fail the test.”

• “I think the uneffective language tests are those that only test one area of language criteria and apply lots of pressure so the candidate is nervous and stressed, as they do not perform at their best.”

Where students sometimesdon’t agree

Test preparation

Level of difficulty

Content

(e.g., concerning oral tests and grammar)

• “In my opinion a good exam is one that has similar things to those worked in class, with similar exercises and vocabulary, a level that the students can pass if they have studied.”

• “I think a good test is unprepared. If we have an unprepared test I don’t feel any pressure because I haven’t study well enog. In a unprepared test you get to show what you really know about english. It’s not hard to get good grades in a prepared test, if you study hard. If I study very hard the day before a test I forget very much after a week. Good luck with the reasearch!”

• “I think a good test is an easy test, because then I don’t have to learn so much.”

• “I think that the best language test is test, that is not easy. That’s why you have learn a lot. I think that the worsest language test is test, that is too easy…”

— — — — —

• “Test is hard and listening even horrible.” 

• “I like listenings and vocabulary exercises, but grammar is to me like a black magic…”

• “I am happy becauce we haven’t got any ”speaking on my own” or ”talking with others” tests. I get easily so nervous if I must talk English. I like to do normal test because then I can work on my own. Normaly we have 2-4 things. We read them and then we have a test…”

Students’ attitudesto the language in focus (1)

5-point Likert scales (5 = agree strongly)

1

2

3

4

5

I like learning thelanguage

I think I do well...

Students’ attitudes (2)

1

2

3

4

5

..chance toshow what I

know

I learn a lot I often feelnervous

..importantthings

assessed

I normallydo well

ExamsLessons

Students’ perceptions of the frequency of testing and assessment within

different areas

1

2

3

4

5

ExamsLessons

Students’ and teachers’ perceptions of frequency compared

Students TeachersGrammar Written production

Writing texts Reading compr.

Reading compr. Words & phrases

Words & phrases Grammar

— — — — — — — — — — — — — — — — — —

Pronunciation Oral interaction

Talking with others  Pronunciation

Students as collaborative partnersThe Swedish example

• Extensive system of national testing and assessment

• Formative and summative materials

• Advisory function — teachers responsible for final grading

• Continuous development work in broad groups of experts

• Pre-testing of all tasks, for all materials, in large, randomly

selected groups of students in the country (n≈400)

• Systematic collection of students’ opinions — Likert

scale statements & open comments

• Qualitative and quantitative analyses of results and opinions

Examples of questions asked

• General acceptance • Level of perceived difficulty

• Aspects of time/speed; content/relevance; clarity

• Familiarity • Retrospective self-assessment

…and analyses performed• Characteristics of good tasks / bad tasks

• Attitudes and self-assessment of different subgroups: gender, ability, age…

• Relationships between attitude and performance

How results are used

To better understand how items function

To trace and eliminate possible obscurity and ambiguity

To choose texts, topics and tasks

To compose tests

To optimize sequencing of tasks

To improve information and guidelines

Students as contributors to validity (1)

Objecting to construct under-representation

“A bad test/assessment is the one which is only about grammar, because if a person knows grammar well, it doesn’t mean he/she can speak the language as well and communication is the most important thing in language study…” (European student, 2005)

“A bad test is when you only get tested in one thing and you dont get to show what you realy are good at…”(European student, 2005)

“…the test is too narrow and fails to include important dimensions or facets of the construct…” (Messick, 1989)

Students as contributors to validity (2)

Highlighting and criticizing construct irrelevant variance

“… if you don’t understand the questions, you can’t show what you know…” (European student, 2005)

“… bad test is when we just have a little time to do it on. Because then you have to stress thru the test and maybe you cannot show how much you can…” (European student, 2005)

“… A bad test is when other things than language are assessed (e.g. knowledge)…” (European student, 2005)

“…the test contains excess reliable variance that is irrelevant to the interpreted construct…” (Messick, 1989)

In conclusion

Students provide developers of tests and assessments at all levels with unique information about different aspects of materials and methods

used — information that contributes to

“…the adequacy and appropriateness of inferences and actions based on test scores and other modes of

assessment…” (Messick, 1989),

i.e.

enhances validity.

And finally…

“It’s nice to do nice

exercises, that is the thing

which teachers should

remember.

Even tests can be nice.”