Principles of test design Objectives, measurement qualities, challenges KAYSERI, 29-31 JANUARY 2014

Aims

!  Definining our assessment objectives !  Measurement qualities: ‚test usefulness’ !  Testing challenges !  Considering impact

Testing: a definition

!  “the systematic gathering of information for the purpose of making decisions” (Weiss 1972)

!  “the collection of reliable and relevant information” (Bachman 1990)

!  What information? !  What decisions?

Defining our test objectives

1)  What kind of test is it going to be? 2)  What decisions do we need information for? 3)  What abilities are tested? 4)  How much detail do we need? 5)  How accurate should our results be? 6)  Is there any washback effect? 7)  What are the practical constraints?

! examinations !  formal tests ! oral tests ! home assignments !  term reports ! continuous assessment ! projectwork, research work ! student self-evaluation

Assessment in the classroom

!  for placement or level testing: to decide on the level of learners

!  for diagnostic testing: to identify individual strengths or weaknesses

!  to evaluate progress !  to evaluate skills for specific needs !  to give marks for performance !  to evaluate and update syllabus and objectives !  also to prepare/train for exams

Purposes of tests

Contrasting categories of ESL tests

Knowledge Performance (or skills)

Subjective Objective

Productive Receptive

Language sub-skills Communication skills

Norm-referenced Criterion-referenced

Discrete-point Integrative

Proficiency Achievement (or progress)

Types of tests (Madsen, 1983)

"  A correspondence between language test performance and language use ◦  “In order for a particular language test to be useful for its

intended purposes, test performance must correspond in demonstrable ways to language use in non-test situations.”

"  A clear and explicit definition of qualities of test usefulness ◦  “Usefulness = Reliability + Construct validity + Authenticity +

Interactiveness + Impact + Practicality” (Bachman & Palmer, 1996)

Key principles of testing

"  consistent across populations "  consistent within the same test

irrespective of: !  setting

!  marker

!  item set When conditions of the test remain unchanged, it always produces essentially the same results.

Qualities of test usefulness: reliability

! Types of validity (Alderson et al, 1995) !  internal validity

!  face validity

! content validity

! response validity

! external validity ! concurrent validity

! predictive validity

! construct validity !  meaningful? appropriate? representative?

Qualities of test usefulness: validity

CHECKLIST FOR VALID TESTS Tests what it claims to test?

Best format for what you want to test?

Content relevant to student’s real-life needs?

Items typical/representative of learner use?

Items don’t accidentally test other things?

Test skill, NOT knowledge, creativity or logic?

Clear and appropriate marking criteria?

Construct validity

Testing competence from Language Testing in Practice by Bachman & Palmer (1996, OUP)

Misconceptions !  one best test for a given

situation

!  misunderstanding the nature of testing and test development

!  unreasonable expectations

!  blind faith in measurement

Resulting problems !  inappropriate tests

!  failure to meet specific needs

!  uninformed use of popular testing methods

!  frustration about not finding the perfect test

!  loss of faith in self ! testing can only be done by experts

!  defending the indefensible (stakeholders’s expectations)

"  testing = “real life”? “authentic”? "  “communicative language testing”? "  defining abilities in a test "  selecting tasks and items ! competence? "  measurement principles:

can “unidimensional scores for locally independent test items reflect authentic language use”? (Bachman)

"  outside factors ! performance?

Challenges in testing

Thank you!

zoltan.rezmuves@digimedia.hu

consonantvoiced.blogspot.com