establishing evidence of construct: a case study · establishing evidence of construct: a case...
TRANSCRIPT
![Page 1: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/1.jpg)
Establishing evidence of construct: A case study
Barry O’Sullivan
Roehampton University
Cathy Taylor
Trinity College London
Dianne Wall
Trinity College London & Lancaster University
![Page 2: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/2.jpg)
Purpose of our paper
• To describe approach taken by Trinity College London to gather empirical evidence of the construct underlying its examinations
• To present main research findings
• To present implications for Trinity and the language testing community
![Page 3: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/3.jpg)
Outline• Why Trinity decided to carry out this research
• The examination
• The validation model
• The method
• Research findings
• Benefits of going through such an exercise
![Page 4: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/4.jpg)
Purpose of the study
To gather empirical evidence of
the construct underlying one of our examinations
![Page 5: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/5.jpg)
purpose for the study
ILTA A test designer must decide on the construct to be measured and state explicitly how that construct is to be operationalised.
![Page 6: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/6.jpg)
Purpose for the study
EALTA Section 1:Test purpose and specificationAre the constructs intended to underlie the test/subtest(s) specified?
Section 5: ReviewAre validation studies conducted?
![Page 7: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/7.jpg)
Graded Examinations in Spoken
English: an outlineTrinity Stages
Trinity grades
CEFR levels
12 C2
Advanced 11 C1
10 C1
9 B2
Intermediate 8 B2
7 B2
6 B1
Elementary 5 B1
4 A2
3 A2
Initial 2 A1
1 -GESE syllabus
![Page 8: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/8.jpg)
GESE: exam tasks
Advanced25 mins
Intermediate15 mins
Elementary10 mins
Initial5, 6 or 7 mins
Conversation
Topic discussion
Topic discussion
Conversation
ConversationInteractive task
ConversationListening task
Topic Discussion
Topic presentation
Interactive task
Language skills� Communicative skills
� Language functions
� Grammar
� Lexis
� Phonology
![Page 9: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/9.jpg)
Syllabus requirements
![Page 10: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/10.jpg)
Syllabus requirements
![Page 11: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/11.jpg)
Canale & Swain model of communicative competenceComponent Definition
Grammatical Knowledge of lexical items and of rules of morphology, syntax, sentence-grammar semantics, and phonology
Sociolinguistic Knowledge of the socio-cultural rules of language and of discourse
Discourse Ability to connect sentences in stretches of discourse and to form a meaningful whole out of a series of utterances
Strategic The verbal and non-verbal communication strategies that may be called into action to compensate for breakdowns in communication due to performance variables or due to insufficient competence
![Page 12: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/12.jpg)
Validity & validation
validity A theoretical model that underpins an argument supporting the use of a test with a specified population for a specified purpose.
validation The generation of evidence concerning the operationalisation and interpretation of the validity model
![Page 13: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/13.jpg)
The validation model
• Rejected these in favour of the updated Weir (2005) model described by O’Sullivan & Weir (2011) & O’Sullivan (2011)
• First looked to Messick & Mislevy
![Page 14: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/14.jpg)
The validation model
CONSTRUCT
![Page 15: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/15.jpg)
The methodology
1. An overview of the level from the candidate and cognitive domain
2. A specific look at the parameters that define the level (contextual evidence) – this part was divided into a series of tables, one to reflect each of the five task types.
3. An overview of the appropriacy of the scoring system
4. Some general comments on the underlying language model
![Page 16: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/16.jpg)
The test taker I
![Page 17: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/17.jpg)
The test taker II
![Page 18: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/18.jpg)
The test taker III
![Page 19: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/19.jpg)
The test taker IV
![Page 20: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/20.jpg)
The test task
![Page 21: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/21.jpg)
The scoring system
![Page 22: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/22.jpg)
Communicative Competence
![Page 23: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/23.jpg)
The findings I
The Test Candidate
Appropriate from all perspectives
The Scoring System
Procedures appropriate [25% - 30% multiple marking]
Recommendation to reconsider including a specific range of lexis or syntax as rating criteria across all tasks
The Test Task
Range of tasks ensure suitability for purpose
Some issues with the listening task
![Page 24: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/24.jpg)
The findings II
Strategic CompetenceDiscourse Competence
Linguistic Competence
Emerging evidence up to level 3
More systematic growth at each subsequent level
Sociolinguistic Competence
Emerging evidence up to level 6
More systematic growth at each subsequent level
![Page 25: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/25.jpg)
Benefits of the project
• Confirmation of ‘communicativeness’
• Confirmed by external experts
• Research projects
• GESE development
• Transferability
![Page 26: Establishing evidence of construct: A case study · Establishing evidence of construct: A case study ... Trinity Stages Trinity grades CEFR levels 12 C2 ... 5, 6 or 7 mins Conversation](https://reader031.vdocuments.mx/reader031/viewer/2022020316/5b81c4617f8b9a2b6f8d0b7d/html5/thumbnails/26.jpg)
Thank you!