qualities speech sample analysis test ... meetings seminars and...qualities of good testing practice...
TRANSCRIPT
![Page 1: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/1.jpg)
12/14/2010
1
QUALITIES OF GOOD TESTING PRACTICE
Sergey MELNICHENKO,
CompLang Aviation Training Centre (Moscow),
Deputy Director p y
International Civil Aviation English Association, Member of the Board
Flight Safety Foundation,
Member
QUALITIES
Qualities of Good Testing Practice
SPEECH SAMPLE ANALYSIS
TEST ADMINISTRATION ISSUES
![Page 2: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/2.jpg)
12/14/2010
2
QUALITIES
Qualities of Good Testing Practice
RELIABILITYCONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESSIMPACTPRACTICALITY
RELIABILITY
Qualities of Good Testing Practice
CONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESSIMPACTPRACTICALITYPRACTICALITY
![Page 3: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/3.jpg)
12/14/2010
3
Qualities of Good Testing Practice
Same results every time we administer the test
Objective rating
Qualities of Good Testing Practice
‐ inter‐rater reliability
‐ intra‐rater reliabilityy
![Page 4: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/4.jpg)
12/14/2010
4
RELIABILITY
Qualities of Good Testing Practice
CONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESSIMPACTPRACTICALITY
Qualities of Good Testing Practice
Meaningfulness and appropriateness
Are we measuring what we h d i t d d t ?had intended to measure?
![Page 5: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/5.jpg)
12/14/2010
5
RELIABILITYCONSTRUCT VALIDITY
Qualities of Good Testing Practice
CONSTRUCT VALIDITY
AUTHENTICITYINTERACTIVENESSIMPACTPRACTICALITY
Qualities of Good Testing Practice
Rich in topic, theme or context
Degree to which test task characteristics match those of th TLU t kthe TLU tasks
![Page 6: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/6.jpg)
12/14/2010
6
Qualities of Good Testing Practice
RELIABILITYCONSTRUCT VALIDITYAUTHENTICITY
INTERACTIVENESSIMPACTIMPACTPRACTICALITY
Qualities of Good Testing Practice
Are there test tasks to evaluate:
‐Comprehension of various accents?‐Ability to paraphrase?‐Language behavior in linguistic complications?Ad d i t d t f‐Adequacy during unexpected turn of events?‐Familiarity with cultural subtleties? ‐etc…
![Page 7: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/7.jpg)
12/14/2010
7
RELIABILITY
Qualities of Good Testing Practice
CONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESS
IMPACTIMPACTPRACTICALITY
Qualities of Good Testing Practice
Aviation English test results are felt around the globe
![Page 8: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/8.jpg)
12/14/2010
8
Qualities of Good Testing Practice
RELIABILITYCONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESSIMPACT
PRACTICALITY
Qualities of Good Testing Practice
A
DesigningDevelopingAdministeringRating
VS Available resources
![Page 9: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/9.jpg)
12/14/2010
9
RELIABILITY
Qualities of Good Testing Practice
CONSTRUCT VALIDITYAUTHENTICITYINTERACTIVENESSIMPACTPRACTICALITY
SPEECH SAMPLE ANALYSIS
Qualities of Good Testing Practice
SpeakingListening ComprehensionReadingWriting (Typing)g ( yp g)
![Page 10: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/10.jpg)
12/14/2010
10
SPEECH SAMPLE ANALYSIS
Qualities of Good Testing Practice
PronunciationStructuresVocabularyFluencyComprehensionInteractions
‐Accents
Qualities of Good Testing Practice
PRONUNCIATION
![Page 11: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/11.jpg)
12/14/2010
11
Qualities of Good Testing Practice
‐Pronunciation errors• Mispronunciation of meaning‐di i i hi d (b b )distinguishing sounds (burn – born)
•No difference between short and long vowels (feet – fit, lead – lid)•Devocalization of sonorous•Devocalization of sonorous consonants in the end of words (surge – search, bag – back)
Qualities of Good Testing Practice
‐Pronunciation errors•Wrong stress in “international”
d ( d h l)words (academy, hotel)• Intonation typical for the native language• Mispronunciation of interdentalconsonants (think sink)consonants (think – sink)
![Page 12: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/12.jpg)
12/14/2010
12
STRUCTURES
Qualities of Good Testing Practice
• Basic and complex structures• Errors do not affect meaning• Errors affect meaning• Grammatically accurate structures
d i it ti ff tused in a wrong situation affect meaning
VOCABULARY
Qualities of Good Testing Practice
•General English lexis• Semi‐technical lexis • Terminologies• Ability to paraphrase• Switching to native language
![Page 13: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/13.jpg)
12/14/2010
13
Qualities of Good Testing Practice
FLUENCY •Smoothness of speechp• Rate of speech• Absence of excessive pausing• Absence of disturbing hesitation markers
Qualities of Good Testing Practice
FLUENCY •Length of utterancesg• Speech connectedness
![Page 14: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/14.jpg)
12/14/2010
14
Qualities of Good Testing Practice
COMPREHENSION•Phonological form of wordsg• Morphological form of words• Syntactical and semantic connections
Qualities of Good Testing Practice
COMPREHENSION•Ability to recognize y gcommunicative function of utterances• Ability to deduce meaning of unfamiliar lexical items from context
![Page 15: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/15.jpg)
12/14/2010
15
Qualities of Good Testing Practice
INTERACTIONS•Immediate?• Informative?•Appropriate?
Qualities of Good Testing Practice
QUANTITATIVE MEASURING• consistentlyy
• almost never• rarely• sometimes• frequently• usually• not always• often• occasionally• mostly• generally
![Page 16: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/16.jpg)
12/14/2010
16
Qualities of Good Testing Practice
Adequate test+
Collection of evidence for both negative andCollection of evidence for both negative and positive examples of language use
+Quantitative measuring
+Correlation of resultsCorrelation of results with the ICAO Scale
=Adequate evaluation
of language proficiency levels
TEST ADMINISTRATION
Qualities of Good Testing Practice
ISSUES
![Page 17: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/17.jpg)
12/14/2010
17
Qualities of Good Testing Practice
≠
‐Division between testing and training services
T R A I N I N G
Qualities of Good Testing Practice
Teachers shall not be interlocutors or‐Teachers shall not be interlocutors or raters for the group they have taught
![Page 18: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/18.jpg)
12/14/2010
18
Qualities of Good Testing Practice
‐Test service providers shall be responsible for their domestic andresponsible for their domestic and
foreign affiliates
Qualities of Good Testing Practice
Level 4
Publishing a “black list” of TSP‐ Publishing a black list of TSP
![Page 19: QUALITIES SPEECH SAMPLE ANALYSIS TEST ... Meetings Seminars and...Qualities of Good Testing Practice ≠ ‐Division between testing and training services T R A I N I N G Qualities](https://reader030.vdocuments.mx/reader030/viewer/2022040212/5e8588c0259228054009193e/html5/thumbnails/19.jpg)
12/14/2010
19
Bibliography1. ICAO Doc 9835 –Manual on the Implementation of ICAO Language
Proficiency Requirements, Second Edition – 2010
2. ICAO Circ 318 – Language Testing Criteria for Global Harmonization
3. L.F. Bachman, A.S. Palmer – Language Testing in Practice, Oxford University Press, 1996
4. J. Charles Alderson, Caroline Clapham, Dianne Wall – Language Test Construction and Evaluation, Cambridge University Press, 1995
5. Dan Douglas – Assessing Languages for Specific Purposes, Cambridge University Press, 2000
6. James E. Purpura – Assessing Grammar, Cambridge University Press, 2004
7. John Reed – Assessing Vocabulary, Cambridge University Press, 2000
8. Sari Luoma – Assessing Speaking, Cambridge University Press, 2004
9. Gary Buck – Assessing Listening, Cambridge University Press, 2001
10. Lyle F. Bachman – Statistical Analysis for Language Assessment, Cambridge University Press, 2004
11. Test of English Language Level for Controllers and Pilots (TELLCAP®) ‐Statute and testing materials, 2005‐2010
CompLang Aviation Training Centre
Molodogvardeyskaya Str., 61MoscowMoscow,
121351, [email protected]
Tel (+7495) 589 1356(+7495) 991 3703
Thank You
Fax (+7495) 589 1356www.complang.ru