dubeau testing research bucharest
TRANSCRIPT
-
7/25/2019 Dubeau Testing Research Bucharest
1/37
STANAG 6001-OPI Testing
Julie J. Dubeau
Bucharest BILC 2008
-
7/25/2019 Dubeau Testing Research Bucharest
2/37
Julie J. Dubeau
Bill Who???
-
7/25/2019 Dubeau Testing Research Bucharest
3/37
Julie J. Dubeau
-
7/25/2019 Dubeau Testing Research Bucharest
4/37
Julie J. Dubeau
Are We All On the Same
Page?
An Exploratory Study of OPI RatingsAcross NATO CountriesUsing the NATO STANAG !!" Scale#
*This research was completed in 2006 as part of a M.A. Thesis inApplied Linguistics
-
7/25/2019 Dubeau Testing Research Bucharest
5/37
Julie J. Dubeau
Pesentation Outline
Context
Research Questions
Literature Review
Methodolo!
Results
" Ratins
" Raters" #cale
Conclusion
-
7/25/2019 Dubeau Testing Research Bucharest
6/37
Julie J. Dubeau
NATO !anguage Testing "onte#t
#tandardi$ed Lanuae %ro&ile '#L%(based on the )*+, #+*)D*RDI-*+I,)
*R//M/)+ ')*+, #+*)*( 001Lanuae %ro&icienc! Levels '/d 1 /d 2(
" 2 )*+, countries3 20 %artnershi4 &or %eace
'%&%( countries 5 others6
-
7/25/2019 Dubeau Testing Research Bucharest
7/37
Julie J. Dubeau
InteroperabilityProblem?
Lanuae trainin is central within ar7ed &orces dueto the increasin nu7ber o& 4eacesu44ort
o4erations3 and is considered as havin an
i74ortant role in achievin interoperability
a7on the various 4la!ers.
9+he sinle 7ost i74ortant 4roble7 identi&iedb! al7ost all 4artners as an i74edi7ent to
develo4in intero4erabilit! with the *lliancehas been shortco7ins inco77unications: '/*%C '%*R%( D3 1;;
-
7/25/2019 Dubeau Testing Research Bucharest
8/37
Julie J. Dubeau
O$ea%hing &esea%h 'uestion
#ince no =nown stud! had investiated interrater
reliabilit! in this context3 the 7ain research >uestion
was?
How comparable or consistent are ratings across
AT! raters and countries"
-
7/25/2019 Dubeau Testing Research Bucharest
9/37
Julie J. Dubeau
&esea%h 'uestions
Research questions pertaining to the ratings
RQ1
Research questions pertaining raters training
and bacground RQ!
Research questions pertaining to the rating
process and to the scale RQ"
-
7/25/2019 Dubeau Testing Research Bucharest
10/37
Julie J. Dubeau
&esea%h 'uestions
RQ1#Ratings$ (o) *o atings o+ the sa,e oal o+i%ien%
inte$ie)s /OPIs %o,ae +o, ate to ate?
Woul* the use o+ lus le$els in%ease ateagee,ent?
(o) *o the atings o+ the OPIs %o,ae +o,
%ount to %ount?
Ae thee *i++een%es in s%oes )ithin the sa,e
%ount?
-
7/25/2019 Dubeau Testing Research Bucharest
11/37
Julie J. Dubeau
&esea%h 'uestions
RQ!#Raters training and bacground$ Ae thee *i++een%es in atings bet)een ates )ho
ha$e e%ei$e* $aing *egees o+ testeatetaining an* STANAG taining?
Di* $e e#eien%e* ates s%oe ,oe eliablthan lesse e#eien%e* ones? Ae e#eien%e*ates s%oing as eliabl as taine* ates?
Ae thee *i++een%es in atings bet)eenati%iants )ho test at-ti,e $esus +ull-ti,e2ae nati$e o non-nati$e sea3es o+ 4nglish2 an*ae +o, 5Ol*e an* 5Ne)e NATO %ounties?
-
7/25/2019 Dubeau Testing Research Bucharest
12/37
Julie J. Dubeau
&esea%h 'uestions
RQ"#Rating process and scale use$
Do *i++eing ating a%ti%es a++e%t atings? Do ates aea to use the s%ale in si,ila
)as?
What ae the ates %o,,ents ega*ing theuse an* ali%ation o+ the s%ale?
-
7/25/2019 Dubeau Testing Research Bucharest
13/37
Julie J. Dubeau
!iteatue &e$ie)
+estin Constructs
" @hat are we testin
A eneral 4ro&icienc! 5 @h!
A Ratin scales
Rater ariance
" ow do raters var!
A Raterscale interactionA Rater trainin 5 bac=round
-
7/25/2019 Dubeau Testing Research Bucharest
14/37
Julie J. Dubeau
7etho*olog
#esign of study$/x4lorator! surve!" 2 ,ral %ro&icienc! Interviews ',%Is * 5 B(
" Rater data >uestionnaire
" Questionnaire acco74an!in each sa74le ,%I
%artici4ants ? Countries recruited at BILC #e7inarin #o&ia 200E
10F raters &ro7 18 countries and 2 )*+, units
-
7/25/2019 Dubeau Testing Research Bucharest
15/37
Julie J. Dubeau
Analsis8" Ratin co74arisons
" ,riinal ratins" G%lusH ratins
" Rater co74arisons
" +rainin
" Bac=round
" Countr! to countr! co74arisons
A @ithin countr! dis4ersion
" Ratin 4rocess
A Ratin &actors
" Raterscale interaction
A #cale user&riendliness
-
7/25/2019 Dubeau Testing Research Bucharest
16/37
Julie J. Dubeau
&esults &'1- Su,,a
Ratins ? +o co74are ,%I ratins and to
ex4lore the e&&icac! o& G4lus ratinsH.
" #o7e ratertorater di&&erences
" G%lusH levels brouht ratins closer to the 7ean
" #o7e countr!tocountr! di&&erences" reater Gwithincountr!H dis4ersion in so7e
countries
-
7/25/2019 Dubeau Testing Research Bucharest
17/37
Julie J. Dubeau
withinlevel Fwithinlevel 2withinlevel 1%tac&ed'iewofA
I0F020100
View of OPI ratings sample A
-
7/25/2019 Dubeau Testing Research Bucharest
18/37
Julie J. Dubeau
&esultsSample A %&1'All Ratings %(ith )'
Levels )u7bers
@ithin Level 1 rane
-
7/25/2019 Dubeau Testing Research Bucharest
19/37
Julie J. Dubeau
All *ountries +eans ,or Sample A
2.02.202.001.801.01.01.201.00
!'erall (ountry Mean
(ountrynu
mbers 20
))+
),)6
)-)
)/)2
)))0
+
,
6-
/
2)
-
7/25/2019 Dubeau Testing Research Bucharest
20/37
Julie J. Dubeau
All Ratings ,or Sample -%le.el !'
Levels )u7bers
1 2 1.;
1K 1 1.0
2 < E.
2K 8
-
7/25/2019 Dubeau Testing Research Bucharest
21/37
Julie J. Dubeau
/ie( o, OPI ratings sample -
withinlevel Iwithinlevel Fwithinlevel 2withinlevel 1%tac&ed'iewof0I0F020100
-
7/25/2019 Dubeau Testing Research Bucharest
22/37
Julie J. Dubeau
All *ountries +eans ,or Sample -
.F.002.
-
7/25/2019 Dubeau Testing Research Bucharest
23/37
Julie J. Dubeau
&esults &'9- Su,,a Raters? +o investiate rater trainin and scale trainin
and see how 'or i&( the! i74acted the ratins3 and to
ex4lore how various bac=round characteristics
i74acted the ratins
" +rained raters scored within the 7ean3 es4eciall!&or sa74le B
" /x4erienced raters did not do as well as scale
trained raters
" ullti7e raters scored closer to 7ean" G)ewH )*+, raters scored slihtl! closer to 7ean
" ))# raters scored slihtl! closer to 7ean
-
7/25/2019 Dubeau Testing Research Bucharest
24/37
Julie J. Dubeau
substantial to lotsnone to little
-
7/25/2019 Dubeau Testing Research Bucharest
25/37
Julie J. Dubeau
:eas o+ 4#eien%e
E !ears K to E !ears2 to F !ears0 to 1 !ear
E0
0
F0
20
10
0
2re3uency
.-4
)-.+4
).+4
).+-4
-
7/25/2019 Dubeau Testing Research Bucharest
26/37
Julie J. Dubeau
substantial to lotsnone to little
0
E0
0
F0
20
10
0
5ercent
0.04
60.04
%TAA %cale Training
-
7/25/2019 Dubeau Testing Research Bucharest
27/37
Julie J. Dubeau
5Ol* $s. 5Ne) NATO "ounties
Summary of Tester Trg
Little Lots
Total
14
20
236
Newer NATO
member? Yes
No
Total
6
23
29
30
28
58
36
51
87
-
7/25/2019 Dubeau Testing Research Bucharest
28/37
Julie J. Dubeau
5Ol* $s. 5Ne) NATO "ounties
Rating OPI B Corret?
Yes No
Total
14
20
236
Newer NATO
member? Yes
No
Total
27
27
54
6
!6
32
37
55
92
Other!issi"g
4
2
6
-
7/25/2019 Dubeau Testing Research Bucharest
29/37
Julie J. Dubeau
&esults &ates Ba%3goun*
Conducts +estin ullti7eA Nes F 'FF.0 (
A )o < 'E.0 (A ullti7e testers 7ore reliable 'accurate(
" ))# '0( raters better trained
" G)ewH raters better trained
-
7/25/2019 Dubeau Testing Research Bucharest
30/37
Julie J. Dubeau
&esults &';- Su,,a
#cale? +o ex4lore the wa!s in which raters used the
various #+*)* state7ents and ratin &actors to
arrive at their ratins.
" Ratin 4rocess did not a&&ect ratins sini&icantl!
" F 7ain Gt!4esH o& raters e7ered?
A /videncebased
A Intuitive
A /xtracontextual
-
7/25/2019 Dubeau Testing Research Bucharest
31/37
Julie J. Dubeau
&esults
*n GevidencedbasedH ratin &or #a74le B 'level 2(?
I co74ared the candidateHs 4er&or7ance with
the #+*)* criteria 'levels 2 and F( and
decided that he did not 7eet the
re>uire7ents &or level F with reard to
&lexibilit! and the use o& structural devices.
/rrors were &re>uent not onl! in low&re>uenc! structures3 but in so7e hih
&re>uenc! areas as well. $Rater %! & rated '(
-
7/25/2019 Dubeau Testing Research Bucharest
32/37
Julie J. Dubeau
&esults
*n GintuitiveH ratin &or #a74le * 'level 1(?
I )oul* sa that
-
7/25/2019 Dubeau Testing Research Bucharest
33/37
Julie J. Dubeau
&esults
*n GextracontextualH ratin &or #a74le * 'level 1(?
Level 3 is the basic level needed for officers in
(my country). I think the candidate could performthe tasks required of him. He could easily be
bulldozed by native speakers in a meeting but
!ould hold his o!n !ith non"native speakers. Hemakes mistakes that very rarely distort meaning
and are rarely disturbing.$Rater %) & rated '(
-
7/25/2019 Dubeau Testing Research Bucharest
34/37
Julie J. Dubeau
I,li%ations
+rainin not e>ual in all countries
#cale inter4retation
%lus levels use&ul
Di&&erent rids3 s4ea=in tests
Institutional 4ers4ectives
-
7/25/2019 Dubeau Testing Research Bucharest
35/37
Julie J. Dubeau
!i,itations = >utue &esea%h
%artici4ants 7a! not have rated thiswa! in their own countries
,%Is new to so7e 4artici4ants
uture research could
" et 4artici4ants to test" Investiate ratin rids
" Loo= at other s=ills
-
7/25/2019 Dubeau Testing Research Bucharest
36/37
Julie J. Dubeau
"on%lusion o+ &esea%h
#o3 are we all on the sa7e 4ae
N/#O *UT+
%lus levels were instru7ental in bridina4
+rainin was &ound to be =e! to reliabilit!
More incountr! trainin should be the &irst
ste4 toward international bench7ar=in.
-
7/25/2019 Dubeau Testing Research Bucharest
37/37
Than3 :ouAre 7e All !n the %ame 5age"
An Exploratory Study of OPI Ratings
Across NATO Countries
Using the NATO STANAG !!" Scale
#ubeau.889forces.gc.ca
The full thesis is a'ailable on the (#A website
http$::www.cda;acd.forces.gc.ca!r google #ubeau thesis