Download - Pt 2: Rogan presentation
-
Language Proficiency Assessment
Seumas RoganChief, Test Design & Analysis
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Develop, validate, administer, sustain, and
assess results of standardized language
proficiency tests; educate on standards
and evaluate student feedback, all in support
of the Defense Foreign Language Program.
Mission Statement
2
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Priorities
Sustainmentofdeployedassessments. Validityandreliabilityofalltestmodalities. Alignmentoftheworkforcewiththemissionandstrategy.
Stabilizationandcontrolofthebusinessprocesses.
Definition/implementationofthenextgenerationofassessments.
3
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
LPAD Org Chart
TestProductionDivision(TP)
TestAnalysisandDesignDivision
(TAD)
TestManagementDivision(TM)
OralProficiencyStandardsDivision
(PSD)
TestReviewandEducationDivision
(TRE)
DeputyDirector
Director,LanguageProficiency
Assessment
4
EvaluationsDivision(EV)
Technology
StakeholderRelations
Dr. Pradyumna Amatya
Brent EickholtDr. Tom Parry Dr. Gerd Brendel
Dr. Chung Yao Kao Dr. Seumas Rogan
Susan Hagan
Kalman Weinfeld
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Summary of LPAD Functions
TP: Produce language proficiency assessments in Listening and Reading Comprehension.
TRE: Review test items and train faculty and staff in ILR. TAD: Design and analyze the performance of language
proficiency assessments. PSD: Certify and manage the performance of Oral
Proficiency Interview (OPI) testers. TM: Schedule and administer DLI graduation tests and
external OPI. EV: Survey students and report statistics and red-flags. Technology: Design and maintain automated solutions for
language testing business processes. 5
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
QA/QCEthical
Practices
NEEDSANALYSIS
TESTSPECS
FRAMEWORK
DEVELOPMENT
PRETEST&ANALYZE
STANDARDSETTING
DEPLOYTEST
MAINTAINEVALUATE
LESSONSLEARNED
IMP
LEM
EN
TATI
ON
DLPT Lifecycle
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Test Production
Test platforms: Multiple Choice (MCT) high volume (>200). Constructed Response (CRT) low volume.
Standard development latency: ~30 months. Standard development cost (Listening + Reading):
~$1.25M. Sustainment and Computer Adaptive Tests require
pools of characterized items: Generated via automatic seeding of new items in the
released test forms.7
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Test Production
Automatic Seeding Status: Multiple Choice Tests (MCT): on line for 6 languages (Chinese-
Mandarin, Modern Standard Arabic, Pashto, Persian Farsi, Russian, Spanish); 4 additional languages scheduled/year.
Constructed Response Tests (CRT): Reduction in scored items (via a reduction in scored levels) required to enable seeding.
Scheduled Test Releases (2015-2017): 9 MCT: Urdu, Chinese-Cantonese, Tagalog, Portuguese, German, Hindi,
Thai, Swahili, Vietnamese 6 CRT: Haitian-Creole, Yoruba, Kazakh, Amharic, Hausa, Malay
Computer Adaptive Tests (CAT): Planning implementation for 13 DLI high-volume languages.
8
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Test Review and Education
Conduct independent target language reviews of each test-item set for correctness, completeness, appropriateness, and adherence to specified ILR levels.
Provide Text Typology and Passage Rating training under the ILR Guidelines to all DLPT5 test developers, independent reviewers, and DLIFLC faculty (by request).
Represent DLI at the ILR testing work-group to ensure interoperability among various government agencies and international partners.
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Test Analysis & Design
Coordinate DLPT design initiatives: DLPT5 Validity Framework
What do DLPT scores mean? How should they be used?
Item Bank Specification Are we asking the right questions?
Web-Based Field Testing Can we obtain a representative sample of examinees to
calibrate items?
Small-n Standard Setting Study How do we set passing scores without item parameter data?
10
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Highlight: Design
Issue LackoffieldtestingparticipationforDLPT5sinCantonese,Tagalog,Hindi,German
Resultsin: Examineeproficiencymisclassification
Redress WebBasedFieldTesting
11
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Test Analysis & Design
Conduct statistical analysis of: Item response data
Do test items perform within specified tolerances?
Test form reliability Are results on test forms equivalent?
Standard setting cut score recommendations Are cut scores fair, reliable and valid?
Examinee comments /feedback Have examinees expressed actionable concerns?
Item bank characteristics What is the distribution of content across the range of
difficulty?12
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Highlight: Analysis (1)
Issue Questionablestandardsettingpanelistrecommendations
Resultsin: Examineeproficiencymisclassification
Redress Smallnstandardsettingstudy
Additionalfieldtestingdata 13
0.1
.2.3
Den
sity
0.5
11.
5D
ensi
ty
0 5 10 15 20 25 30 35 40 45 50 55 60NC Cuts
ILR LVL 1+ ILR LVL2ILR LVL 2+ ILR LVL 3
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Highlight: Analysis (2)
Issue DLIFLCfacultyquestionwhyfewexamineesareawardedILR2+inKoreanDLPT5
Redress Demonstratethat,consistentwithtestspecifications,maximumscoreprecisionatILRlevels2/2+/3
14
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Train and certify select faculty as OPI testers at DLIFLC.
Provide orientation training for DLIFLC faculty on the ILR and OPI.
Ensure appropriate interpretation and uniform implementation of the ILR at DLIFLC and the DLIFLC contract entities.
Provide quality assurance that OPI testers (DLIFLC and contract) are providing consistently fair and accurate assessments.
Proficiency Standards Division
15
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Test Management Division
Schedule, test, and report results for DLIFLC resident students.
Grade all Constructed Response Tests administered. FY14 volume: DLPT (RC+LC)@ DLI: 8,949
(Worldwide: ~123,000); OPI: DLI: 3,407, External: 15,463; ICPT: 10,745; CRT Gradings: 10,193.
Test capacity: 5 DLPT test labs; 7 OPI studios. New semester-based scheduling requires 3 additional OPI
studios.
16
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Evaluation Division
Provide DLIFLC leadership with valid and reliable evaluative information.
Outcomes (FY14): 5,330 ISQ/ESQ Evaluation surveys.121 Red Flag reports.198 Snapshot reports.256 Attrition surveys.394 Non-Resident Surveys.
17
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Technology
18
DOMINO (Test Item Development and Workflow Automation System) Centralized and version controlled system hosting 49
DLPT5 development projects 52 active users 7,670 test passages; 11,464 test items
Enforces standardized workflow and task assignment processes
Expanding for contractor-model of DLPT item development, Psychometric support and direct reporting
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
SharePoint Platform for ongoing DLPT5 External Review contract
150 active users 5,008 test items
Repository of previously contracted DLPT5 items 32,952 test items
TDMS (Test Management System) Automatic scheduling and scoring processes for OPI and
DLPT Repository of score data Expanding to support Psychometric Item-Response data
requirements 19
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
A few words about the ILR
InterAgencyLanguageRoundtable Datesto1950s Skillleveldescriptionsfor
Speaking,Listening,Reading, UsedastheprimaryreferencebyUSGovernmentAgencies
http://www.govtilr.org/
20
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
ILR Proficiency Scale
SetofgenerallanguageproficiencydescriptionsindicatingwhatlanguagelearnersCANorCANNOTdousingthetargetlanguage i.e.,Knowledge,Skills&Abilities(KSAs)of/withthetargetlanguage
6baselevels,andplus(+)levelsinbetweentoindicatethatproficiencyexceedsonelevelbutisnotsustainedatthenextlevel
UsedbyDLIFLCindevelopingDLPT5andclassifyingtheresults
21
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
5
0
1
2
3
4
0+
1+
2+
3+
4+5 = Functional Native Proficiency
4 = Advanced Professional Proficiency
3 = General Professional Proficiency
2 = Limited Working Proficiency
1 = Elementary Proficiency
0 = No Proficiency
4+ = Advanced Professional Proficiency +
3+ = General Professional Proficiency +
2+ = Limited Working Proficiency +
1+ = Elementary Proficiency +
0+ = Memorized Proficiency
22
ILR Proficiency Scale
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
5
0
0+
1
1+
2
2+
3
3+4
4+
Focus of the Lower Range DLPT5
23
ILR Proficiency Scale
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Speaking 1 (Elementary Proficiency):
Initiates,maintains,andbringstoclosesimpleconversationsbyaskingandrespondingtosimplequestions.Canbeunderstood,withsomerepetition,byspeakersaccustomedtodealingwithnonnativespeakers.
24
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Speaking 2 (Limited Working Proficiency):
Narratesanddescribesinmajortimeframesanddealseffectivelywithanunanticipatedcomplication.Canbeunderstoodwithoutdifficultybyspeakersunaccustomedtodealingwithnonnativespeakers.
25
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
Speaking 3 (General Professional Proficiency):
Discussestopicsextensively,supportsopinionsandhypothesizes,dealswithalinguisticallyunfamiliarsituation.Errorsneverinterferewithcommunicationordistractthenativespeakerfromthemessage.
26
-
DEFENSE LANGUAGE INSTITUTE FOREIGN LANGUAGE CENTER
For More Information:
http://www.dliflc.edu/
[email protected] [email protected] [email protected] [email protected]
27