world languages initiative - american councils

AMERICANCOUNCILSFOR INTERNATIONAL EDUCATION

COLLEGE BOARD

WLAAC

MARCH 25, 2014

American Councils and ACTFL Test Development and Online Test Administration:

World Languages Initiative

• Arabic • Chinese (?) • Hindi • Korean • Persian • Russian

Language Assessed:

•  Portuguese •  Swahili •  Turkish •  Swahili •  Bangla** •  Bahasa** •  Azeri** ** in 2016 Others?

Communication Modalities: Reading, Listening, Writing, Speaking Modes: Interpersonal, Presentational, Interpretive

Goal: to provide opportunity and access to college for students who have studied or learned a critical language in K-12, at home, in the military, through participation in language training programs, such as STARTALK, NSLI-y, ROTARY, school exchanges, gap year, or other overseas experience) to document their functional proficiency in the language and provide an official report of their language levels (speaking, reading, listening, and writing) to the college, university or CTE of their choice.

Partners: The program is an initiative of American Councils Assessment Division in cooperation with the College Board and ACTFL. AC would be responsible for reading/listening test development, as well as overall test administration, registration, and score reporting. ACTFL/LTI will provide OPI test reporting. Writing tests would be available, when required, either via ACTFL/WPT or AC/Writing, depending on the language. CB would sanction the overall effort, recommending it to potential test takers through AP Central, Regional Offices, AP Coordinator networks, and main websites. Score distribution via the existing ETS system for distributing AP test scoring to university admissions offices t b d.

Cooperating universities: a group of leading US colleges and universities where critical languages are offered will accept the proposed proficiency-based AP grade for advanced placement and/or AP credit purposes at their institutions, based on a score of 3 corresponding to 3rd semester, 4 corresponding to 4th semester, and 5 corresponding to 5th semester study.

PROPOSAL FOR AN EXPANDED WORLD LANGUAGE AP© TEST PORTFOLIO TO COVER 12-15 ADDITIONAL MAJOR LANGUAGES

ACLASS: CORE FEATURES

Item Development • Web-based, collaborative • Robust version control • Blind rating • Discrete user levels • Workflow management

Registration • Testing centers • Admin personnel • Examinees • OPI scheduling

Test Administration Support for both low and high stakes testing Low Stakes

• No testing center/proctor • Students login with access code

High Stakes • Lab setup • Proctored administration • Real-time monitoring and support • Lock-down browser features

Security • Password protection • SSL Encryption • Cisco ASA 5510 Firewall • Protected RTMP streaming

with SWF verification • External security audit forthcoming

ONLINE TESTS ADMINISTERED VIA THE A-CLASS SYSTEM

IRS Funding: 2007-2010

Flagship Development Grant 2013-2014

Test Administrations, Jan – Feb, 2014

Examinees Tested

ACLASS: READING ITEM ADMINISTRATION (FIELD TEST)

ACLASS: ITEM DATA REVIEW PANEL

Item Key

Options 2 and 4 are unpopular distractors.

ACLASS: AN ITEM IS REJECTED

Assessment Development Process (High-level View)

Test Content Specification

Item Pool Development

Scoring & Reporting

Operational Test Administration

Assessment: Content Specifications

Test Content Specifications


Scoring & Reporting


Assemble Content Standards

Language- & Culture-Specific Rules

Item Writing Rules & Style Guide

Create Form Blueprints

Assessment: Item Development



Scoring & Reporting


Select Passages & Generate Items

Reviews: •  Passage & Item Content •  Distractors •  Bias & Style •  Blind

Field Test Psychometric Analysis I & Item Data Review

0111101010 1110111010 1111111111

Operational Item Pool

Assessment: Operational Administration



Scoring & Reporting


Form Blueprints

Operational Test Form(s)

Final QC Operational Administration

0111101010 1110111010 1111111111

Operational Item Pool

Assessment: Scoring and Reporting



Scoring & Reporting


Examinee Responses

Psychometric Analysis II

Standard Setting

Score Reporting

1 2 3 4 5 6 7

0.0

0.4

0.8

X

Res

pons

e Pr

obab

ility

Advanced ---------------- Intermediate

AMERICAN COUNCILS…

Psychometrics

Field Test Sampling, Item & Test Statistics, Item Data Reviews, Standard Settings, Technical Reports, General Statistical

Support

Assessment Division

Test Delivery

Software Construction & Maintenance for…

Item Banking, Student Rostering,

Final Form Assembly, Online

Administration, Results Reporting, Technical Reports

Content Development

Content and Test

Specifications, Item Writers, Item Pool,

Content and Bias Reviews,

Item Data Reviews, Final Form Assembly,

Standard Settings, Technical Reports

Current Reliability Statistics of American Councils’ Test Forms (Summer 2013 Field Testing)

Language Proficiency levels Name of Test Reading Listening

N KR20 N KR20 Arabic 1, 1+, 2, 2+ AOF 2013 Preprogram 53 .78 53 .79 Chinese 1, 1+, 2, 2+ COF 2013 Fall Pretest -‐ Simplified/

TradiGonal Combined 48 .83 48 .80 Russian 0+,1,1+,2,2+ RDF 2012-‐13 AY Post-‐Program -‐ Form A 78 .86 77 .82 Arabic NH-‐IH CLS 2013 Summer Arabic Preprogram 145 .90 143 .83 Chinese NM-‐IH NSLI-‐Y 2013 Chinese Summer Pre-‐

Program -‐ Form A 80 .83 80 .82 Chinese NH-‐AL NSLI-‐Y 2013 Chinese Summer Post-‐

Program -‐ Form A 46 .85 45 .79 Chinese NH-‐AL NSLI-‐Y 2013 Chinese Summer Post-‐

Program -‐ Form B 44 .87 44 0.81 Russian NH-‐IH Prototype AP® Russian Exam 2013 267 .90 267 .94

Items Developed Under 2012-13 Flagship Development and Maintenance Grants (per 09/25/13)

Reading items by level Total items Reading

Listening items by level Total items Listening

Total items Both Skills

0+ 1 1+ 2 2+ 3 3+/4 0+ 1 1+ 2 2+ 3 3+/4 Korean 6 53 30 113 36 36 15 289 10 51 24 94 45 42 13 279 568

Port. 7 42 48 115 38 64 27 341 7 22 49 128 46 61 22 335 676

Hindi 10 44 36 174 73 48 16 401 2 23 24 113 49 19 5 235 636

Urdu 8 32 25 48 59 56 20 248 9 32 24 49 57 77 20 268 516

Turkish 9 35 33 120 69 50 33 349 11 24 25 99 53 41 12 265 614

Swahili 5 15 13 45 28 20 12 138 5 12 11 47 26 40 4 145 283

Arabic 1 14 7 34 31 43 14 144 6 0 0 90 28 93 1 218 362

Russian 6 30 24 42 30 20 30 182 11 36 18 18 11 11 29 134 316

Chinese 22 15 16 57 8 19 1 138 19 10 12 71 30 41 1 184 322

Persian 3 8 10 31 18 18 3 91 1 13 12 68 29 53 4 180 271

•  Schedule •  November 11-December 3, 2013

•  IDR Workshop Staffing •  Content Developer •  Language Expert •  Psychometrician

•  Classical Test Theory Statistics •  P-Values •  Biserial Correlations •  Distractor Analyses

•  Nonparametric Item Response Curves

•  IRT Analysis (1-PL Model Using Mildly Informative Priors) •  Item Parameters •  Item Fit •  DIF Statistics, Provided that Focus and Reference Group Ns > 50

•  Females vs. Males •  Heritage Students or Native Speakers vs. Second Language Learners •  Unproctored vs. Proctored Administrations

2013 Field Tests: Item Data Reviews: COMPLETED.

•  Schedule •  January 7-9, 2014

•  Languages •  Hindi, Korean, Portuguese, Turkish, Urdu

•  Standard Setting Workshop Staffing •  Workshop Leaders and Moderators •  Eight Panelists and Two Tables per Language •  Content Development Staff •  Two Psychometricians •  Support Staff

•  Bookmark Method •  Item Parameter Estimation Augmented by Expert Judgment •  Table Leader and Panelist Trainings •  Study of Ordered-Item Booklet •  Setting Standard for ILR Level 2—Two Rounds •  Setting Standards for ILR Levels 1+ and 2+—Two Rounds •  Moderation and Articulation

•  Report

2013 Field Tests: Standard Setting

ACLASS TEST PRODUCTS Prototype AP® Russian Exam – High Stakes

Population: High school students

(typically grades 11-12)

Test range: NH-IH [Lower range]

Administration: Practice Exam in Jan/Feb

Once a year in April/May

Alternate test date in May

Numbers: Average 300-350 students

ACLASS TEST PRODUCTS NSLI-Y: Russian, Arabic, Chinese, Hindi, and Turkish Test range: NM, NH, IL, IM, IH, AL

Low Form and High Form Administration: Summer Pre-Program and Post-Program

(Arabic post-program only) Academic Year Pre-Program and Post- program

Numbers: Russian: 58 students

Arabic: 20 students Chinese: 90 students Hindi: 45 students Turkish: 38 students

ACLASS TEST PRODUCTS

Flagship — Arabic Test range: Lower Level [1,1+,2,2+]

Upper Level [2, 2+, 3, 3+/4 ] Administration: Pre-program—January

Spring [Post-program] Summer—[Post-program]

Numbers: Average 20-70 per administration


Flagship — Chinese

Test range: Lower Level [1, 1+, 2, 2+]

Upper Level [2, 2+, 3]

Administration:

Spring Entry Group: Preprogram—September/ October

Post-program—September/October

Fall Entry Group: Preprogram—January/February

Post-program—May/ June

Numbers: Average 39-100 students per admin.


Flagship — Russian Overseas

Test range: Low Level [ 1+,2, 2+, 3]

High Level [2, 2+, 3, 3+]

Administration: Qualifying --January

Fall [Pre-program]

Spring [Post-program]

Numbers: Average 30 students per test admin.


RLASP & Flagship — Russian Domestic

Test range: Lower Level [0+, 1, 1+]

Upper Level [2, 2+, 3]

Administration: Spring [Post-program]

Numbers: Average 50 students per admin


Flagship —Persian Overseas

Test range: Low Level [1, 1+,2,2+]

High Level [2, 2+, 3, 3+]

Administration: Pre-program—January


Summer—[post program]

Numbers: Average 20 students


AFLI and Flagship —Swahili

Test range: Low Level [1, 1+,2,2+]

Upper Level [2, 2+, 3, 3+]

Administration: Pre-program—January


Summer—[Post-program]

Numbers: Average 40


African Languages Initiative –Swahili, Yoruba

Test range: Lower Level [1, 1+,2, 2+]

Administration: Pre-program

Post-program

Numbers: Average 4-15 per admin.

ACLASS TEST PRODUCTS Project GO: Russian and Chinese: Summer 2013

Test range: 1, 1+, 2+

Administration: Proctored overseas administrations

in a variety of locations throughout

Summer 2013.

Numbers: Russian: 54 students

Chinese: 53 students

world languages initiative - american councils

Documents