a new aptitude test -...

A New Aptitude Test• CASL Research on Language Aptitude

DLAB 2 Increase prediction of basic proficiency (ILR 2) & predict attritionALAB Language neutral testHi-LAB Predict ultimate attainment (ILR3+/4)ATI Aptitude-by-treatment interaction: Match training to aptitude

Theoretical Framework Develop a research program for getting USG language professionals to very high levels of proficiency?

Aptitude‐by‐Treatment Interaction!

Little research directly investigating aptitude for high‐level language.

Uses of Aptitude Tests

3

• Selection

• Diagnosis

• Placement

• Counseling

• Tailored Training (ATI)

• Predicting performance outside classroom

• SLA Research

4

Predicting Ultimate AttainmentProblem

Need to identify individuals who can attain near‐native proficiency, given motivation and opportunity.

RelevanceAdult language learning is time consuming, effortful and expensive.

SolutionCognitive abilities that underlie advanced language learning processes are measurable. High levels of cognitive abilities may predict near‐native attainment.

Why is language learning so hard?

5

What predicts LL success?

Talent: Cognitive

Ability

Experience

Motivation

Personality Facets

Many factors

Natural abilityDrivePerseveranceTrainingCoaching

“L2 aptitude is the individual’s initial state of readiness and capacity for learning a foreign language, and probable facility in doing so [given the presence of motivation and opportunity].”

J. B. Carroll, 1981

7

Second Language Aptitude

Talent: Cognitive

Ability

What language aptitude explains

• Speed of language acquisition

under intensive learning conditions (rate)

(Carroll and others following)

• Individual differences in

post‐CP ultimate attainment (ceiling)

(CASL)

8

Carroll’s 4 Aptitude Constructs

DLAB MLAT PLAB

Phonetic Coding Ability

Part 2, Recognition of Stress PatternsPart 3, Foreign Language Grammar

Part 2, Phonetic Script Part 3, Spelling Clues

Part 5, Sound DiscriminationPart 6, Sound-Symbol Association

Grammatical Sensitivity

Part 3, Foreign Language GrammarPart 4, Foreign Language Concept Formation

Part 4, Words in Sentences

Part 4, Language Analysis

Rote Learning Ability

Part 3, Foreign Language Grammar

Part 1, Number Learning Part 5, Paired Associates


Explicit Inductive Language Learning Ability

Part 4, Foreign Language Concept Formation

Part 1, Number Learning


9

Sources of Individual Differences

10

Hi‐LAB = Cognitive Aptitude

the working memory system

11

Hi‐LAB’s unique prediction

• Hi‐LAB is designed to predict the ultimate level of attainment by adult starters.

• While recognizing the importance of other individual differences and experience

• to success in language learning, the purpose of Hi‐LAB is to measure cognitive aptitude.

12

Summary

• Critical period offsets disrupt language learning.

• Empirical studies show wide variation in ultimate attainment post‐critical period.

• Successful individuals may be those with language aptitude which helps to counteract critical period effects.

14

InstrumentationHi‐LAB is a more direct battery

•Innovative: computer‐delivered behavioral measures (E‐prime CRO)

•Data types: accuracy, reaction time

•Not transparent to learners

•Instructions provided & response type and individual responses PRACTICED

•Fun!?

Hi‐LAB Constructs and MeasuresHi-LAB Constructs and MeasuresWorking Memory

Executive Functioning

Updating Running Memory Span

Inhibitory Control Antisaccade

Stroop Simon

Task Switching Task Switching Numbers (mix cost and switch cost)

Phonological Short-term Memory Letter Span

Non-Word Span

Associative Memory Paired Associates

Long-term Memory Retrieval (Priming) ALTM Synonym

Implicit Induction (have added explicit) Serial Reaction Time (Letter Sets)

Processing Speed Serial Reaction Time (could be from other measures)

Auditory Perceptual Acuity Phonemic Discrimination: Hindi, English Pseudo-Contrastive

Phonemic Categorization: Russian

15

Our iterative research approach

• Propose new constructs (motivated by theory)

• Develop measures, assess usabilityusability revise

• Gather evidence for reliabilityreliability revise

• Gather evidence for validityvalidity revise

• Gather information about testing environment construct operational tests

Aims of Reliability Research

• Improve the reliability of Hi‐LAB tests

• Reduce the size of the test battery to fit within reasonable testing time limit

(7 2.5 1.5 hours)

• Regularize scoring procedures

• Examine theoretical construct validity

(factor analysis)

17Doughty et al (2007) (RS1) and (2009) (RS2)

Study MethodParticipants

• RS1: 195 NSs of English (UMD)• RS2: 142 NSs of English (DLIFLC)Materials

• 42 measures 22 measures (based on RS1)• 2 parallel versions of the battery: Hi‐LAB Version A and Hi‐LAB Version B

Design

18

TestAB 2 weeks + Re-test

BA

Hi‐LAB Factors and Measures

19

Hi‐LAB ConstructsCognitive Construct Factor Reliable Stable

Jointly Acceptable

Short Term Memory 2 of 2 1 of 2 1 of 2

Updating 2 of 2 1 of 2 1 of 2

Inhibition ? 2 of 2 1 of 2 1 of 2

Task‐switching 2 of 2 1 of 2 1 of 2

Rote Memory 2 of 2 1 of 2 1 of 2

Perceptual Acuity 3 of 4 2 of 4 2 of 4

Speed 2 of 2 1 of 2 1 of 2

Semantic Primability 2 of 2 1 of 2 1 of 2

Implicit Induction 1 of 2 1of 2 1 of 2

Explicit Induction Letter Sets

20

20

Simon

Aims of Hi‐LAB Validity Research

• Phase 1: Factor analyses to theoretical construct validity

• Phase 2: Group discrimination

• (Phase 3: Gather longitudinal validity evidence )

• Phase 4: Operational validation (October 2013)

Mislevy et al. 2010 - AUA

Study Method

Aim – Investigate whether Hi‐LAB discriminates groups with known levels of L2 ability

(Pairs – propensity score matching;

Group discrimination – logistic regression)

Moderately Successful Group

Highly Successful Group

Age Gender Level of Education

Linck, Hughes, Campbell, Silbert, Tare, Jackson, Smith, Bunting & Doughty (in press, Language Learning 2013).

Hi‐LAB correctly classifies high‐level and normal attainment

GROUPListening (n = 152)

Reading(n = 188 )

Either(n = 206)

High‐Level vs.Normal Attainers

**70% **59% **67%

the working memory system

** p < .01

Validity: Hi-LAB is accurate in group discrimination

Linck, Hughes, Campbell, Silbert, Tare, Jackson, Smith, Bunting & Doughty (in press, Language Learning 2013).

Best predictors in regression modelListening Reading Either

Task switching

Letter Span Letter span Letter span

Paired Associates Paired Associates Paired Associates

Serial Reaction Time Serial Reaction Time Serial Reaction Time

24Linck, Hughes, Campbell, Silbert, Tare, Jackson, Smith, Bunting & Doughty

(in press, Language Learning 2013).

Further Test Development

• Can the aptitude dimensions be further reduced?

• Is language aptitude distinct from IQ?

25

Study methodParticipants

• 428 NSs of English

Analyses• Principal components analysis

• Hierarchical and non‐hierarchical cluster analyses

26

Hi‐LAB Component Reduction?

27

Not yet …

Language Aptitude distinct from IQ?CQBN Measures (n = 132)

Hi-Lab Measures (n = 428) Variable NameDescription

Processing Speed CQBN1PMPattern Matching

Serial Reaction Time CQBN2MT Matrices

Phonemic Discrimination CQBN3EU English Usage

Phonemic Categorization CQBN4WM Wording Meaning

Phonemic Cat. Learning CQBN5SPSpelling

Mix-Cost RT CQBN6DRDirections

Switch-Cost RT CQBN7AR Arithmetic

Anti-Saccade Accuracy CQBN8NN Name & Number Check

Stroop CQBN9AL Artificial Language

Letter Span

Running Span

Non-Word Span

ALTM

28

Is language aptitude unique?

29

Hi‐LAB CQBN

Hi‐LAB + CQBN

Significant

Non‐significant

Hi-LAB accounts for substantial unique variance

Quartimax rotated loadings, 4 Factors, Hi‐LAB only

Measure Factor 1 Factor 2 Factor 3 Factor 4

Processing.Speed.TS0.761

Serial.Reaction.Time

Phonemic.Disc0.576

Phonemic.Cat0.630

Phonemic.Learn0.364

MixCost.RT‐0.564

SwitchCost.RT0.514 0.404

Anti.ACC‐0.522

Stroop.RT

Letter.Span0.959

Running.Span0.512

NonWord.Span

ALTM‐0.528

30

Conservative and less conservative analyses of # of components to retain

31

• CQBN 2 or 8 of 9 components

• Hi‐LAB 4 or 10 of the 13 components

• Hi‐LAB

and CQBN 7 or 21 of the 22 components

32

32

33

33

Theoretical Framework : ATI

Assess effectiveness

Apply treatments (training)

Identify groups based on aptitudes

Collect data on participants

All participants

Group A

Training A

Training B

Group B

Training A

Training B

36

37

Aptitude‐by‐Treatment InteractionProblem

Individuals who are already selected for intelligence and motivation and have attained functional ability need to acquire higher levels of language

RelevanceHigher‐level ability is essential to the job

SolutionMatch cognitive aptitude profile to language training procedures to optimize training and boost outcome levels

QuestionTo what extent do aptitude profiles exist?

Cluster Analyses

• Solutions from hierarchical and non‐hierarchical clustering algorithms indicate that some clusters may usefully be defined with respect to average aptitude profiles.

38

Aptitude Profiles: 2 Cluster Solution: (1) all high (2) all low

39

Aptitude Profiles: 3 Cluster Solution: (1) all high (2) all low (3) mixed (speed)

40

Mixed group strength = memory and perceptual acuity

4 Cluster Solution

41

1 Cluster good almost across the board; 4th poor3rd cluster: good at only STM, Updating, Perceptual Acuity4th cluster: good only at Speed, Induction, Inhibition, Priming, STM

Steps to develop ATI materialsTAILORING SECOND LANGUAGE TRAINING: STEPS

Step 1. Identify a learning problem relevant for the learners’ and instructors’ needs

Step 2. Consider the learners’ aptitude profile card

Step 3. Consider practical constraints

Step 4. Design the training

Step 5. Determine the language outcome measures (pre‐ and post‐training tests)

Step 6. Implement the training

42

Example of ATI: Variety in speech

• L2 language use need: Interact with people speaking many registers and dialects of the target language.

• SLA principle: Rich and varied input is necessary for second language acquisition to high levels.

• General learning principle: Complex information is best learned in a structured way at first, moving on to variety.

• Aptitude interaction: People with high aptitude can benefit from the variety sooner. People with low aptitude may need to learn only in the structured format.

43

How to choose which aptitudes and treatments to investigate?

• Information about the participants– Focus on abilities that will impact language use– Target abilities that differ most

• Information from science– Research‐based training techniques– SLA measures (in addition to proficiency)– Stronger predictions for real‐world performance, beyond the lab

47

Triangulation: Needs and constraints

48

Needs analysis• Use• Learning

Practical limits on interventions• time available• time for design &

implementation• instructor role• accessibility• expertise availability

Aptitude profiles• distribution• groupings?• aptitude predictions

testable by intervention

Needs Analysis

• Russian is a highly‐inflected language complex inflectional morphology

– Instruction bias: focus on vocabulary, don’t pay attention to the ’endings’

– Students develop a strategy to ignore inflectional morphology

49

Needs Analysis

• Grammatical case especially problematic: – Nominal inflectional morphology marks grammatical roles of nouns in

a sentence;

– Russian allows scrambling/limited constraints on word order inflections are the only reliable marking of grammatical function.

• What makes it worse:‐ existing L1 bias – English has strict SVO word‐order;‐ a universal processing strategy to interpret (or to assign) the role of

the agent to the first noun (for L2: VanPatten’s First Noun Processing Strategy, FNPS);

‐ Shallow Structure Processing hypothesis (Clahsen) – L2 semantic processing bias (e.g., animacy)

50

Pre‐testPerception TasksTask 1: Picture/audio match (CASES/WO) (auditory presentation)‐ to assess whether students can correctly interpret the meaning of SVO and OVS

sentences.

Мама читаетмальчику. Мальчик читает маме.

The Mom.NOM reads to the boy.ACC The boy.NOM reads to the Mom.ACC

51

Pre‐test

Task 2: Plausibility task (CASES/WO) (visual presentation)‐ similar to Task 1, but allows for inanimate objects to be subjects in grammatically

correct sentences. Мужчину держит ложка.

The man.ACC holds the spoon.NOM

Task 3: Auditory Grammaticality Judgment task (CASES)‐ assesses how sensitive learners are to case violations (e.g., two nouns both

marked for nominative case).

На завтрак бабушка варила кашу.

For breakfast the grandmother.NOM was cooking oatmeal.ACC.

52

Pre‐test

Production TasksTask 1: Picture description/translation (CASES/WO) (visual presentation)• uses English/Russian translation to force participants to produce both SVO and

OVS sentences with appropriate inflections.

Prompt: dog/girl/brush

Собаку причесывает девочка.The dog.ACC brushes the girl.NOM

Task 2: Elicited imitation (CASES/WO) (auditory presentation)‐ are able to correctly reproduce grammatical inflections.

53

Aptitudes• Implicit/Explicit learning designated as possibility

• Profiles (Hi‐LAB) confirmed split in class, though not fully crossed

• All students showed strength in Explicit Learning

• Half also showed strength in Implicit Learning

• Intervention design adjusted to take this into account

54

Intervention challenges

• “Case” too broad for narrow intervention• Instrumental case chosen• Within‐subjects design needed• Two target constructions for instrumental available for contrast: passive subjects and instruments

• Explicit intervention: what instructor already had planned

• Implicit intervention: “murder mystery” logic puzzles

55

Design details

• Study design:– All participants received both interventions– Implicit intervention only on instruments (not passive subjects)

– Matched training (implicit training for implicit aptitudes) should show greater benefit

– Training designed to be useful for all participants (just more useful for some)

– Existing instruction plans disturbed very little

56

Intervention details

• Critical design features:– Implicit: no overt instruction on or excessive attention towards instrumental case

– Meaning‐based: students needed to comprehend the statements for meaning to engage in the task

– Exposure: frequent use of weapons in sentence provided high‐density exposure to the instrumental case, using instruments

57

Murder mystery

• Like “passive Clue”

• Had to guess murderer, location, weapon, etc.

• Students listened to a number of statements (between 4 and 15)

• Used witness statements to eliminate possibilities

58

1. The mechanic.NOMki lled the cook.ACC Механик убил повара.2. The nurse.NOMki lled the cook.ACCwith a belt.INSTR Медсестра убила кого‐то ремнём.3. The nurse.NOM ki lled the victim.ACC in the school.PREPwith a spade.INSTR Медсестра убила кого‐то в школе лопатой.4. Someone.NOMki lled the professor.ACC in the school.PREP Кто‐то убил профессора в школе.5. The mechanic.NOMki lled the victim.ACCwith a spade.INSTR Механик убил кого‐то лопатой.6. The nurse.NOMki lled the professor.ACC Медсестра убила профессора.7. The mechanic .NOM ki lled the professor.ACCwith a belt.INSTR Механикубил профессораремнём.

Outcome measure details

• Auditory grammaticality judgment task (GJT)

• Taps into knowledge of students without asking specific questions on case

• Auditory chosen to match auditory intervention

• Quickly designed and implementable

• Provides specific measure for outcomes

59

Procedure

• Monday: Pre‐test GJT

• Tuesday, Wednesday, Thursday: Implicit instruction,3 x 20 minutes

• Friday: Mid‐test GJT (after the implicit instruction)

• Later on Friday: Explicit instruction, 1 hour• Tuesday (following week): Post‐test GJT (after the explicit instruction)

• 3 weeks later: Delayed post‐test GJT

60

Results

• Learners with explicit and implicit learning profiles take advantage of both types of instruction, and (tentatively) benefit more from the combination of the two.

• Learners with an explicit aptitude profile also show a gain in accuracy as a result of the explicit instruction.

• The information on the individual differences among learners predicts different learning outcomes as a result of methodological intervention

feasibility of effective task tailoring to individual strength for accelerated learning outcomes

61

RESEARCH PROBLEM

• Learners have incomplete knowledge of lexical items, and when they sound similar to other items, they confuse the two.

• ATI study will leverage participants’ aptitude profiles to determine whether instruction designed to match their aptitudes will promote increased mastery of confusable words as measured by accuracy & automaticity of perception and retrieval of words.

62

63

Study Design

• What constitutes exceptional ability?

• How many different aptitude profiles are there?

• At what level should the intervention be – approach, technique?

• Are there genetic correlates of language aptitude?

64 64

Solutions?

• Select people with talent, that is cognitive aptitude for language

• Tailor training to aptitude profiles, language learning needs, and workplace language needs

• Train the cognitive aptitude

• Train the job performance

• Coach the job performance

65

66 66

Thank you!Questions?

a new aptitude test -...

Documents