evaluating a tblt spanish immersion program
DESCRIPTION
Evaluating a TBLT Spanish immersion program. Greg Burwell – BPA – [email protected] Marta Gonzalez- Lloret –UHM – [email protected] Katie Nielson – UMD (CASL) – [email protected]. COLLOQUIUM: Evaluating task-based language programs. Overview. TBLT at the Border Patrol Academy (BPA). - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/1.jpg)
EVALUATING A TBLT SPANISH IMMERSION PROGRAM
COLLOQUIUM: Evaluating task-based language programs
Greg Burwell – BPA – [email protected]
Marta Gonzalez-Lloret –UHM – [email protected] Nielson – UMD (CASL) – [email protected]
![Page 2: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/2.jpg)
Overview
Background
Student assessment
Pilot study
Ordinate study
Student evaluation
Discussion
![Page 3: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/3.jpg)
TBLT at the Border Patrol Academy (BPA)
BPA Training Environment Spanish at the BPA (before and after) Students, instructors, and supervisors
sought practical training Developed in 2006-2007
Needs Analysis Genuine audio/video Extensive input from actual Border Patrol
agents
![Page 4: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/4.jpg)
TBLT at the Border Patrol Academy (BPA)
First implemented in December of 2007 First cohort of 15 students graduated in
March 2008 As of August 2009, 2,813 students have
completed the program To date, the program has started 180
classes with over 3,000 students 100% pass rate
![Page 5: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/5.jpg)
TBLT at the BPA- Curriculum 8-week program following Long’s (1985, 2005a) stages
for designing and implementing a TBLT program
7 target tasks
Several target task
types
Multiple pedagogic tasks
![Page 6: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/6.jpg)
TBLT at the BPA - CurriculumTarget Task Types:
Extract subject
Interview
subject
Determine
citizenship status
Inspect Vehicle
![Page 7: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/7.jpg)
TBLT at the BPA - Curriculum
Input (video, audio)
Output (guided)
Role-play tasks
COMPLEXITY
+ Lab Activities + La Chamba interactive video game
![Page 8: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/8.jpg)
TBLT at BPA- Assessment
•Computer-based language test •Performance-based assessment where trainees interact with native Spanish speaking role-players as they complete that module’s target task•One-on-one conference with instructor
Weekly formative
assessment:
•Critical scenarios with professional role-players•Students role-play key course tasks over two days•Criterion-referenced scoring•Each scenario scored by two instructors
End-of-course summative
assessment:
![Page 9: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/9.jpg)
TBLT at BPA - Assessment
![Page 10: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/10.jpg)
TBLT at BPA- Assessment Multiple versions of end-of-module
assessments Questions considered in each iteration:
Are these elements required to complete the task?
Which are critical? How do we assess language?
Global accuracy, fluency, and pronunciation ratings
Inclusion of “using appropriate Spanish” with each required task
Removal of any mention of language
![Page 11: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/11.jpg)
TBLT at BPA - Assessment
![Page 12: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/12.jpg)
TBLT at BPA - Assessment
![Page 13: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/13.jpg)
TBLT at BPA - Assessment
![Page 14: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/14.jpg)
TBLT at BPA - Assessment Instructor training
Video evaluation Role-play evaluation
Reliability Rubric modification
Perceived grade inflation Language versus task accomplishment Non-obligatory steps removed Average grade changed from 94.10% to
95.48%
![Page 15: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/15.jpg)
ParticipantsGrammar-based group (GB) (n=20)TBLT group (TBLT) (n=20)
Oral narrative based on picture storyHypotheses:
• TBLT will perform better on measures of fluency, lexical complexity and structural complexity
• GB will perform better on measures of grammatical accuracy
Pilot Study – Oral narration task
![Page 16: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/16.jpg)
Pilot study- cont Compared on:
Fluency [syllables/second] (Gilabert, 2004) Lexical complexity
[lexical words/words] (Robinson, 1995; Gilabert, 2004)
[1st time use of lexical word/words] Ratio of lexical/functional words (Gilabert, 2004)
Structural complexity [words/utterance] (Ortega, 1999)
Grammatical accuracy TLU of noun-modifier agreement (Ortega, 1999) TLU of noun-verb agreement
![Page 17: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/17.jpg)
Pilot study- Results - Fluency
GB TBLT0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00sy
llabl
es/
seco
nd
![Page 18: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/18.jpg)
Pilot study- Results – Lexical complexity
Lexical complexity 1 Lexical complexity 2 Lexical complexity 30.00
0.20
0.40
0.60
0.80
1.00
1.20
GBTBLT
Lexical words/words
1st use lexical w/words Ratio of lexical/ functional words
![Page 19: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/19.jpg)
Pilot study- Results- Structural complexity
GB TBLT0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
GBTBLT
#of w
ords
/ utte
ranc
e
![Page 20: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/20.jpg)
Pilot study- Results –Grammatical Accuracy
Grammatical accuracy 1 -(S-N agrement) Grammatical accuracy 2 (S-V agreement)0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
GBTBLT
TLU of noun-modifier agreement TLU of noun-verb agreement
![Page 21: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/21.jpg)
Pilot study - Discussion
Very small n-size due to difficulty
obtaining samples
Significant effect for fluency and
structural complexity (.05)
Students from TBLT course tend to produce more fluent and complex speech that is less grammatically accurate
but contains a greater variety of lexical items.
![Page 22: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/22.jpg)
ParticipantsFirst 256 TBLT students
Performance on computerized oral proficiency interviewHypotheses:
• TBLT will improve students’ overall proficiency• TBLT will affect overall proficiency regardless of
language abilities at the start of the course
Ordinate study
![Page 23: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/23.jpg)
Ordinate Study - continued Ordinate (Versant) Computerized Oral
Proficiency Exam Used at BPA as a placement test Scored between 20 – 80
Overall Sentence Mastery Vocabulary Fluency Pronunciation
Students took the test before and after the 8-week TBLT program
![Page 24: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/24.jpg)
Ordinate Study - Mean Scores
Overal
l
Sente
nce M
astery
Vocab
ulary
Fluen
cy
Pronu
nciati
on0
1020304050
Pre-testPost-test
![Page 25: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/25.jpg)
Ordinate Study – Change Scores
* Overall:Mean: 7.52SD: 5.42
* Sentence mastery:
Mean: 7.55SD: 8.21
* Vocabulary:Mean: 7.16 SD: 10.99
* Fluency:Mean: 5.86SD: 7.06
* Pronunciation:
Mean 3.53SD 4.92
All change scores
significant at .01
![Page 26: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/26.jpg)
Ordinate Study - Correlation
![Page 27: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/27.jpg)
Ordinate Study - Discussion
Hypothesis #1Overall proficiency will
improve as a result of TBLT
Yes: Post-test means improve an average of 7.5 points overall on an
unrelated, standardized proficiency test
![Page 28: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/28.jpg)
Students’ evaluations
Electronic (anonymous) surveys 2 Groups:
Pre-experienced
Current students of the TBLT Spanish
program n=4
1
ExperiencedCurrent BA Agents trained with TBLT Spanish program
n=14
![Page 29: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/29.jpg)
Students’ evaluation (pre-experienced)
![Page 30: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/30.jpg)
In general students liked the best… The work done with the role-players. The direct application of the class to their job.
“Role playing is very helpful. I feel that it helped me learn more spanish (sic) and become more comfortable speaking it.”
“I enjoyed the hands on experience the most. Being able to read and write is important when learning a new language, but being able to incorporate it into your daily duties is beneficial.”
“I have enjoyed that we get to have a lot of hands on in this program. Also helps a lot for us to be able to apply all that we have learned in basic and build apon (sic) basic.”
Students’ evaluation
![Page 31: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/31.jpg)
Students’ evaluations Students suggestions:
More conversational Spanish (not for the job)
More grammar in class More role-play work with Spanish speakers
![Page 32: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/32.jpg)
Agents’ evaluation (experienced)
The Spanish program (topics, activities, lab) was interesting.
I completed most of the lab exercises.
The topics I studied are now useful in the field.
The vocabulary & expressions I studied are now useful in the field.
The class exercises are now useful in the field.
I apply most of what I learned in class in the field.
The lab exercises helped me learn Spanish.
The Spanish lessons gave me confidence to use Spanish in the filed.
The Spanish lessons reflect well what happens in the filed.
I played the video game La chamba at least twice a month.
The video game helped me learn important Spanish words & expressions.
0.00 0.50 1.00 1.50 2.00 2.50 3.00
![Page 33: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/33.jpg)
Agents’ evaluations What they found most useful from the
Spanish course was: Commands and expressions Knowing the procedures and forms in
Spanish What they would have liked to learn:
More conversation (outside scenarios) More vocabulary Grammar and verb conjugation
![Page 34: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/34.jpg)
TBLT at BPA
Students are all
passing
Overall student
proficiency improves after the
TBLT course
Student speech is
more fluent and complex
than in the old course
What does this
mean?
Students find the course
practical
![Page 35: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/35.jpg)
•Difficulty of training instructors properly on methods and principles of TBLT•Instructors reluctant to adopt new approach•Lack of trained personnel to collect and analyze data•Lack of support for follow-up research
Difficulties with evaluating TBLT at the BPA
![Page 36: Evaluating a TBLT Spanish immersion program](https://reader036.vdocuments.mx/reader036/viewer/2022062323/5681642c550346895dd5f080/html5/thumbnails/36.jpg)
THANK YOUGRACIAS
MAHALO
Questions?