eurocall 2006 – granada an empirical study on the influence of feedback on learning performance...
TRANSCRIPT
Eurocall 2006 – Granada
An empirical study on the influence of feedback on learning performance
for second language learners of French using a web-based CALL environment:
Experimental design & results of the preliminary study
Barbara Louwagie – Piet Desmet
Eurocall 2006 – Granada
Objectives
• The development of (automatic) feedback is a central point in CALL. =>The general purpose of our study: feedback = effective supplementary opportunity for learning?
• Aim of the preliminary study: optimalization of the design
• => This presentation is about the search for an adequate design
Eurocall 2006 – Granada
Structure
• Definition of the object of study
• Preliminary study
• New design
Eurocall 2006 – Granada
Definition of feedback
“In the context of teaching in general, feedback is information that is given to the learner about his or her performance of a learning task, usually with the objective of improving this performance.”
(Ur 2003)
Eurocall 2006 – Granada
Feedback?
feedback
quantitative
qualitative(correctiv
e)
evaluation
summative
formative
Eurocall 2006 – Granada
• Aim: to create learning opportunities
• Consists of– identification of errors committed– explanation of errors (cognitive strategies)– suggestions to correct the errors
(Janssens ea 2003)
Corrective feedback in formative Corrective feedback in formative evaluationevaluation
Eurocall 2006 – Granada
Advantages FB in CALL
• immediate (Heift 2001)
• individualized (Kornum 1989)
• objective (Kornum 1989)
• …
Eurocall 2006 – Granada
• Verify if the feedback provided in the learning environment used at our university is effective
• Definition of ‘efficacy’ or ‘effectiveness’“The power to produce effects or intended results. This suggests a strong causal relationship between an intervention, such as the use of a particular item of technology in a learning situation and a discernible change in the learning process, the learning climate or the learning achievement.”
(Felix 2005)
Eurocall 2006 – Granada
Preliminary study: structure
• Aim• Design• Subjects• Procedure• Learning Environment (Idioma-tic)• Statistical analyses• Results• Evaluation of the preliminary study
Eurocall 2006 – Granada
Preliminary study: aim
• To verify the effect of different kinds of feedback on the score obtained for the final test
• To test the design of the study
Eurocall 2006 – Granada
5 feedback configurations, schematical representation
no feedback feedback
closed activity
closed activity half-open activity
general & specific FB
general, specific & error-specific FB
intermediary FB intermediary, general & specific FB
Eurocall 2006 – Granada
• 5 themes, 3 components:– Speech acts
• Telephone conversation
– Vocabulary• Business vocabulary• Quantitative vocabulary
– Grammar• Conjunctions• Grammatical orthography
• Different feedback conditions for each of these themes
Eurocall 2006 – Granada
Type of FB as ‘within-subjects’ factor
Grammar: grammatical orthography
Grammar: Conjunctions
Speech acts: Telephone conversation
Vocabulary: The enterprise
Vocabulary: Quantitative vocabulary
Group 1: Series 1
ClosedSpec & Gen FB ESF
Half-OpenIntermediary FB
Half-OpenIntermediary FBSpec & Gen FB
Closed ClosedSpec & Gen FB
Group 2: Series 2
ClosedSpec & Gen FB
ClosedSpec & Gen FB ESF
Half-OpenIntermediary FB
Half-OpenIntermediary FBSpec & Gen FB
Closed
Group 3: Series 3
Closed ClosedSpec & Gen FB
ClosedSpec & Gen FB ESF
Half-OpenIntermediary FB
Half-OpenIntermediary FBSpec & Gen FB
Group 4: Series 4
Half-OpenIntermediary FBSpec & Gen FB
Closed ClosedSpec & Gen FB
ClosedSpec & Gen FB ESF
Half-OpenIntermediary FB
Group 5: Series 5
Half-OpenIntermediary FB
Half-OpenIntermediary FBSpec & Gen FB
Closed ClosedSpec & Gen FB
ClosedSpec & Gen FB ESF
Eurocall 2006 – Granada
+ other variables
• Personal variables (sex, level of proficiency in French, learning style (Kolb-test), educational background, …)
• Situational variables (subject of the exercises, number of tries, time spent on a certain exercise, …)
Eurocall 2006 – Granada
Subjects
• n = 107• Undergraduate students in applied economic
science & commercial engineering• Learners of French as a second language
• 5 groups (n ~ 20)• Stratification solely based on current branch of
study, otherwise at random
Eurocall 2006 – Granada
Procedure
Duration: 9 weeksDuration: 9 weeksWeek 1• (short demo of Idioma-tic)• Orientation test itemset 1, 50 MC-questionsWeek 3• Practice session 1 (1 ½ hour) itemset A1, A2, A3, A4, A5 (± 30 questions)Week 5• Practice session 2 (1 ½ hour) itemset B1, B2, B3, B4, B5 (± 30 questions)Week 7• Practice session 3 (1 ½ hour) itemset C1, C2, C3, C4, C5 (± 30 questions)Week 9• Evaluation test itemset 2 (5*(3+3) items)
Eurocall 2006 – Granada
Learning environment
• Authoring tool IDIOMA-TIC
• Tracking and logging system
Overview of activities (± 2000)
Grammar: Grammatical orthography
Grammar: Conjunctions
Speech acts: Telephone
conversation
Vocabulary: The Enterprise
Vocabulary: Quantitative vocabulary
Total
Group 1: Series 1
Closed, FB, ESF Half-open Halfopen, FB Closed Closed, FB
427
M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3
50 30 50 30 30 25 30 25 20 20 20 27 25 20 25
Group 2: Series 2
Closed, FB Closed, FB, ESF Half-open Half-open, FB Closed
430
M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3
50 30 50 30 29 26 30 25 20 20 20 30 25 20 25
Group 3: Series 3
Closed Closed, FB Closed, FB, ESF Half-open Half-open, FB
424
M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3
50 30 50 30 29 26 22 26 24 20 20 30 25 18 24
Group 4: Series 4
Half-open, FB Closed Closed, FB Closed, FB, ESF Half-open
399
M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3
50 28 30 30 29 26 22 26 24 20 20 27 25 18 24
Group 5: Series 5
Half-open Half-open, FB Closed Closed, FB Closed, FB, ESF
402
M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3 M1 M2 M3
50 28 30 30 30 25 22 26 24 20 20 27 25 20 25
TOTAL208
2
Eurocall 2006 – Granada
Statistical analyses*
• Cross-tabulations (tabulate procedure)
• Correlations (corr procedure)
• Multi-level analysis (REML-algorithm**)
* Many thanks to M. Vandewaetere
** residual restricted maximum likelihood
Eurocall 2006 – Granada
Results: Descriptive statistics
Cross-tabulation (tabulate procedure)Distribution for sex, high school curriculum and current curriculum
sex
male female
High school curriculum 59 48
Maths-sciences 56 43
Sciences 3 3
other 0 2
Current curriculum 59 48
Applied Economic Science
39 35
Commercial engineering 20 13
Eurocall 2006 – Granada
• Distribution for exercise series and current curriculum
Series
Series 1
Series 2
Series 3
Series 4
Series 5
total
Commercial Engineering
6 6 7 8 6 33
Applied Economic Science
13 15 16 14 16 74
Total 19 21 23 22 22 107
Eurocall 2006 – Granada
Correlations (corr procedure)• Indication of influence of FB,
but just below statistical significance• Some obvious significant results:
– Correlation between the number of exercises made and the final score ( = 0.13, α = 0.0036)
– …
Analytic statistics
Eurocall 2006 – Granada
Multi-level analysis I
Multiple themes per subject are measured (5). Themes are clustered for any single student.
• Feedback has a significant effect (F(4,415) = 2.99, p < 0.05 )
• Theme has a significant effect (F(4,415) = 69.84, p < 0.0001)
students (107)
… … … …
themes (5)
individual and environmental variables
Eurocall 2006 – Granada
Additional remarks
• The foregoing analysis is not complete• There are repeated measuring moments (3) for
any single student.The scores obtained during these measuring moments for a single subject will correlate more strongly than the scores of different subjects on a single measuring moment.
• The following analysis takes into account the repeated mesures design.
Eurocall 2006 – Granada
Multi-level analysis IIMultiple themes per subject are measured (5); moreover, each theme is measured three times. Themes are clustered for any single moment, and the moments are clustered for any single student.
• There is a difference between the three measuring moments (F(2,202) = 68.55, p < 0.0001)
• There is a difference between the types of feedback (F(4,424) = 1456.89, p < 0.0001)
• There is a difference between the themes(F(4,424) = 114.39, p < 0.0001)
students (107)
… … … …
moments (3)
… …
themes (5)
individual and environmental variables
Eurocall 2006 – Granada
• These significant main effects (previous slide) can, however, not be interpreted unequivocally, – because the two-way interaction terms are significant
• feedback*moment, F(8,768) = 15.52, p < 0.0001 • theme*moment, F(8,720) = 9.17, p < 0.0001
– Id. for three-way interaction • thema*feedback*moment: F(48,720) = 8.97, p < 0.0001
• Nevertheless, first indication of influence of FB• => Optimize the design to obtain significant
results
Eurocall 2006 – Granada
Evaluation of the preliminary study
• Some practical problems– Over-parametrization (1000 cells per subject) =>
many interaction effects– Only one observation per cell: 2 themes can not be compared– Stratification solely on the basis of current curriculum
• Personal variables– Maintain these variables– Level of proficiency in French: Orientation test: OK– Learning style test – Kolb test: not validated psychometrically
speaking, the results can not be validated
Eurocall 2006 – Granada
• Exercise series– Re-usable – Pay attention to differences in the level of difficulty
between themes (grammatical orthography is comparatively easier than the four other themes)
• Authoring tool:– Stable learning environment– Tracking & logging: no scoring provided for half-open
assignments
Eurocall 2006 – Granada
New design
• Add qualitative study (questionnaires)• Feedback configurations: maintain 5 types• Themes: simplification: 5 => 3 themes
– Speech acts: telephone conversation– Vocabulary: business vocabulary– Grammar: conjunctions
• Other variables: maintain and extend:– Self-evaluation as to knowledge of and attitude towards French
(motivation) – Familiarity with the on-line tool– …
Eurocall 2006 – Granada
Type of FB as a ‘between-subject’ factor
Grammar: Conjunctions
Vocabulary: The enterprise
Speech acts: Telephone conversation
Group 1: Series 1
ClosedSpec & Gen FB
ESF
Group 2: Series 2
ClosedSpec & Gen FB
Group 3: Series 3
Closed
Group 4: Series 4
Half-OpenIntermediary FBSpec & Gen FB
Group 5: Series 5
Half-OpenIntermediary FB
Eurocall 2006 – Granada
Procedure
• Orientation test itemset 1, MC questions– Exercise moment 1 itemset A1, A2, A3, A4, A5
• evaluation 1 itemset 2 (20 closed assignments, 20 half-open assignments)
– Exercise moment 2 itemset B1, B2, B3, B4, B5
• evaluation 2 itemset 3 (different from 2)– Exercise moment 3 itemset C1, C2, C3, C4, C5
• evaluation 3 itemset 4 (different from 2 & 3)
Eurocall 2006 – Granada
• Extended stratification: – current curriculum,– sex, – score for orientation test, – educational background, …
• Each time trial run• Items at random, for 2 reasons:
– Concentration effect– Time lapse
Eurocall 2006 – Granada
More information
Presentation: http://www.kuleuven-kortrijk.be/ALT/
Barbara LouwagieK.U.Leuven Campus [email protected]
Piet DesmetProfessor of French and Applied Linguistics
K.U.Leuven (Campus Kortrijk)[email protected]
www.ling.arts.kuleuven.ac.be/franling_e/pdesmet