earli 2009, amsterdam roel van steensel (sco-kohnstamm instituut – uva) project 4: reading and...
Post on 20-Dec-2015
215 views
TRANSCRIPT
EARLI 2009, AmsterdamRoel van Steensel (SCO-Kohnstamm Instituut – UvA)
http://salsa.socsci.uva.nl
Project 4: Reading and writing development
Assessing the reading and writing proficiency of at-risk adolescents
in multilingual contexts
Roel van Steensel, Ron Oostdam, Amos van Gelderen and Mirjam Trapman
Study into Adolescent Literacy of Students At-risk
SALSA
2Roel van Steensel, EARLI 2009
Project SALSA
• Large groups of students in secondary education lag behind in literacy skillsOECD (2001)
• Relatively little is known about the factors that influence at-risk students’ literacy developmentCurtis (2002)
• Hence: Project SALSA
3Roel van Steensel, EARLI 2009
Project SALSA
Project 4Reading and writing
development
Environment Student
Project 1Literacy context
in schoolDe Milliano
Project 2Literacy context outside schoolVan Kruistum
Project 3Literacy-related
skills and attitudesTrapman
4Roel van Steensel, EARLI 2009
Project SALSA
• Central research questionWhich factors contribute to/impede the reading and writing development of at-risk adolescents in multilingual contexts?
• Sample– Lower tracks of prevocational secondary education (vmbo): basic
and middle-management program– Grades 7, 8 and 9 (age range: appr. 12-15)– 30% lowest scoring on CITO End of Primary School Test– Year 1 (2007/08): N=63– 32 monolinguals, 31 bilinguals (based on student questionnaire)
• International contextParallel projects in Canada en Switzerland
5Roel van Steensel, EARLI 2009
The test
• Newly developed literacy test:– Reading: text comprehension– Writing: text production
• This presentation: (construct) validity of the reading test
6Roel van Steensel, EARLI 2009
The test
• 9 reading tasks
• Text selection on the basis of:
– Text type: narrative, expository, instructive, argumentative
– Medium: school books, newspapers/magazines, internet, ‘official documents’ (house rules)
– Subject: context-neutral, relevant for the target group
7Roel van Steensel, EARLI 2009
The test
Narrative Expository Instructive Argumentative
School books NAR1 EXP1
Newspapers/ magazines
NAR2 EXP2
EXP3
ARG1
ARG2
Official documents
INS1
Internet INS2
8Roel van Steensel, EARLI 2009
The test
• Item construction on the basis of hypothesized subskills:
– Retrieving: retrieving relevant details from the text
– Interpreting: making inferences from shorter passages in the text (e.g. cause-effect relationships)
– Reflecting: reflecting on larger passages/the text as a whole
Davis (1968); Rosenshine (1980); Goldman & Durán (1988); Alderson (2000); OECD (2003); Song
(2008); Cerdán et al. (2009)
9Roel van Steensel, EARLI 2009
The test
Text comprehension
Local level/microstructure
Retrieving
Interpreting
Global level/macrostructure
Reflecting
10Roel van Steensel, EARLI 2009
Validity study
• Messick (1989) - aspects of (construct) validity:
– Construct (ir-)relevance: does the test only reflect aspects of the construct to be measured? Or are the test results influenced by variables irrelevant to the construct?
– Internal structure: can (theoretical) expectations about relations between items be confirmed?
– External structure: can (theoretical) expectations about relations with other measures be confirmed?
11Roel van Steensel, EARLI 2009
Validity study
• Test administered to 200 students in the lower tracks of prevocational secondary education (school year 2007-08)
• Additional tests administered to a subsample of 63 students
12Roel van Steensel, EARLI 2009
Construct (ir-)relevance
• Assumption: difficulty of the tasks is determined by difficulty of the texts
• Comparison between:– Text difficulty measured by Flesch/Douma readability
index– Task difficulty indicated by mean task score as % of the
maximum task score
13Roel van Steensel, EARLI 2009
Construct (ir-)relevance
Douma Rank Rank Mean/max*100%
NAR2 89.7 1 1 79
NAR1 83.8 2 3 65
EXP2 70.8 3 5 60
INS2 67.9 4 9 51
EXP1 62.2 5 5 60
ARG1 57.2 6 4 64
ARG2 56.3 7 7 57
EXP3 49.7 8 8 52
INS1 48.3 9 2 68
14Roel van Steensel, EARLI 2009
Construct (ir-)relevance
• INS1 en INS2 are both discontinuous texts:– INS1 is a set of house rules
discontinuity makes the task easier
– INS2 is a website
discontinuity makes the task more difficult
15Roel van Steensel, EARLI 2009
Internal structure
Text comprehension
Local level/microstructure
Retrieving
Interpreting
Global level/macrostructure
Reflecting
16Roel van Steensel, EARLI 2009
Internal structure
• Do the data show a:– 1-factor model?– 3-factor model?– 2-factor model?
• Confirmatory factor analysis using Structural Equation Modeling (EQS 6.1)
• Individual test items were combined into 6 parcels (RTR1, RTR2, INT1, INT2, RFL1, RFL2)
17Roel van Steensel, EARLI 2009
Internal structure
1-factor model
Text comprehension
RTR1
RTR2
INT1
INT2
RFL1
RFL2
Excellent fit! χ2=6.27, df=9, p=.713, N=175, SRMR=.026, RMSEA=.000, 90% confidence interval=.000-.060.
18Roel van Steensel, EARLI 2009
Internal structure
3-factor model
Text comprehension
RTR1
RTR2
INT1
INT2
RFL1
RFL2No improvement!Difference score 1-factor/3-factor model: χ2=3.36, df=3, p>.05.
retrieving
interpreting
reflecting
19Roel van Steensel, EARLI 2009
Internal structure
2-factor model
Text comprehension
RTR1
RTR2
INT1
INT2
RFL1
RFL2No improvement!Difference score 1-factor/2-factormodel: χ2=2.41, df=1, p>.05.
local level skills
global level skills
20Roel van Steensel, EARLI 2009
External structure
• Assumption: text comprehension is facilitated by:
– Well-developed decoding skills
– A rich vocabulary
– Well-developed morpho-syntactic skills
– A good working memory
– (Metacognitive) knowledge of reading and writing
• And: text comprehension is associated with IQ
21Roel van Steensel, EARLI 2009
External structure
Variable Correlation with SALSA reading score (N=63)
Decoding skills .22
Receptive vocabulary .59**
Morpho-syntactic knowledge .66**
Working memory .42**
Metacognitive knowledge .41**
Nonverbal IQ .36**
** p<.01
22Roel van Steensel, EARLI 2009
Conclusions: support for validity?
• Construct (ir-)relevance:
Task difficulty largely corresponds with text difficulty– Exceptions (INS1, INS2) are explainable
• Internal structure:
No support for the presence of subskills
Data do show that the test measures one construct
• External structure:
Substantial correlations with other skills– Exception (decoding skills) is explainable
23Roel van Steensel, EARLI 2009
• Thank you for your attention!
• Contact?– [email protected]– Or visit our website: http://salsa.socsci.uva.nl/