university of groningen fostering the learning potential ... · contents chapter 1 chapter 2...

University of Groningen

Fostering the learning potential of at-risk students in the classroomTiekstra, Marlous

IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.

Document VersionPublisher's PDF, also known as Version of record

Publication date:2016

Link to publication in University of Groningen/UMCG research database

Citation for published version (APA):Tiekstra, M. (2016). Fostering the learning potential of at-risk students in the classroom: Studies into theconsequential validity of dynamic assessment. University of Groningen.

CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).

Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.

Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.

Download date: 27-03-2021

https://research.rug.nl/en/publications/fostering-the-learning-potential-of-atrisk-students-in-the-classroom(9b054195-0e2d-4390-b186-bb6fe12e5218).html

Fostering the learning potential of at-risk students in the classroomStudies into the consequential validity of dynamic assessment

Marlous Tiekstra

© 2016, Marlous Tiekstra

All rights reserved. No part of this publication may be reproduced, stored in a retrieval

system of any nature or transmitted in any form or by any means, electronic, mechanical,

photocopying, recording or otherwise, without prior permission of the author, or when

appropriate, the publishers of the papers.

ISBN 978-90-367-8577-8

ISBN E-version 978-90-367-8576-1

NUR-code 848

Cover photo photo by Marlous Tiekstra, vignobles de Belvèze-du-Razès

Graphic design myra nijman concept & design | [email protected]

Printed by CPI - Koninklijke Wöhrmann

Fostering the learning potential of at-risk students in the classroomStudies into the consequential validity of dynamic assessment

Proefschrift

ter verkrijging van de graad van doctor aan de Rijksuniversiteit Groningen

op gezag van de rector magnificus prof. dr. E. Sterken

en volgens besluit van het College voor Promoties.

De openbare verdediging zal plaatsvinden op

donderdag 11 februari 2016 om 14.30 uur

door

Marlous Tiekstra

geboren op 6 september 1985te Delft

Promotores Prof. dr. A.E.M.G. Minnaert Prof. dr. M.G.P. Hessels Beoordelingscommissie Prof. dr. A.J.J.M. Ruijssenaars Prof. dr. W.C.M. Resing Prof. dr. K.H. Wiedl

To

Gjalt Christiaan Tiekstra

a very wise and honourable man who is deeply missed

thank you for your inspiration

Contents

Chapter 1

Chapter 2

Chapter 3

Chapter 4

Chapter 5

Chapter 6

Chapter 7

References

Samenvatting [summary in Dutch]

Dankwoord [acknowledgements in Dutch]

About the author

09

25

57

81

99

119

141

155

171

181

187

General introduction

A review scrutinising the consequential validity of

dynamic assessment

Can didactic resistance be measured by using the

Response-to-Instruction model? A pilot study

At-risk students and the role of implicit theories of

intelligence in educational professionals’ actions

Voices from practice: when is the gap between diagnosis

and intervention apparent?

The next step in bridging the gap: a pilot study to

enhance teacher’s skills with at-risk children

General discussion

“On trace la route,on s’regarde pasOn change de route sans savoir pourquoi”(C. Maé)

Chapter 1General Introduction

This chapter is partly based on: Tiekstra, M., Hessels, M. G. P., & Minnaert, A. E. M. G. (2009).

Learning capacity in adolescents with mild intellectual disabilities. Psychological Reports, 105, 804-814. doi:10.2466/PR0.105.3.804-814

CHAPTER 1

10

GENERAL INTRODUCTION

11

1

Yamina, a ten year-old girl of Moroccan origin, is in third grade with 27 classmates. Lately, Yamina

struggles to keep up with her peers. In the classroom she appears easily distracted and during breaks she

is alone more often. Her school results are poor since the start of the academic year already, especially

in reading. Since five months Yamina is supported by the remedial teacher. Twice a week she receives

additional reading time outside the classroom. Due to suspicion of dyslexia she has been recently assessed

by a school psychologist. Results of this assessment revealed an IQ of 96, and low scores on both

reading and arithmetic achievement tests. Yamina was not identified as dyslectic. The remedial teacher

noticed that Yamina grasps some parts of his instruction very well, especially in arithmetic, but much

less in other domains. Meanwhile, the regular classroom teacher does not know how to support Yamina

with her reading. He and the remedial teacher decide to continue the general didactical instruction as

modelled in the class’s reading program, since it worked for other students previously. Besides, they decide

that Yamina should receive more time to finish her tasks. As a result, she often stays inside the classroom

during breaks to finish her tasks.

This is an example of daily educational practice. The questions that arise are: Is this

the right way to remediate Yamina’s problems? How to develop the student’s learning

potential? What does the general classroom teacher think about the learning potential

of this student? How does this relate to the general classroom instruction? How should

‘care’ around this student be effectuated? The present thesis is grounded in these

questions.

The rationale of this thesis is that every student can learn. Some students learn

easier than others, but every student has potential to learn. To optimise capacities

in learning every student, either a high or low achiever, needs assistance in learning

from a more experienced significant other. This is underlined in literature, as well as

in the case of Yamina: it is recognised that she needs assistance from the remedial

teacher. Vygotksy (1978) emphasised the importance of this significant other, or so-

called mediator in the terminology of Feuerstein, Rand, and Hoffman (1979), in the

learning processes of students. This significant other (mediator), for example a teacher

or a parent, both organises information by filtering or structuring, and gives meaning to

CHAPTER 1

12

the new information so that it can be integrated in the learning processes of students.

Many other researchers underlined the importance of significant others to the learning

outcomes of students. The quality of teacher-student interpersonal relationships has

been shown to substantially contribute to motivational, and consequently, learning

outcomes, especially in at-risk populations (see e.g., Den Brok, Van Tartwijk, Wubbels,

& Veldman, 2010).

Learning occurs when the mediator operates in the Zone of Proximal Development

(ZPD) of a particular student. This ZPD, the difference between the current independent

achievement of the student and the performances a student may show when tailored

help is provided (see also Lidz, 1995), is an important aspect in the optimisation of

learning potential. Consequently, Vygotsky’s conceptualisation of the ZPD has several

implications for psycho-educational practice, such as assessment, and the diagnosis of

learning disabilities, next to intervention, and general classroom instruction.

Assessment and diagnosisIn the case of ongoing difficulties in learning, as in the case of Yamina, students are often

subjected to an assessment procedure to investigate the causes of their difficulties. Such

an assessment can be effectuated in different ways and for different purposes, e.g., it can

be formative or summative, focused on general abilities or specific domains, and it can

be either for classification purposes or rather focused on the special educational needs

of the student. Moreover, for the diagnosis of a specific learning disorder an estimation

of the student’s intelligence is often requested (see e.g., the DSM-5). Assessment is,

however, subject to an ongoing debate about the test administration itself: it can be

administered either statically, or dynamically.

Static assessment. ‘Static’ refers to the fact that only scant interaction between the

examiner and the examinee during testing is allowed: the examiner asks the questions

following the standardised protocol and the child has to respond. No additional

information, explanation, or training is provided, although sometimes a question may


13

1

be repeated or the child may be asked to “say a bit more”. The underlying assumption

with static tests is that in order to obtain reliable test scores, the psychological trait

that is to be measured must be stable over time and, therefore, no training or teaching

can be established during testing (Hessels-Schlatter & Hessels, 2009). The results of

the assessment will be the product of previous learning, but will not reveal the child’s

capacity to learn new things. Also, no information about learning processes is provided

by such tests, although some static intelligence tests do try to focus on underlying

neuropsychological processes (Daniel, 1997). The examiner, however, still has a relatively

passive role, since there is no explicit teaching incorporated in these tests.

Dynamic assessment. As defined by Thorndike (1924) intelligence is the ability to

learn. Therefore, in dynamic assessment, examinees are offered an opportunity to learn

during testing. This opportunity to learn, referred to as learning phase, is a moment

during test administration in which the examiner provides hints and feedback that

address processes needed to solve the task in the test. After the learning phase, items

are administered to measure the extent to which the examinee has profited from the

learning phase. Outcomes, then, will give an indication of the examinee’s learning

potential. Since a learning phase is part of the administration of the test, this is the

main difference to static testing, in which only products of former learning experiences

can be measured.

In a previous study (Tiekstra, Hessels, & Minnaert, 2009), we investigated the differences

between static and dynamic assessment of intelligence in a specific population. In static

tests it is implicitly assumed that all children have had the same opportunities to learn

and have had the same learning experiences, and that differences in test scores reflect

differences in children’s capacity to profit from their learning experiences. However,

since learning experiences vary from one child to the other, many other authors argue

that intelligence tests only deliver information about products of former learning

opportunities, but do not provide any information about current learning opportunities.

With regard to specific populations, such as people with intellectual deficiencies, the

CHAPTER 1

14

use of these tests might lead to a serious underestimation of their learning capacities

(Hessels & Tiekstra, 2010). Moreover, these tests do not specify which cognitive processes

have been employed and do not give useful information for the development of tailored

cognitive interventions (Bosma & Resing, 2006).

The results of our study in adolescents with mild intellectual disabilities showed that an

IQ test predicted 8,8% of the variance in a novel school related learning task, whereas

the dynamic test of learning potential improved this prediction to 48%. This added

value was highly significant (ΔR2=.40; ΔF1,20 =15.28, p=.001), with effect sizes for IQ

η2 = .24 and for the learning potential test η2 = .43. The learning potential test, thus,

provided a more accurate prediction of future learning outcomes compared to an IQ

measure. More specifically, whereas static tests evaluated these students’ capacities in the

same range (i.e., low IQ), the learning potential test showed the ability to differentiate

substantially between students’ capacities to learn (Tiekstra, et al., 2009).

InterventionDynamic assessment is scarcely implemented in educational practice (Deutsch & Reynolds,

2000; Elliott, 2003). While being studied extensively for more than fifty years, dynamic tests,

still, come off worst compared to traditional intelligence tests with regard to their application

in practice (Grigorenko & Sternberg, 1998). IQ tests receive great attention and appreciation

due to their designated necessity for diagnosing learning disabilities, and subsequently, for

financial resources for children with special educational needs. Consequently, the outcomes

of these tests are used for setting up Individualised Educational Plans (IEP). However, IQ

test outcomes lack specific information that could be used to adapt instruction (Freeman &

Miller, 2001; Ysseldyke, 2001). In the present case, Yamina gets support from the remedial

teacher, but it might not be the support she actually needs to improve her reading skills.

The more tailored an intervention is, the more effect it will have. The outcomes of the

assessment procedure revealed that Yamina had an IQ of 96 and low scores on school

achievement tests. This information did not provide useful clues for the construction of

an intervention or for an adaptation of the present classroom instruction. Therefore, the


15

1

teacher and remedial teacher had difficulty in providing the appropriate help to Yamina.

This is common for educational practice: actions in the classroom are often not based on the

IEP of students. A gap remains between diagnosis and actions in- or outside the classroom

(Shapiro & Kratochwill, 2000).

In the new edition of the Diagnostic and Statistical Manual of Mental Disorders (5th ed.;

DSM-5; American Psychiatric Association, 2013), used by practitioners as a guideline for

diagnosis of disabilities, it is highlighted that ‘targeted intervention’ is the ultimate aim of an

accurate diagnosis. Since dynamic assessment has proved to be a better estimator of future

learning outcomes (e.g., Elleman, Compton, Fuchs, Fuchs, & Bouton, 2011; Swanson,

2011; Swanson & Howard, 2005), especially in populations in which learning problems are

eminent (e.g., Beckmann, 2006; Tiekstra et al., 2009), and could provide useful information

for classroom practice (e.g., Bosma & Resing, 2006; Daniel, 1997), one should wonder why

these tests are not used in educational practice. A decade ago, Elliott (2003) provided several

reasons for this, such as lengthy test administrations and the focus on rather small ranges

of age groups. Meanwhile, attempts have been made to overcome these issues, but these

have not yet led to more use of dynamic assessment in educational practice to date. Elliott

(2003) underlined the need for studies that investigate the potential of dynamic assessment

to bridge the gap between diagnosis and interventions. This can be considered a prerequisite

for applicability in educational practice.

Although information from dynamic test reports is appreciated by teachers (Bosma,

Hessels, & Resing, 2012; Freeman & Miller, 2001), the question remains to what extent

principles from these reports are translated into the classroom practice. Here, too, the

gap between assessment information and classroom practices remains present (Haywood,

2012). Therefore, from our point of view, more aspects play a role when providing students

with targeted intervention: next to adequate (dynamic) test results, skills of educational

professionals need to be developed or improved in order to fully optimise students’ learning

potential in educational practice.

CHAPTER 1

16

General classroom instructionAlready underlined in 1987 by Delclos, Burns, and Kulewicz, dynamic assessment

positively influences the view of teachers about a child’s ability due to the teaching

component (i.e., the learning phase) during testing. This is especially important

in the case of at-risk students. At-risk students are students that are susceptible for

developing learning delays and/or problems due to their risk for non-optimal learning

opportunities and experiences. Generally, this population is composed of immigrant

children, children that have been brought up in low-income families or families with

low socioeconomic status, children with severe learning difficulties, communication

disorders, and/or behavioural or emotional problems, or children with intellectual

disabilities. These children neither had equal learning opportunities, nor the same

learning experiences as average indigenous, middle class children without any learning

difficulties. At-risk students are often confronted with lower educational expectations,

and consequently, lower educational efforts of educators. Eventually this will lead to

a self-fulfilling prophecy: the student is not inclined to learn (Haywood, Tzuriel, &

Vaught, 1992; Sternberg, 2004).

General classroom instruction is oriented towards average students, as illustrated by

the case of Yamina: she received no specific adapted instructions, since the method

used worked well in other students. However, it is questionable whether this is the best

solution for the specific problems of Yamina. Actually, it relates to a bigger problem:

how to differentiate general classroom instruction in such way that each individual

student benefits from it?

To resolve this issue, the Response-to-Instruction/Intervention, or so-called RTI-model,

has been developed in the United States. This organisational model requires regular

progress monitoring of each student in the classroom, which in turn allows for regular

adaptation of teacher’s instruction. The core assumption of the RTI model is that every

child reacts differently to whole-class teaching. The extent to which each student is

sensitive to instruction can be estimated by regular progress monitoring of each student.


17

1

Progress monitoring is carried out regularly (daily, weekly, or fortnightly) with short

curriculum-related tests, which can be developed by the teacher. The model consists of

three levels of instruction-intensity (Fuchs & Fuchs, 2006). After approximately six weeks

of general classroom instruction (tier one), progress monitoring shows which students

did not progress (compared to the mean progress of the group). These students, then,

will be offered intensified instruction in small groups several times a week (tier two). If

extra instruction does not provoke enough progress, as demonstrated by the regular

progress monitoring, an individualised route is set up for these students (tier three).

This model is not (yet) implemented in European countries, whereas in the United

States it is even used for the diagnosis of learning disabilities. Several authors (e.g.,

Kavale, 2005; Reynolds & Shaywitz, 2009), however, criticised the model since its

effectiveness has not yet been demonstrated empirically. Interestingly, in theory, the

model acknowledges the role of the teacher in providing quality teaching. Therefore, it

could be used in the operationalisation of the concept of ‘didactic resistance’, a criterion

used for the diagnosis of learning disabilities. The concept refers to the responsiveness

of the student to teaching. Thus, the quality and amount of teaching should be carefully

documented. Due to the progress monitoring the teacher becomes aware of the effects of

his/her own teaching. Moreover, the model accounts for an opportunity to differentiate

between different needs of students. In the general classroom, students with IEPs could

receive as much quality teaching as regular students, if the model is well-implemented.

Significant othersThe theory of learning by Vygotsky, and his conceptualisation of the ZPD in particular,

underlined the importance of significant others. As mentioned before, significant others

play an important role in students’ learning processes: exploration of the student’s ZPD,

quality instruction in the ZPD, and motivational techniques combined can be expected

to lead to the student’s responsiveness to instruction. In the case of at-risk children this

implies the following:

CHAPTER 1

18

1. The assessment of capacities should be done dynamically to explore the

child’s ZPD;

2. The activities of the assessor should be highlighted during test interpretation;

3. The role of the educator should be highlighted in diagnosing disabilities (i.e.

didactic resistance);

4. The roles of both the educator and assessor are critical when aiming at

tailored instruction and targeted interventions.

Significant others, thus, play an important role in the ‘care’ processes concerning at-risk

children when optimising their learning potential. Therefore, the focus of this thesis will

be on these significant others in the educational context, involving teachers and support

professionals, such as remedial teachers and school psychologists.

As demonstrated by many (e.g., Azjen, 2005; Pajares, 1992), each of these significant

others have their own beliefs about learning, teaching, their self, and these beliefs affect

people’s actions. This means that the actions of significant others in the issues mentioned

above are influenced by specific beliefs. Especially in the case of at-risk children this is

very relevant. With respect to Yamina, the teacher might have lowered his educational

expectations about her due to the outcomes of the assessment, leading to less effort in

teaching Yamina, since the results of the IQ test confirmed his expectations about her

intelligence. Why, then, put the effort in providing her with different instruction from

the rest of her classmates? This might be in contrast with the beliefs of the remedial

teacher who might think that Yamina is able to learn new things, since he noticed

progress in Yamina’s response to his individualised instruction.

This example illustrates the importance of one’s implicit theory of intelligence. Every

one of us has an idea about the malleability of intelligence, nonetheless, we are often

unaware of this idea. As demonstrated by Dweck and colleagues (see e.g., Dweck, Chiu,

& Hong, 1995) one’s implicit theory of intelligence greatly influences one’s actions and

own learning. Especially in the case of at-risk students, implicit theories of intelligence


19

1

could play a substantial role. Moreover, the self-efficacy of educators, i.e., their beliefs

about the influence they have on their students’ learning (Bandura, 1997; Gibbs &

Powell, 2012; Imants & De Brabander, 1996) has an impact on teacher’s behaviour

in the classroom (Tschannen-Moran & Woolfolk Hoy, 2001). It is for these reasons,

and in line with the concluding remarks of Ghesquière (2002) in the book on learning

potential assessment by Van der Aalsvoort, Resing, and Ruijssenaars (2002), that in the

present thesis we consider the influence of beliefs of educational professionals, being

the significant others in the learning processes of at-risk students, of importance.

Consequential validityIn the case of Yamina, the static assessment did not provide any useful clues for specific

intervention. As a result, the teacher as well as the remedial teacher, did not know which

instruction or intervention to provide, and therefore, continued to follow the proposed

instruction as provided by the conventional reading method. Moreover, we could state

that the school psychologist might have provided more support to the teacher and

remedial teacher. The gap between diagnosis and intervention is very visible. The focus

of this thesis is to provide information about bridging this gap.

Actions should be based upon outcomes of assessment. Consequences of, or actions

that follow the assessment procedure fall within the concept of consequential validity

of assessment. Messick (1995) defined this type of validity as “evidence and rationales

for evaluating the intended and unintended consequences of score interpretation

and use in both the short- and long-term” (p. 7). The enhancement of consequential

validity of assessment will lead to bridging the gap between diagnosis and intervention.

Subsequently, the central research question of this thesis is: How to increase

consequential validity of (dynamic) assessment?

Research regarding consequential validity demands for a two-sided approach. Not only

the assessment procedure itself should be investigated, but also the skills of professionals

involved who interpret and translate the outcomes into educational practice, i.e.,

CHAPTER 1

20

the significant others. Questions that will be addressed are: What is the role of test

outcomes in psycho-educational practice, and how do current dynamic assessment

procedures contribute to this issue? How should educational professionals’ beliefs and

skills be enhanced in the case of at-risk students? The present thesis brings the results

of several explorative studies that address these questions in an attempt to enhance the

consequential validity of (dynamic) assessment procedures.

Overview of the thesisDue to the focus on significant others in the optimisation of learning potential of at-risk

students, and herewith the consequential validity of dynamic assessment, a strong focus on

contextual aspects is inevitable. Therefore, to warrant ecological validity, a mixed method

approach is required for the thesis. The combination of qualitative and quantitative studies

allows examining the current educational practices in depth.

In the next chapter, chapter 2, first the results of a literature review on dynamic assessment

procedures will be presented. The introduction of this article deals with the different formats

of dynamic assessment and provides evidence for the use of dynamic assessment in practice.

However, the consequential validity of existing dynamic procedures needs to be investigated.

Do outcomes indeed provide more accurate clues to bridge the gap between assessment and

intervention than static procedures? This chapter will provide an overview of the current

dynamic assessment procedures, the differences in their administrations and outcomes,

and its consequences for psycho-educational practice. A qualitative analysis of the learning

phase of each instrument revealed an overview of actions undertaken by examiners. This

will provide useful information to the above-mentioned bridging gap issue.

The issue of diagnosis of learning difficulties, and differentiated instruction in the

general classroom is addressed in the third chapter. More specifically, the issue of ‘didactic

resistance’ is central to this chapter. From a Vygotskian perspective didactic resistance also

implies an assessment of teachers’ activities instead of a focus upon students’ outcomes

solely. As described above, the Response to Instruction (RTI) model is an instrument


21

1

to assess both student’s progress and teachers’ instruction, in theory. To examine its

usefulness in practice, a study was carried out in which the model was implemented at

11 schools in The Netherlands. A qualitative multi-method study provided a thorough

analysis of the implementation process. The chapter gives information about the RTI

model and its application in the Dutch context with respect to its value for assessment

of didactic resistance.

The outcomes of the third chapter revealed a difference between beliefs and behaviour

of teachers. The fourth chapter elaborates on this issue, since the behaviour of teachers

has shown to have great influences on students’ learning outcomes. A quantitative

study amongst 44 teachers and 57 support professionals has been carried out to study

the influence of beliefs, in particular implicit theories of intelligence, to the actions

of educational professionals. Hierarchical regression analyses and Structural Equation

Modelling revealed the underlying links and moderations. This chapter shows results

on the interplay between implicit theories of intelligence and behaviours of educational

professionals in the case of at-risk students.

The above-mentioned chapters provide detailed information, but all these studies have

been carried out in a specific context. To warrant ecological validity, and to examine

the possibility for implementation of dynamic assessment in the Dutch educational

practice, a careful examination of the Dutch educational context should be carried

out. Therefore, chapter 5 describes the results of two qualitative studies. The first study

is a qualitative analysis of in-vivo questionnaires among 36 Dutch educational advisors,

21 remedial teachers and 44 teachers. Moreover, a case study has been conducted to

investigate outcomes of the first study more in depth. The ‘care’ processes around

an at-risk student in first grade has been followed during a four-month period. The

combination of both studies revealed a detailed overview of the current practices in

case of at-risk students. The chapter has a strong focus on the outcomes of assessment

procedures in Dutch educational practice.

CHAPTER 1

22

Since an underlying aim of the thesis was to optimise learning potential in the general

classroom, an intervention was developed to enhance teachers’ skillfulness in handling

at-risk students in the general classroom. Chapter 6 describes a pilot study in which a short

intervention of four sessions was implemented at a Dutch regular primary school (N=21

teachers). By making use of multilevel growth curve analyses, the effects of the training

have been analysed quantitatively. The results are presented in the sixth chapter.

In sum, with respect to the two-sided approach of the issue around consequential validity

these chapters can be grouped. Chapters 2 and 5 explore consequential validity from

an assessment-perspective, whereas chapter 3, 4, and 6 focus on the significant others-

perspective. The results of the above-mentioned studies gave thought for reflection

on theoretical assumptions. Consequential and ecological validity will be mirrored,

next to the methods used in the thesis. The value of dynamic assessment for (psycho-)

educational practice will be stressed from a classroom perspective. Hence, theoretical

and practical implications of the studies will be listed in the final chapter.


23

1

Chapter 2A review scrutinising the consequential validity of dynamic assessment

This chapter is published as: Tiekstra, M., Minnaert, A., & Hessels, M. G. P. (2016).

A review scrutinising the consequential validity of dynamic assessment. Educational Psychology, 36, 112-137. doi:10.1080/01443410.2014.915930

CHAPTER 2

26

Abstract

This literature review explored whether dynamic assessment procedures in psycho-

educational practice might bridge the well-known gap between diagnosis and

intervention. Due to a learning phase included in the testing procedure, qualitative

information about the child’s learning needs can be revealed by means of dynamic

assessment. The question is, however, what the consequential validity, i.e. the extent

to which assessment influences instructional and learning processes, of dynamic

assessment procedures really is. The review of 31 articles that met the inclusion criteria

showed that proximal consequential validity of dynamic assessment is warranted, but

distal consequential validity is warranted to a lesser extent (e.g., some guidelines for

practice). Furthermore, it can be noticed that motivational aspects never played an

explicit role during learning phases. In order to design student-tailored interventions

following dynamic assessment, there is a need for more explicitness of learning phases

and types of feedback in the development of these instruments.

A REVIEW SCRUTINISING THE CONSEQUENTIAL VALIDITY OF DYNAMIC ASSESSMENT

27

2

Introduction

Nowadays, in cohesion with current inclusive education policies, it is a challenge for

educational practitioners to include children having severe learning difficulties into

regular education (UNESCO, 1994). To be able to adopt these children, a proper

assessment of the child’s capacities is requested. On top of this, in case of special needs

education, individualized education plans (IEP) are obligatory in order to construct

adequate interventions. This, in turn, requires an indication of the child’s ability to

learn. Not only is the necessity for appropriate assessment procedures underlined here,

but also the need for appropriate intervention guidelines. Nevertheless, in practice, the

gap between diagnosing and intervention is far too often neglected or even disregarded

(Minnaert & Vermunt, 2006; Resing, 2006; Ruijssenaars, 2001; Shapiro, 1987; Shapiro

& Kratochwill, 2000; Wilson, 1998).

A way to resolve this issue is to highlight the importance of consequential validity

during assessment procedures. Additionally, already in 1989, Messick pointed at the

lack of focus on consequences of test use in validity research of tests. He stated that

validity refers to the “degree to which empirical evidence and theoretical rationales

support the adequacy and appropriateness of inferences and actions based on test scores”

(Messick, 1989, p.5). Accordingly, Gielen, Dochy and Dierick (2003) investigated the

influence of assessment on learning of students, which could be defined as one form

of consequential validity of assessment. Besides, consequential validity refers to the

influence of assessment on the instructional process (Messick, 1995). Furthermore, a

division can be made in the extent to which or the moment that consequential validity

occurs: proximal consequential validity refers to the consequences that influence

learning and teaching during the testing procedure, while distal consequential validity

denotes consequences of testing beyond the testing procedure, i.e., later on in the

instructional processes in the classroom.

CHAPTER 2

28

In static testing both types of consequential validity can profoundly be questioned.

Amongst others, Shapiro and Kratochwill (2000) pointed at the great demands

from educational practice for cues following the outcomes of an IQ test. A dynamic

assessment procedure is a more promising instrument to bridge the gap between

assessment and instruction. During dynamic assessment a learning phase is included in

which the examiner has an active role. During testing the examinee is being taught the

prerequisites of the test procedure and interaction plays a major role. In general, the aim

of dynamic assessment is twofold, namely to be able to make a more accurate evaluation

of a child’s capacities and to provide useful information for clinical purposes. One could

state that during dynamic testing the proximal consequential validity is warranted, since

an explicit opportunity to learn is provided during testing and the examiner should

act in such way that an optimization of the child’s learning is revealed. Moreover, it is

measured to what extent the examinee profits from explanations and feedback, which

could give clues for future actions in the classroom. Therefore, dynamic testing could be

a very promising tool to bridge the gap between diagnosis and intervention.

Dynamic assessmentThe theoretical framework for dynamic assessment procedures is based on the theory

of Vygotsky and his conceptualization of the Zone of Proximal Development (Vygotsky,

1978). This Zone of Proximal Development (ZPD) is the difference between the actual

independent achievement of the child and the performance level of the child when

tailored help is provided (see also Lidz, 1995). Teaching in the Zone of Proximal

Development helps fulfilling the child’s learning potential. Hence, the aim of the

learning phase in dynamic assessment procedures is to optimize learning processes

by means of teaching learning strategies or provide scaffolding and prompts during

learning processes.

Due to teaching and consequently the active role of the examiner, the testing

situation is adaptive to the examinee. In other words, during dynamic assessment the

responsiveness of the child to adaptive teaching will be evaluated which provides an


29

2

individualized diagnosis of the child’s ability to learn and the learning processes that

have been addressed. This is of value while diagnosing children, especially for children

having learning difficulties. Resing (2006) underlined the fact that a dynamic approach

provides qualitative information and much more resembles classroom learning than

static procedures. Meijer (2001) demonstrated that dynamic assessment procedures are

less biased than static measures according to test anxiety and the lack of self-confidence.

Several authors showed the added value of dynamic procedures in the assessment of

at-risk children (Beckmann, 2006; Carlson & Wiedl, 1992; Hessels, 1993, 2000; Resing,

1990; Schlatter & Büchel, 2000).

Different approaches to dynamic assessment exist, e.g., domain-specific versus

general capacity evaluation or focus on intervention versus focus on evaluation. The

main distinction in approaches, however, lies in diverging goals and, thus, manifests

in two different procedures (distinction in role of examiner). In this, there is the

clinical approach of Feuerstein (Feuerstein, Rand & Hoffman, 1979) on the one side.

Supported by amongst others Tzuriel (Shamir & Tzuriel, 2004), this approach aims

at the evaluation of the cognitive modifiability of the child during mediated learning

experiences. On the other side, there are standardised approaches: dynamic tests or so-

called learning potential tests, aiming at the evaluation of learning abilities of children.

These tests try to measure to what extent a child is able to apply new information

that has been taught during the learning phase. In a (pre-test –) training – post-test

format the learning abilities of a child are assessed by measuring the extent to which

the child has profited from the learning phase. In another format, the train-within-test

format, the examinees are offered graduated prompting or feedback. Both formats are

standardised and require an active role of the examiner in teaching strategies required

for resolving test items.

Standardisation of testing procedures makes it possible to replicate testing and diminish

the influence of personal characteristics of the examiner, which is a condition to meet

scientific criteria for tests (objectivity, reliability, and validity). Moreover, criteria for the

CHAPTER 2

30

identification of disabilities applied by the World Health Organization or as indicated

in the Diagnostic Statistic Manual (DSM) are based on assessments showing satisfactory

psychometric characteristics.

During dynamic assessment examiners play an important role, since it is their duty to

optimize performances of the examinee. In dynamic tests this role is more standardised,

whereas it is not in the clinical approaches. Dynamic testing is regarded as a promising

tool to estimate the real capacities of at-risk children and to make better distinguishes

between children with learning disorders and children having learning delays (as

a consequence of their learning environments and former learning experiences).

Dynamic tests fit into these requirements of psychometric characteristics. Therefore,

the focus of this article is rather on dynamic testing as on the more clinical assessment

approaches. More precisely, only dynamic tests with short term learning phases are used

for this review, in order to make a clear distinction between tests that try to measure

learning capacity as opposed to tests that try to change capacity. Consequently, in this

article, the term dynamic assessment refers to a dynamic procedure with a standardised

approach in which a short term learning phase is included in testing.

Aim of the present reviewCurrently, articles about dynamic assessment merely focus on comparisons to

traditional (static) measures or on psychometric characteristics. Since one of the aims

of dynamic assessment is to optimize learning, the purpose of this review is to focus at

the dynamic part in dynamic assessment and to examine what has been written about

the consequential validity of dynamic tests. It is of added value to investigate what the

consequences of dynamic assessment are for practice. Over time, quite a few dynamic

assessment procedures have been developed. In order to clarify what makes dynamic

assessment ‘dynamic’, the learning phases of dynamic assessment procedures will be

scrutinized. The learning phases in these procedures differ latently and manifestly. The

differences will be revealed due to a screening of the presence of cognitive, metacognitive

or motivational strategies (Dina & Efklides, 2009; Wigfield, Byrnes, & Eccles, 2006)


31

2

as a goal of the examiner’s activities during the learning phase. In other words, the

focus is on the activities the examiner carries out and whether these are more cognitive,

regulative or affective oriented. Cognitive strategies are strategies that the student

applies to process information, e.g., relating, structuralizing, analysing, memorizing

and applying. Metacognitive strategies are strategies to regulate learning processes,

like planning, evaluating, adjusting, diagnosing, reflecting. Whereas motivational

strategies are more affective oriented, like adjustment of attributions, values, emotions,

expectations and concentration. A further elaboration on these strategies can be found

in the classification schemes of Vermunt (1996).

Moreover, the outcomes of different dynamic instruments will be investigated to

unravel the value of dynamic assessment for construction of interventions that might

follow the assessment. In this way the consequential validity of each instrument will be

emphasised.

Over years, several dynamic assessment procedures have been developed for higher

education (e.g., Antón, 2009; Meijer & Elshout, 2001; Nirmalakhandan, 2009). In

these assessment procedures the focus is more on career or professional perspectives

than on the optimization of development of young individuals. This review, however,

aims at more insight around the construction of adequate classroom interventions for

children with some kind of special needs, since these will contribute to the optimization

of development of children. Moreover, in order to optimize a child’s development (or in

other words, to adapt the curriculum as best as possible to the child’s learning needs), an

early recognition of a child’s learning difficulties is necessary. An early evaluation of a

child’s learning weaknesses and strengths is a prerequisite to provide early intervention,

which, in turn, paves the way for optimal development. Therefore, this review will only

include dynamic tests estimating the ability, capacity or potential to learn of children

in regular primary (inclusive) educational settings and the transition to secondary or

middle schools. Hence, studies carried out in grades one to eight in regular education

are the review’s objectives.

CHAPTER 2

32

Method

The literature review consisted of an extensive computer search. For this purpose the

database of Web of Science was used. This database includes the Science Citation

Index Expanded (SCIE), the Arts and Humanities Citation Index (A&HCI) and Social

Sciences Citation Index (SSCI). The use of this database implies that the references

found are from articles that are published in international peer reviewed scientific

journals. References from 1995 until 2011 have been included.

In order to capture as many relevant references as possible, a ‘search model’ had been

developed. The model tries to combine several keywords into three categories which

were regarded as most valuable to find literature about dynamic assessment. The three

categories were the following: potential, dynamic, ability/capacity. The category potential

enclosed the keywords ‘learning potential’ and ‘students’ potential’. 245 Articles had

been found with these keywords. The second category dynamic comprised the keywords

‘dynamic assessment’; ‘dynamic test’; ‘dynamic tests’; ‘dynamic testing’. This resulted

in 2828 articles. The category ability/capacity enclosed the keywords ‘cognitive abilities’;

‘ability testing’; ‘ability test(s)’; ‘ability to learn’; ‘capacity testing’. This third category

had been used only in combination with one of the two other categories to narrow

down the number of hits in the database. For example, the keyword ‘cognitive abilities’

was combined with ‘dynamic’ in a first search, and in a second search combined with

‘potential’. A total of 574 articles were found using these combinations.

The search strategy resulted in a total of 3420 articles after having deleted overlapping

articles between (and within) the different categories. The list of 3420 references was

screened on relevance to the field of learning and development. Subsequently, the total

number of articles was reduced since they did not meet the inclusion criteria for the

review study. In line with the previous section, the following inclusion criteria were

formulated:


33

2

1. The article fully describes an empirical study focusing at a dynamic testing

procedure to assess cognitive abilities.

2. The dynamic testing procedure described in the article is standardised and

includes a short term learning phase.

3. The study focuses on children aged 6 to 14 years (in the case of special

education a division has been made based on the average curriculum taught

that fits these ages).

4. The study has been carried out in regular primary or special needs education.

The greatest exclusion of articles was caused by the first criterion, since a large part of

the articles found did not describe a procedure to assess cognitive abilities. Moreover,

some references were excluded since they were not full articles, such as book reviews,

comments, or abstracts for presentation at an international conference (45) or were

not written in English (18). Finally, two references were untraceable: the articles could

neither be retrieved from the internet, nor available at any of the university libraries in

The Netherlands, nor after contact with the main author. The number of articles that

has been used for the literature review is 31.

Results

Tables 1 and 2 present an overview of the results following the literature review. Table

1 summarizes the empirical studies according to the instruments being used and their

psychometric properties (if described). Table 2 zooms in on the dynamic tests that have

been used in the studies. The aim of table 2 is to reveal differences and similarities

between the dynamic tests with respect to their learning phases.

It is emphasized, though, that psychometric properties described below should be

considered critically. According to the authors, some methodological errors exist.

Reliability measures, for example, vary between Cronbach’s alpha or interrater

CHAPTER 2

34

agreements, and some studies use test-retest reliability. The latter form does, however, not refer

to reliability in this context, but rather to a measure for construct validity of the dynamic test.

After all, test-retest reliability can be seen as a measure for stability, which is not in accordance

with the basis of dynamic testing.

As can be seen in table 1, slightly more than half (18 of 31) of the studies enclosed in the review

have been carried out in North-America (USA/Canada), the remaining in Europe, Israel or

Africa. In line with our inclusion criteria, dynamic assessment procedures described were mostly

developed for children in grade 1-8, but the age ranged 4,5 years to 78,6. Some articles report

studies that have been carried out in special education. In these studies the mean chronological

age goes beyond the age of our interest, however, the average curriculum taught or tested fits our

criterion age range. Roughly half of the articles found (18 of 31) describe studies with a sample

size larger than or equal to 100.

Test procedures differed from train-within-test format to (pre-test –) training – post-test format.

Overall, reliability, as mentioned in the method section as well as in results section in the articles,

of the instruments was satisfactory, but a huge variation in reliability coefficients was encountered.

The internal consistencies ranged from α .37 to .98, test-retest or pre-test – post-test reliabilities

from .54 to .72 and interrater agreements from .64 – 1.00. Validity measures include different

values and calculations, as is shown in table 1. Predictive validity (if indicated in the articles) was

of particular interest, since this form of validity matches one of the reasons to do an assessment

in educational settings; namely to provide an estimation or prediction of students’ future

learning outcomes. Therefore, a division was made of predictive validity based on comparisons

of dynamic assessment scores with either school achievement test or static intelligence test

(IQ) scores. The correlations with school achievement test scores varied from .11 to .71, and

described explained variances from 1% to 54%. One study showed an explained variance

of 40% in reading growth. Three studies demonstrate 4%, 25% and 40% explained variance

on top of IQ. The range of correlations with IQ tests varies from -.18 to .74. However, using an

interpretation of the correlation between IQ and DA measure solely is too simplistic. A criterion

measure is needed to be able to estimate the predictive validity.


35

2

Tabl

e 1

| O

verv

iew

of

arti

cles

sel

ecte

d fo

r th

is r

evie

w

Auth

ors

(yea

r of

pu

blic

atio

n)C

ount

ryN

Subj

ects

Dyn

amic

test

Test

pr

oced

ure

Rel

iabi

lity

(Pre

dict

ive)

va

lidit

yC

onse

quen

tial

va

lidit

y

Com

pton

, Fuc

hs,

Fuch

s, Bo

uton

, G

ilber

t, Ba

rque

ro,

Cho

, & C

rouc

h (2

010)

USA

355

Gra

de 1

DA

ps

eudo

wor

dsTr

ain

with

in te

st Te

st-re

test

(4 w

eeks

): .7

2R

ange

: .1

1 to

.65

Scor

es in

dica

te

instr

uctio

nal

scaff

oldi

ng le

vel n

eede

d to

cor

rect

ly so

lve

item

s (m

in. 3

; max

. 15)

. The

lo

wer

scor

es, t

he le

ss

scaff

oldi

ng

De

Beer

(201

0a)

Sout

h A

frica

79 (b

oys)

72 (g

irls)

mea

n ag

e bo

ys: 1

2.44

mea

n ag

e gi

rls:

11.1

8

Lear

ning

Po

tent

ial

Com

pute

rised

A

dapt

ive

Test

(LPC

AT)

Test

– tr

aini

ng

– te

st

α .9

3 to

.98

Scho

ol

achi

evem

ent:

boys

: R² =

.33

girls

: R² =

.32

Post-

test

scor

e,

diffe

renc

e sc

ore

and

com

posit

e sc

ore

Dol

ores

Cal

ero,

Be

len,

&

Aux

iliad

ora

Rob

les

(201

1)

Spai

n12

77

– 11

yA

dapt

ed

subt

ests

of th

e LP

AD

:

Posit

ions

test

The

org

aniz

er

test

Sten

cil d

esig

n te

st

Trai

n w

ithin

test

Test

– tr

aini

ng

– te

st

Test

– tr

aini

ng

– te

st

α .7

0 to

.95

Indi

rect

ly ta

ken

into

acc

ount

by

mak

ing

use

of

Disc

rimin

ant

Ana

lysis

Gai

n sc

ore,

bas

ed o

n as

sista

nce

need

ed

Raw

scor

es o

n pr

etes

t, po

sttes

t, an

d ga

in sc

ore

Gai

n sc

ore,

bas

ed o

n as

sista

nce

need

ed

CHAPTER 2

36

Elle

man

, Com

pton

, Fu

chs,

Fuch

s, &

Bo

uton

(201

1)

USA

100

Gra

de 2

m

ean

age

8y3m

DA

infe

renc

e ge

nera

tion

Test

– tr

aini

ng

– te

st –

tran

sfer

α .7

6

Inte

rrat

er

agre

emen

t 97

,8%

Cog

nitiv

e te

st:

4% u

niqu

e va

rianc

e ex

plai

ned

by D

A

3 sc

ores

: num

ber

of p

rom

pts n

eede

d,

tran

sfer s

core

and

tota

l sc

ore

Fabi

o (2

005)

Italy

287

6 –

11

(mea

n ag

e:

8 y

2 m

)

Dyn

amic

Tes

t of

Pot

entia

l In

telli

genc

e

Trai

n w

ithin

test

α .8

6IQ

: .48

(Rav

en

CPM

)Sc

hool

ac

hiev

emen

t: .2

7 to

.36

Gen

eral

Mod

ifiab

ility

In

dex

(= sc

ore

base

d on

nee

d fo

r hel

p)

Fuch

s, C

ompt

on,

Fuch

s, Bo

uton

, &

Caff

rey

(201

1)

USA

318

Gra

de 1

DA

ps

eudo

wor

dsTr

ain

with

in te

stTe

st –

rete

st (4

w

eeks

): .7

2 2.

3% u

niqu

e ex

plai

ned

varia

nce

in w

ord

iden

tifica

tion

1.0%

uni

que

expl

aine

d va

rianc

e in

read

ing

com

preh

ensio

n

Scor

es in

dica

te

instr

uctio

nal

scaff

oldi

ng le

vel n

eede

d to

cor

rect

ly so

lve

item

s (m

in. 3

; max

. 15)

. The

lo

wer

scor

es, t

he le

ss

scaff

oldi

ng

Fuch

s, C

ompt

on,

Fuch

s, H

olle

nbec

k,

Cra

ddoc

k, &

H

amle

tt (2

008)

USA

122

Gra

de 3

DA

alg

ebra

sk

ills

Trai

n w

ithin

test

Indi

rect

ly ta

ken

into

acc

ount

by

mak

ing

use

of S

truc

tura

l Eq

uatio

n M

odel

ing

(SEM

)

Mat

h (sc

hool

ac

hiev

emen

t)

.41

to .6

0

Scor

es in

dica

te

instr

uctio

nal

scaff

oldi

ng le

vel n

eede

d to

cor

rect

ly so

lve

item

s (m

in. 0

; max

. 21)

. The

m

ore

scaff

oldi

ng, t

he

less

poi

nts

Fuch

s, C

ompt

on,

Fuch

s, H

olle

nbec

k,

Ham

lett,

&

Seet

hale

r (20

11)

USA

122

Gra

de 3

DA

alg

ebra

sk

ills

Trai

n w

ithin

test

The

aut

hors

re

fer t

o th

e ab

ove

men

tione

d ar

ticle

of

2008

Impr

ovem

ent

of c

lass

ifica

tions

du

e to

DA

: 48%

to

73%

Scor

es in

dica

te

instr

uctio

nal

scaff

oldi

ng le

vel n

eede

d to

cor

rect

ly so

lve

item

s (m

in. 0

; max

. 21)

. The

m

ore

scaff

oldi

ng, t

he

less

poi

nts


37

2

Gill

am, F

argo

, Fo

ley,

& O

lszew

ski

(201

1)

USA

646

– 8,

5N

onve

rbal

D

A p

hone

me

dele

tion

Trai

n w

ithin

test

α .8

8C

onstr

uct v

al.:

r. 84

Wor

d le

vel

read

ing:

r. 5

4

Scor

es re

flect

ing

num

ber o

f pr

ompt

s ne

eded

(min

. 21;

max

10

5)

Gum

mer

sall

&

Stro

ng (1

999)

USA

30 (e

xp. 1

) +

12

stude

nts

with

lang

uage

pr

oble

ms

(exp

. 2)

8 –

9 (g

rade

3)

Nar

rativ

e sk

ill

asse

ssm

ent

Test

– tr

aini

ng

– te

st

Inte

rrat

er

agre

emen

t 99

.2%

to 1

00%

.68

to .9

8

?St

atic

scor

e (s

tory

-le

ngth

and

synt

ax)

+ q

ualit

ativ

e in

terp

reta

tion

of e

rror

s (a

udio

tape

d)

Ham

ers,

Hes

sels,

&

Penn

ings

(199

6)T

he

Net

herla

nds

500

5 y

4 m

– 7

y

9 m

T

he L

earn

ing

Pote

ntia

l Tes

t fo

r Eth

nic

Min

oriti

es

(LEM

)

Trai

n w

ithin

test

Ran

ge

α .8

8 to

.92

IQ:

.45

to .6

8Sc

hool

ac

hiev

emen

t:.3

2 to

.69

stand

ardi

zed

LP sc

ores

(n

orm

refe

renc

ed)

Jelto

va, B

irney

, Fr

edin

e, Ja

rvin

, St

ernb

erg,

&

Grig

oren

ko (2

011)

USA

1332

Gra

de 4

Gro

up-

adm

inist

ered

dy

nam

ic

asse

ssm

ent

(GD

A)

mat

hem

atic

s

Test

– tr

aini

ng

– te

st

IRT

relia

bilit

y .7

9 to

.84

for

subt

ests

Take

n in

to

acco

unt b

y m

akin

g us

e of

H

LM

Raw

scor

es o

n pr

e- a

nd

postt

est

Jiten

dra

&

Kam

eenu

i (20

01)

USA

46G

rade

3

(8 y

9 m

–

9 y

10 m

)

Mat

hem

atic

al

wor

d pr

oble

ms

Trai

n w

ithin

test

Inte

rrat

er

agre

emen

t 92

to 1

00%

?N

umbe

r of

prom

pts

and

time

elap

sed

Kan

evsk

y &

Gea

ke

(200

4)C

anad

a25

8 y

10 m

–

10 y

1 m

Mat

hem

atic

al

patte

rn-

reco

gniti

on

task

s

Trai

n w

ithin

test

Inte

rrat

er

agre

emen

t 92

to 9

6%

Thi

s stu

dy:

IQ:

r = .3

4

Nee

d of

supp

ort

(tota

l num

ber o

f hi

nts +

qua

litat

ive

obse

rvat

ions

)

CHAPTER 2

38

Kat

z, G

olsta

nd,

Bar-

Ilan,

& P

arus

h (2

007)

Isra

el38

16-

12 y

The

Dyn

amic

O

ccup

atio

nal

The

rapy

C

ogni

tive

Ass

essm

ent

for c

hild

ren

(DO

TC

A-C

h)

Test

– tr

aini

ng

– te

st

ICC

(int

erra

ter)

.87

to .9

9In

t. co

nsist

ency

α.

61 to

.77

Ran

ge .3

6 to

.71

Leve

l of

med

iatio

n ne

eded

: cut

off

scor

es

+ d

ynam

ic p

erce

ntag

e sc

ores

Kirk

woo

d, W

eile

r, Be

rnste

in, F

orbe

s, &

W

aber

(200

1)

USA

202

(het

erog

eneo

us

lear

ning

pr

oble

ms)

7 –

11R

ey-O

sterr

ieth

C

ompl

ex

Figu

re (R

OC

F)

– dy

nam

ic

adap

tatio

n

Test

– tr

aini

ng

– te

st

??

Raw

scor

es b

ased

on

Dev

elop

men

tal S

corin

g Sy

stem

are

div

ided

in

leve

ls; d

ivisi

on in

two

subg

roup

s (be

nefit

/no

bene

fit)

Lars

en &

Nip

pold

(2

007)

USA

50G

rade

6M

ean

age

12,2

(age

ra

nge:

10

y 9

m –

12

y 10

m)

Dyn

amic

A

sses

smen

t Ta

sk o

f M

orph

olog

ical

A

naly

sis

(DAT

MA

)

Trai

n w

ithin

test

Inte

rsco

rer

agre

emen

t 95

to 1

00 %

Inte

rnal

co

nsist

ency

is

stron

g *

DAT

MA

&

PPV

T-ΙΙ

Ι : r.

36Sc

hool

ac

hiev

emen

t :

DAT

MA

&

OSA

: r.5

0

Scor

es b

ased

on

need

fo

r pro

mpt

s: ra

w

scor

es, m

ax. 7

5

Lauc

hlan

& E

lliot

t (2

001)

UK

30 (s

peci

al

educ

atio

n)M

ean

age

9;

mod

erat

e/se

vere

le

arni

ng

diffi

culti

es

The

Chi

ldre

n’s

Ana

logi

cal

Thi

nkin

g M

odifi

abili

ty

test

(CAT

M)

Prel

imin

ary

phas

e –

test

– tr

aini

ng

– te

st

Rel

. pro

ven

*Va

l. pr

oven

*R

aw sc

ores

(min

. 0;

max

. 18)

Lidz

& M

acrin

e (2

001)

USA

81

Gra

de

1 –

5 N

aglie

ri N

onve

rbal

A

bilit

y Te

st –

mod

ified

fo

r dyn

amic

ad

min

istra

tion

(NN

AT/D

A)

Test

– tr

aini

ng

– te

st

?C

oncu

rren

t va

lidity

K-A

BC

.64

to .7

4

Stan

dard

scor

es, b

ased

on

NN

AT n

orm

s


39

2

Pelte

nbur

g, V

an d

en

Heu

vel-P

anhu

izen

, &

Doi

g (2

009)

The

N

ethe

rland

s37

(spe

cial

ed

ucat

ion)

8,9

– 12

,11

(mea

n ag

e 10

,5 y

ears

)Av

erag

e cu

rric

ulum

7-

8 ye

ars

CIT

O

Mon

itorin

g Te

st fo

r M

athe

mat

ics,

rede

signe

d fo

r IC

T a

nd D

A

Trai

n w

ithin

test

??

Raw

scor

es +

qu

alita

tive

obse

rvat

ions

on

stra

tegy

use

Pelte

nbur

g, V

an d

en

Heu

vel-P

anhu

izen

, &

Rob

itzsc

h (2

010)

The

N

ethe

rland

s37

+ 4

3 (sp

ecia

l ed

ucat

ion)

8 –

12

Aver

age

curr

icul

um

leve

l: en

d gr

ade

2

CIT

O

Mon

itorin

g Te

st fo

r M

athe

mat

ics,

rede

signe

d fo

r IC

T a

nd D

A

Trai

n w

ithin

test

Indi

rect

ly ta

ken

into

acc

ount

by

mak

ing

use

of

diffe

rent

ial i

tem

fu

nctio

ning

(D

IF)

?R

aw sc

ores

+

qual

itativ

e ob

serv

atio

ns

on st

rate

gy u

se

Res

ing,

Tun

tele

r, D

e Jo

ng, &

Bos

ma

(200

9)

The

N

ethe

rland

s10

9 (5

5 et

hnic

- m

inor

ity

child

ren)

7y –

9y

Seria

-Thi

nk

– ad

apte

d ve

rsio

n

Test

– tr

aini

ng

– te

st

Pre-

postt

est

corr

elat

ion:

.5

7; .6

1

?N

umbe

r of

inse

rtio

ns

and

mea

sure

men

ts +

str

ateg

y-le

vel c

ateg

orie

s

Ster

nber

g,

Grig

oren

ko,

Ngo

rosh

o,

Tant

ufuy

e, M

bise

, N

okes

, Juk

es, &

Bu

ndy

(200

2)

Tanz

ania

358

(+ 1

00

child

ren

in

cont

rol g

roup

)

11y

– 13

ySy

llogi

sms

Sort

ing

Twen

ty

Que

stion

s

Test

– tr

aini

ng

– te

st

Test

– tr

aini

ng

– te

st

Test

– tr

aini

ng

– te

st

Test

– re

test

relia

bilit

y: .5

4

Test

– re

test

relia

bilit

y: .5

8 /

.64

Test

– re

test

relia

bilit

y: .6

6

Cog

nitiv

e te

sts:

.21

to .3

1Sc

hool

ac

hiev

emen

t:.2

9 to

.33

Cog

nitiv

e te

sts:

.29

to .3

5 /

.32

to .3

8Sc

hool

ac

hiev

emen

t:.3

5 to

.42

/.3

4 to

.50

Cog

nitiv

e te

sts:

.12

to .3

2Sc

hool

ac

hiev

emen

t:.1

5 to

.28

Num

ber o

f co

rrec

t an

swer

s in

pret

est,

trai

ning

and

pos

ttest

Num

ber o

f pe

rsev

erat

ive

erro

rs

and

cate

gorie

s pr

oduc

ed +

num

ber o

f hi

nts (

trai

ning

pha

se)

Rat

io o

f co

nstr

aint

-se

ekin

g qu

estio

ns

to to

tal n

umbe

r of

ques

tions

CHAPTER 2

40

Swan

son

(199

5a)

USA

Exp.

1: 6

1Ex

p. 2

: 191

Mea

n ag

e 10

.63

Mea

n ag

e 10

.66

S-C

PT

(Sw

anso

n-C

ogni

tive

Proc

essin

g Te

st)

Trai

n w

ithin

test

α' s:

.80

to .9

5

Ove

rall

relia

bilit

y: α

. 9

8

14%

to 2

6%

expl

aine

d va

rianc

e on

scho

ol

achi

evem

ent

In a

dditi

on to

initi

al

stand

ard

scor

e: g

ain

scor

e, p

robe

scor

e an

d m

aint

enan

ce sc

ore

Swan

son

(199

5b)

USA

/

Can

ada

1611

(10%

sp

ecia

l edu

c.)

Age

rang

e:

4,5

– 78

,6

S-C

PT

Trai

n w

ithin

test

α‘s:

.82

to .9

5 32

% e

xpla

ined

va

rianc

e in

read

ing

reco

gniti

on

7 co

mpo

site

stand

ard

scor

es (M

=10

0;

SD=

15)

Swan

son

(201

1)U

SA78

M

ean

age

11.6

24

subt

ests

of S

-CPT

: rh

ymin

g,

digi

t-sen

tenc

e,

visu

al m

atrix

, m

appi

ng a

nd

dire

ctio

ns

Trai

n w

ithin

test

Subt

ests

corr

elat

ion

rang

e .4

0 to

.80

40%

exp

lain

ed

varia

nce

in

read

ing

grow

th

Com

posit

e sta

ndar

d sc

ores

. Ini

tial s

core

, ga

in sc

ore,

pro

be sc

ore

and

mai

nten

ance

scor

e

Swan

son

& H

owar

d (2

005)

USA

70 st

uden

ts w

ith re

adin

g (a

nd m

ath)

pr

oble

ms

Mea

n ag

e 11

.93

2 su

btes

ts of

S-C

PT:

rhym

ing

and

digi

t-sen

tenc

e

Trai

n w

ithin

test

α’s .

76 to

.85

25%

ext

ra

(uni

que)

va

rianc

e on

top

of IQ

Initi

al sc

ore,

gai

n sc

ore,

pro

be sc

ore

and

mai

nten

ance

scor

e

Tie

kstr

a, H

esse

ls, &

M

inna

ert (

2009

)Sw

itzer

land

46 (s

peci

al

educ

atio

n)15

y 6

m

– 19

y 4

m

(mea

n ag

e 17

yea

rs)

Aver

age

curr

icul

um

8 –

10

year

s

Hes

sels

Ana

logi

cal

Rea

soni

ng T

est

(HA

RT)

Trai

ning

–

postt

est

α’s .

69 to

.92

Thi

s stu

dy: .

65

IQ:

-.18

Scho

ol

achi

evem

ent

(CLT

):.5

7

40%

pre

dict

ion

in C

LT o

n to

p of

IQ

Raw

scor

es (m

in. 0

; m

ax. 2

0)


41

2

Tzur

iel (

2000

)Is

rael

48G

rade

2

(mea

n ag

e =

7,5

y)

Seria

-Thi

nk

Instr

umen

tPr

elim

inar

y ph

ase

– te

st –

trai

ning

–

test

α ' s

.37

to .8

5Sc

hool

ac

hiev

emen

t (m

ath)

:R

² = .1

9 R

² = .5

4

Raw

scor

es, d

ivid

ed

in p

erfo

rman

ce le

vel

(max

. 20)

; num

ber o

f in

sert

ions

; num

ber o

f m

easu

rem

ents

Tzur

iel &

Kau

fman

(1

999)

Isra

el52

Gra

de 1

(m

ean

age

= 7

,16

and

Ethi

opia

n gr

oup:

6 y

–

7 y

6 m

)

The

Chi

ldre

n’s

Ana

logi

cal

Thi

nkin

g M

odifi

abili

ty

test

(CAT

M)

The

Chi

ldre

n’s

Infe

rent

ial

Thi

nkin

g M

odifi

abili

ty

test

(CIT

M)

Prel

imin

ary

phas

e –

test

– tr

aini

ng

– te

st

Test

– tr

aini

ng

– te

st –

tran

sfer

phas

e

Rel

. ok

*

α' s

.82

to .9

0

Val.

ok *

Val.

ok *

Raw

scor

es: “

all-o

r-no

ne”

and

“par

tial

cred

it”

Raw

scor

es

Vilj

oen,

Odg

ers,

Gris

so &

Till

broo

k (2

007)

USA

190

(subg

roup

)11

y –

13y

Mac

Art

hur

Com

pete

nce

Ass

essm

ent

Tool

-Crim

inal

A

djud

icat

ion

(Mac

CAT

-CA

)

Trai

n w

ithin

test

Intr

acla

ss

corr

elat

ion

.64

to .9

1

?R

aw sc

ores

Note.

* =

onl

y in

form

atio

n in

dica

ted

in th

e ar

ticle

; ? =

lack

ing

info

rmat

ion

in th

e ar

ticle.

CHAPTER 2

42

Consequential validityWith regard to consequential validity of dynamic assessment procedures included in the

review, it is of importance to investigate the possible consequences of these testing procedures

in practice. During analysis of articles this was reflected by the extent to which the outcomes

of dynamic assessment procedures were made explicit for practice. Indeed, the procedures

differed in their extent of explicit information for practice. There were three categories in

which the dynamic assessment procedures appeared to be divided. It is highlighted to what

extent proximal consequential validity (raw or self-evident scores) and distal consequential

validity (explicit information for practice) occur in assessment procedures in each of these

categories.

Raw or standardised scores. Dynamic assessment procedures that result in a raw or

standardised score have a minimal form of consequential validity. Although the LPCAT

(De Beer, 2010a) provides an indication of the education level of the examinee compared

to national education levels (De Beer, 2010b), the test does not result in explicit clues for the

development of an intervention. Even so, in the studies with the LEM (Hamers et al., 1996),

the NNAT/DA (Lidz & Marcine, 2001), and the study of Dolores-Calero et al. (2011), the test

scores are norm referenced, but what these scores are made of is not indicated in this article

and thus, it is unclear whether the scores provide indications for practice. Other dynamic

assessment instruments (Jeltova et al., 2011; Tiekstra et al., 2009; Tzuriel & Kaufman, 1999;

Tzuriel, 2000; Viljoen et al., 2007) result in raw scores. Lauchlan and Elliott (2001) describe

a study that investigated the extent to which scores of the CATM could predict the learning

progress of children following a cognitive intervention program. Although this indicates

some potential for distal consequential validity, the CATM scores, however, were only raw

scores similar to the version applied in study of Tzuriel and Kaufman (1999) mentioned

before.

Since a learning opportunity is provided during the abovementioned procedures, the

proximal consequential validity is warranted. The outcomes are represented by a number,

and this number should be appropriate for predicting future learning outcomes. However,


43

2

no explicit information about the learning processes is reflected in this number. The gap

between diagnosis and intervention will be apparent after administration of these tests. To

conclude, the distal consequential validity is not guaranteed according to these test outcomes.

Raw scores + indication of level of help needed. Several dynamic assessment

procedures resulted in an indication of the level of support needed in addition to raw

scores. Some of the procedures still result in an isolated number, which needs to be

interpreted in a specific manner. Compton et al. (2010) and Fuchs, Compton, Fuchs,

Bouton et al. (2011), for instance, demonstrated that a dynamic assessment procedure

on pseudo words could forecast a child’s responsiveness to intervention: the procedure

provides a score which indicates the instructional scaffolding level needed which

could link assessment to educational practices, nonetheless implicitly. In the studies of

Fuchs, Compton, Fuchs, Hollenbeck et al. (2008, 2011) , and Larsen & Nippold (2007)

administration of scores is done in a similar way. Minimum and maximum scores are

composed of points for items solved correctly, reduced (or increased) by the number

of hints needed. The tests of Sternberg et al. (2002) and Elleman et al. (2011) result in

raw scores combined with the number of errors made or the number of hints required

during training. This can be seen as a slightly more advanced form of proximal

consequential validity than the previous category, but still, distal consequential validity

is not warranted. No explicit clues for practice could be detected in these procedures.

This also holds for the instruments that result in multiple scores, like the S-CPT

(Swanson, 1995a, 1995b, 2011; Swanson & Howard, 2005), the dynamic administration

of the Rey-Osterrieth Complex Figure Test (Kirkwood et al., 2001), and the adapted

version of the Seria-Think Instrument (Resing et al., 2009). In the latter, though,

an estimation was made of the level of strategies used (least sophisticated strategy to

advanced strategy) in addition to the raw scores. The link to instruction or intervention

after the test is not made explicitly, and the estimation of the child’s competences is

central to these instruments.

CHAPTER 2

44

Other procedures slightly touched distal consequential validity, since these added

qualitative information to the raw scores. Gummersall and Strong (1999) tested a

dynamic procedure for narrative skill assessment. After completion of this assessment

not only static scores are presented, but the qualitative interpretation of errors (audio

taped) provides useful individualized clues for intervention. Similarly, next to raw scores,

Peltenburg et al. (2009, 2010) add qualitative information about strategy use based

on video observations which were made during the assessment procedure. How this

qualitative information is given back to the educational psychologists in order to set up

an IEP remains, however, unclear. Therefore, these instruments only slightly address

distal consequential validity.

Level of mediation needed / need of support. Some dynamic assessment

procedures did not result in raw scores at all. These procedures have been grouped

in this third category. Initially, one would think that this category provides the most

advanced form of consequential validity since scores only reflect the level of help

needed. It appeared, however, that these procedures also never explicitly mentioned

clues for practice. For example, the instruments of Fabio (2005), Gillam et al. (2011),

Jitendra and Kameenui (1996), and Katz et al. (2007) resulted in scores composed of

number of prompts needed during testing. The scores as such do not provide information

for designing interventions. A qualitative analysis of the probes used by each child,

however, can, since the standardised probes depended on the errors made. In this way

the scores provide more information for interventions than the outcomes of dynamic

assessment procedures mentioned previously, although information remains implicit.

Scores that provide more explicit information for practice can be found in the study of

Kanevsky and Geake (2004). They investigated the interactions during testing: concrete,

cognitive or metacognitive instruction by examiner and content-related or spontaneous

(prompting) responses of student. The outcomes of the dynamic procedure indicate the

need of support, composed of the total number of hints, time needed to complete the

tasks, and added by qualitative observations.


45

2

To unravel the possibility of distal consequential validity of dynamic assessment

procedures more into detail, learning phases should be scrutinized. Since the learning

phase enhances a teaching component, which could provide valuable information for

future classroom or intervention actions, the next paragraph describes the learning

phases of each instrument (29 in total).

Learning phaseTable 2 shows differences and similarities between the dynamic tests with respect to their

learning phases. Dynamic assessment procedures cover general or specific domains.

Half of the tests try to assess the general ability of the child, the other half abilities in

specific domains as reading or mathematics. The HART (Tiekstra et al., 2009) and the

DA inference generation (Jeltova et al., 2011) are the only instruments that have been

group administered, including a group wise training. Although, the LPCAT (De Beer,

2010a) and the test of Peltenburg et al. (2009, 2010) can be administered in groups too,

since these are computerized tests.

Furthermore, it is of importance to scrutinize the learning phases of each dynamic

assessment procedure in order to investigate the potential consequential validity

for practice. Table 2 zooms in at the different learning phases with regard to their

manifestations and contents. Overall, one could state that a reasonable amount of

instruments use the ‘graduated prompts’ approach. According to Resing et al. (2009)

this implies a standardised training procedure in which prompts are prescribed and

hierarchically ordered. The latter makes the testing procedure adaptive, since the

examiner does not provide more hints or prompts when the child has found the right

answer. Other learning phases are composed of extra practice, extra instruction on

failed items or on strategies and skills required to solve the tasks. During some of the

learning phases general feedback is given.

In order to scrutinize the learning phases more carefully, their content, i.e., the aims of

examiner’s activities, has been examined. Therefore, a division in cognitive, metacognitive

CHAPTER 2

46

Tabl

e 2

| O

verv

iew

of

inst

rum

ents

use

d in

the

stud

ies

sele

cted

for

this

rev

iew

Test

Dom

ain

(spe

c./

gene

ral)

Adm

inis

trat

ion

(ind

./gr

oup)

Test

dur

atio

nM

anif

esta

tion

of

lear

ning

ph

ase

Con

tent

of

lear

ning

ph

ase

CM

CM

DA

pse

udow

ords

Sp

ecifi

c: re

adin

gIn

divi

dual

ly20

– 3

0 m

in.

Gra

duat

ed p

rom

ptin

g (in

struc

tiona

l sca

ffold

ing)

x-

-

LPC

ATG

ener

alIn

divi

dual

ly &

gro

up

wise

(com

pute

rized

)?

Extr

a pr

actic

e; g

ener

al st

rate

gies

&

hin

ts; fe

edba

ckx

*-

Posit

ions

Tes

t

The

Org

aniz

er T

est

Sten

cil D

esig

n Te

st

Gen

eral

(visu

o-sp

atia

l m

emor

y)

Gen

eral

Gen

eral

(per

cept

ual

struc

tura

tion)

Indi

vidu

ally

Indi

vidu

ally

Indi

vidu

ally

3 se

ssio

ns o

f 50

m

inut

es in

tota

lG

radu

ated

instr

uctio

n on

faile

d ite

ms

Instr

uctio

n on

stra

tegi

es,

mem

ory

and

orga

niza

tion

Trai

ning

on

visu

al tr

ansp

ort a

nd

inte

rnal

tran

sform

atio

n of

the

stim

ulus

x - x

- x -

- - -

DA

infe

renc

e ge

nera

tion

Spec

ific:

read

ing

Indi

vidu

ally

25 m

in. –

1 h

our

Gra

duat

ed p

rom

ptin

g +

in

struc

tion

x-

-

Dyn

amic

Tes

t of

Pote

ntia

l In

telli

genc

eG

ener

al

Indi

vidu

ally

14 it

ems

Gra

duat

ed p

rom

ptin

gx

--

DA

alg

ebra

skill

sSp

ecifi

c: m

athe

mat

ics

Indi

vidu

ally

30-4

5 m

in.

Gra

duat

ed p

rom

ptin

g (in

struc

tiona

l sca

ffold

ing)

x-

-

Non

verb

al D

A p

hone

me

dele

tion

Spec

ific:

lang

uage

Indi

vidu

ally

21 it

ems

Syste

mat

ic p

rom

ptin

g +

co

rrec

tive

feed

back

x*

-


47

2

Nar

rativ

e sk

ill a

sses

smen

tSp

ecifi

c: la

ngua

geIn

divi

dual

ly?

Extr

a in

struc

tion

(sent

ence

-im

itatio

n ta

sk +

pic

ture

cue

s);

gene

ral f

eedb

ack

x*

-

LEM

Gen

eral

In

divi

dual

ly

?R

epea

ting

item

s; no

nver

bal

feed

back

; dem

onstr

atio

n (g

radu

ated

pro

mpt

ing)

x

*-

GD

A m

athe

mat

ics

Spec

ific:

mat

hem

atic

sG

roup

wise

50

– 7

0 m

in.

Gro

up w

ise d

iscus

sion

and

instr

uctio

n on

pro

blem

-solv

ing

strat

egie

s and

thin

king

skill

sx

x-

Mat

hem

atic

al w

ord

prob

lem

s Sp

ecifi

c: m

athe

mat

ics

Indi

vidu

ally

?G

radu

ated

pro

mpt

ing,

feed

back

xx

-

Mat

hem

atic

al p

atte

rn-

reco

gniti

on ta

sks

Spec

ific:

mat

hem

atic

sIn

divi

dual

ly14

-56

min

. G

radu

ated

supp

ort

xx

-

DO

TC

A-C

hG

ener

al

Indi

vidu

ally

Up

to 1

,5 h

r. Pr

ompt

s; fe

edba

ck d

urin

g te

st-

--

ROC

F - d

ynam

ic a

dapt

atio

nG

ener

al

Indi

vidu

ally

?

Prom

ptin

g: st

ruct

ural

izat

ion

of

figur

e *

x-

DAT

MA

Spec

ific:

lang

uage

Indi

vidu

ally

15 it

ems

Gra

duat

ed p

rom

ptin

gx

--

NN

AT/D

AG

ener

alIn

divi

dual

ly?

Gra

duat

ed p

rom

ptin

gx

--

CIT

O M

onito

ring

Test

for

Mat

hem

atic

s – IC

T/D

ASp

ecifi

c: m

athe

mat

ics

Indi

vidu

ally

15 –

20

min

utes

Visu

aliz

atio

n an

d op

port

unity

to

use

exte

rnal

mem

ory

x*

-

Seria

-Thi

nk –

ada

pted

ver

sion

Gen

eral

: ind

uctiv

e re

ason

ing

Spec

ific:

mat

hem

atic

s

Indi

vidu

ally

3 se

ssio

ns o

f 15

-20

min

utes

(of

whi

ch o

ne se

ssio

n is

trai

ning

)

Gra

duat

ed p

rom

ptin

gx

x-

CHAPTER 2

48

Syllo

gism

sG

ener

alIn

divi

dual

ly

Tota

l adm

inist

ratio

n tim

e ra

nge:

42

– 10

1 m

in. (

mea

n tim

e: 7

1 m

in.)

Inte

rven

tion

time

rang

e: 1

1-28

Feed

back

afte

r pre

test,

than

te

achi

ng th

e sk

ills t

o so

lve

item

s. x

--

Sort

ing

Gen

eral

(exe

cutiv

e fu

nctio

ning

)In

divi

dual

lym

in.

(mea

n tim

e: 1

7 m

in.)

Teac

hing

skill

s to

solv

e th

e ta

sk

(visu

aliz

atio

n)

x-

-

Twen

ty Q

uesti

ons

Gen

eral

(exe

cutiv

e pr

oble

m-so

lvin

g sk

ills)

Indi

vidu

ally

Teac

hing

skill

s to

solv

e th

e ta

sk

x-

-

S-C

TP

Gen

eral

(wor

king

m

emor

y ab

ility

)In

divi

dual

ly±

150

min

. / ±

50

min

. (ab

brev

iate

d ve

rsio

n: 5

/6 su

btes

ts)

Gra

duat

ed p

rom

ptin

gx

--

HA

RTG

ener

al (a

nalo

gica

l re

ason

ing)

Gro

up w

ise

20 m

in. t

rain

ing

+ 2

0 ite

ms p

ostte

st Tr

aini

ng o

n sk

ills n

eede

d fo

r so

lvin

g ite

ms (

prom

ptin

g, e

xtra

in

struc

tion,

ver

baliz

atio

n an

d vi

sual

izat

ion)

xx

-

Seria

-Thi

nk In

strum

ent

Spec

ific:

mat

hem

atic

sIn

divi

dual

lyTe

achi

ng p

hase

: 30

-40

min

.In

struc

tion

on st

rate

gies

+

feed

back

xx

x

CAT

MG

ener

al (a

nalo

gica

l th

inki

ng)

Indi

vidu

ally

Teac

hing

pha

se =

30-

40 m

in.

Instr

uctio

n on

pro

blem

-solv

ing

strat

egie

s x

*x

CIT

MG

ener

al (i

nfer

entia

l th

inki

ng)

Indi

vidu

ally

12 it

ems e

ach

phas

e an

d 10

item

s tra

nsfe

r ph

ase

Extr

a in

struc

tion

+ in

struc

tion

on st

rate

gies

x

*-

Mac

CAT

-CA

Spec

ific:

adj

udic

ativ

e co

mpe

tenc

es (l

egal

un

ders

tand

ing)

Indi

vidu

ally

Onl

y 6

out o

f 8

item

s of

one

subs

cale

w

ere

dyna

mic

ally

ad

min

ister

ed

Extr

a in

struc

tion

on fa

iled

item

sx

--

Note.

Whe

re x

mea

ns e

xplic

itly

men

tione

d in

the

artic

le, a

nd *

mea

ns im

plic

itly

men

tione

d in

the

artic

le, a

nd –

mea

ns n

ot m

entio

ned

at a

ll in

the

artic

le.

C =

cog

nitiv

e le

arni

ng st

rate

gies

, MC

= m

etac

ogni

tive

lear

ning

stra

tegi

es a

nd M

= m

otiv

atio

nal l

earn

ing

strat

egie

s.? =

lack

ing

info

rmat

ion

in th

e ar

ticle.


49

2

and motivational content has been made. The majority of tests contains a learning

phase in which the examiner’s activities are oriented towards cognitive strategies only.

In a few instruments, metacognitive strategies are applied next to cognitive strategies.

These are, for example, activities explicitly aimed at planning behaviour or evaluating

one’s own activities and not primarily focused on the solution of task.

In the instruments CATM and Seria-Think (Lauchlan & Elliott, 2001; Tzuriel &

Kaufman, 1999; Tzuriel, 2000) a preliminary phase is included to familiarize the child

with the testing procedure and to observe potential difficulties (Tzuriel, 2000). Lauchlan

and Elliott (2001) report that the examiner addresses cognitive and affective factors

during the learning phase of the CATM. In the article of Tzuriel (2000) about the Seria-

Think Instrument it is mentioned explicitly that during mediation (learning phase) the

examiner gives feedback in order to assure the child’s feeling about competence, even

if the answer is not correct. This can be interpreted as an affective, or motivational,

strategy of the examiner. The remaining instruments do not describe motivational

factors other than some encouragements (e.g., Jeltova et al., 2011) which are not

equivalent to motivational factors as described in the introduction.

It must be mentioned, though, that these results are based on rather vague and implicit

information, in such way that most of the studies do not describe the metacognitive

or motivational aspects explicitly. Mostly, only cognitive aspects were mentioned

unambiguously.

Discussion

In an attempt to bridge the gap between diagnosis and intervention, which is nowadays

a reasonable quest in educational practice, this study examined the contribution of

dynamic assessment to this issue. Dynamic assessment is meant to be a promising tool

for estimating capacities of at-risk children and for distinguishing appropriately between

CHAPTER 2

50

children with learning disorders and children having learning delays (as a consequence of

their learning environments and former learning experiences). To date this is still one of the

challenges of educational practice. In spite of the advantages of dynamic testing procedures

they are, still, too rarely employed in practice due to the robustness (in research as well as in

practice) of traditional intelligence tests. However, a lot of dynamic assessment procedures

have been developed, demonstrating valuable psychometric properties and proving to be of

added value in the assessment of capacities of at-risk children compared to static intelligence

measures. Overall, they show a high predictive validity, and four studies (Elleman et al.,

2011; Swanson, 2011; Swanson & Howard, 2005; Tiekstra et al., 2009) even demonstrate

significant added predictive validity on top of IQ.

Over the last two years the contribution of articles about dynamic assessment to scientific

journals has been increasing: more than one-third of the articles found for this review was

published in 2010 and 2011. This trend also sustains in 2013, but was beyond the scope of

this article. The increased interest in dynamic assessment makes the necessity for review

studies on this subject evident.

Since dynamic assessment implies a teaching component during testing, the question

arose whether these dynamic assessment procedures provide valuable information for the

construction of interventions following the test, and could, thus, bridge the gap between

diagnosis and intervention. The outcomes of 29 dynamic instruments were examined with

regard to their consequential validity and presented at a continuous range, diverging from

proximal consequential validity to distal consequential validity. This means that outcomes

vary from only raw scores to more qualitative information. Although one could state that

qualitative observations during testing do provide a lot of information for intervention, it

appeared that none of the instruments resulted in explicit clues for practice. Next to raw

or standardised scores, there are tests that provide an indication of the level of help the

child needs. This can be seen as a more advanced type of consequential validity, since it

provides more information about the capacities of the child. Nevertheless, explicit clues

for practice are still unknown.


51

2

This outcome of the review study underlines the fact that, although dynamic

assessment procedures have potential, these procedures do not contribute to bridging

the gap between diagnosis and intervention in their current formats. The remaining

quest for bridging effects of dynamic assessment was also highlighted by Haywood

(2012). Dynamic assessment procedures that result in raw scores only do contribute

to proximal consequential validity, since an opportunity to learn is provided during

testing. However, in these cases, the tests will particularly be used to better estimate

learning capacities as opposed to traditional intelligence tests, and not as indicative

for the design of interventions or adaptations in the instructional process afterwards.

Hence, distal consequential validity is not warranted in current dynamic assessment

procedures. It should be mentioned, though, that it is likely that some of the articles

describe instruments that have been developed for research purposes only, and not (yet)

for psycho-educational practices. And, thus, do not describe consequences for practice

explicitly. This aspect could have caused some biased results.

Despite the present absence of distal consequential validity in procedures, attempts to

bridging effects are being made sporadically resulting in some qualitative information

for practice. However, the move to fully distal consequential valid procedures would not

be too far, and is actually near. The teaching component during testing provides the

opportunity to carefully administering the actions that have been taken during learning

phases. The analysis of actions of both examiner and examinee could be divided in, for

example, different categories (i.e., cognitive, metacognitive, and motivational). By explicitly

mentioning these actions during the test administration, outcomes will provide information

for practice. Outcomes of the testing procedures, then, not only indicate a score of learning

potential (or raw score), but also enhance systematic cues for setting up IEPs.

Therefore, in addition to outcomes, learning phases of each dynamic assessment procedure

have been scrutinized in this review. The content, i.e., the aims of examiner’s activities,

has been divided in cognitive, metacognitive and motivational strategies. Unfortunately, it

turned out that this information was barely explicitly mentioned in the articles. Nonetheless,

CHAPTER 2

52

a striking conclusion can be made: most of the tests apply cognitive and/or metacognitive

strategies during the learning phase, but motivational factors never played a major role

explicitly. As has been noted before by Borkowski, Carothers, Howard, Schatz, and Farris

(2007), self-regulation strategies should be integrated in dynamic assessment. This review

shows that progress still has to be made in this area. Only recently, Teo and Roodenburg

(2013) demonstrated the additive effects of including self-regulation strategies during an

assessment procedure. Moreover, in some dynamic tests general feedback is given (De Beer,

2010a; Gillam et al., 2011; Gummersall & Strong, 1999; Hamers et al., 1996; Jeltova et

al., 2011; Sternberg et al., 2002; Tzuriel, 2000), but the kind and type of feedback is not

adequately mentioned or registered. The extent to which examinees benefit from feedback

is an important factor in dynamic assessment. Woide, Beckmann, Elliott and Guthke (2005)

demonstrated that motivational, affective and non-intellective factors influence the effects

of feedback and that feedback increases the capacity to evaluate oneself. Although one of

the promising aspects of dynamic assessment approaches is the attention for non-intellective

factors (Lidz & Macrine, 2001), this study demonstrated that there is no explicit focus on

these factors during learning phases. This is an interesting finding, since it is generally known

that motivational factors interact with the learning process (e.g., Ryba, 1998) and test results

(Meijer, 2001). Moreover, one of the groundings of dynamic assessment is that during

testing social interaction takes place in order to provide a more motivating testing

situation, which is less biased to e.g. test anxiety.

These aspects should, thus, play a role during learning phases. The type of feedback

that has been given should be made explicit in the test result which will provide useful

information for the instructional design in the classroom or for the design of individualized

interventions. Motivating strategies of the examiner, like effort stimulation or explicit

feedback on the performance of the task, have been demonstrated to result in different

impacts on aspects of learning, such as task interest and self-efficacy respectively (Perry,

Turner, & Meyer, 2006; Prince, Minnaert, & Opdenakker, 2014)


53

2

These suggestions also underline the interdependency of cognitive, metacognitive

and motivational strategies during learning processes. These aspects are indispensably

related to each other, and thus, each of these aspects should play a prominent role

during test administration, and, consequentially in the test result. Only in this way,

an ecologically valid contribution to the consequential validity of dynamic assessment

procedures can be made.

Of course, this relates to questions about the test duration, domain specificity and

age group of the tests. Each of these variables has its implications for the question

of consequential validity. However, the overall lack of motivational components and

explicitness of learning phases, underlines the importance of qualitative reports about

the interplay between examiner and examinee during test administration, in addition to

raw scores if applicable. This implies that examiners should be reflective by nature on

both their own activities and the activities of the examinee.

There are, however, some important limitations while interpreting the outcomes of this

review. Firstly, the literature search has been carried out in a database (Web of Science)

which only includes peer reviewed journal articles. Consequently, articles which have

been published in non SSCI journals are not warranted, as well as book chapters and

dissertations. This implies a certain restriction to the scope of articles included in this

review.

Secondly, there is no consensus about the definition of dynamic assessment in the

field. Moreover, different approaches exist (standardised vs. clinical), which may cause

confusion in research as well as in practice. Next to the robustness of traditional tests,

this could contribute to the suppression of dynamic procedures in practice. Therefore

it is suggested to use clear distinctions between dynamic assessment and intervention

(see also Hessels-Schlatter & Hessels, 2009). In addition to these authors, we would like

to maintain these distinctions to promote a clearer concept of dynamic assessment.

Subsequently, in this article, the term dynamic assessment has been defined as a

CHAPTER 2

54

dynamic procedure with a standardised approach in which a learning phase is included

in testing. This form of dynamic assessment is comparable to static testing. To our

opinion, dynamic assessment does not contain an intervention phase expanded to

multiple sessions during several weeks. In the latter, one is testing the effectiveness of an

intervention, instead of making an evaluation of a child’s capacities.

Conclusion

To conclude, dynamic assessment remains a promising tool to bridge the gap between

diagnosis and intervention, but there is still room for improvement. Due to a learning phase

included during the testing procedure, qualitative information can be revealed. This, in turn,

provides an estimation and evaluation of the types of information that are essential to the

child in order to complete a task. Eventually, this could contribute to the development of an

adaptive intervention, which will provoke better learning results of the child.

Implications for practice. In order to establish this, however, the focus on consequences of

assessment for educational practice should hold a more prominent place in the development

of dynamic assessment procedures in future. There is a need for more explicitness of learning

phases and types of feedback, in order to construct student-tailored interventions. Moreover,

motivational factors should be highlighted during learning phases. Students encountering

learning difficulties generally have experienced a lot of negative emotions and failure, and

dynamic assessment procedures are the more promising if they could anticipate on these

factors pre-eminently. If the learning phase is administered and documented carefully, the

consequential validity will be more warranted, resulting in explicit clues for practice.

Additionally, the authors would like to make a plea for a more prominent place of

consequential validity in test evaluation reports. When evaluating the quality of tests, it is

important to focus on consequences for practice, next to internal validity criteria.


55

2

From a researcher’s point of view, future research should focus on the bridging effect of

dynamic assessment procedures to intervention practices. This could be done, e.g., by

analysing qualitative (video) observations and focusing on motivational components during

assessment procedures and consequences for educational practice. While investigating the

distal consequential validity of dynamic assessment procedures, research contributes to

practice, which is an important aim of science.

If the gap between diagnosis and intervention can be bridged, the quality of psycho-

educational practices will be elevated. Since a careful administration of the learning

phase provides useful information for interventions or instructional processes in the

classroom, professional actions will be more grounded on empirical results. Educational

psychologists play a prominent role in this process, being the core communicator

between test result and interventions. If tests provide helpful clues, next to raw scores,

the quality of interventions and instructions will increase. Only in this way the learning

outcomes of students will be optimized.

“You can never solve a problem with the same kind of thinking that created the problem in the first place” (A. Einstein)

Chapter 3Can didactic resistance be measured by using

the Response-to-Instruction model?

A pilot study

This chapter is based on:Tiekstra, M., Prince, A., & Minnaert, A. (in prep.).

Can didactic resistance be measured by using the Response-to-Instruction model? A pilot study.

CHAPTER 3

58

Abstract

Didactic resistance is viewed as an important criterion in the definition of learning

disabilities. This study explored whether the Response-to-Instruction (RTI) model can be

an instrument to measure didactic resistance. Using Curriculum Based Measurements

(CBMs) the interaction between teacher and students can be monitored. This qualitative

multi-method study (N = 11 schools) demonstrated that in order to be an instrument to

assess didactic resistance, several aspects need to be taken into account before, after and

during the process of implementing the RTI model. Data triangulation, illustrated by

quotations from the field, revealed the interrelatedness of process and content on the one

hand, and change of beliefs and behaviours of all educational staff members on the other

hand. The results showed that there is an urgent need for a dynamic view on learning

and instruction when regarding the RTI model as an instrument to measure didactic

resistance. Besides, didactic resistance should be measured at both student and teacher

level. Theoretical and practical implications will be discussed.

DIDACTIC RESISTANCE AND THE RTI MODEL

59

3

Introduction

The diagnosis of learning disabilities has been a major issue in debates over the years

(Hallahan & Mock, 2003). Not only the way to define, but mainly how to diagnose and how

to respond to the diagnosis of a learning disability in practice is still surrounded by vagueness

(Büttner & Hasselhorn, 2011). Traditionally, the discrepancy between IQ and actual (school)

achievement was the decisive factor. In this, generally, a child is identified as learning disabled

when scores on achievement tests at school show a large discrepancy with a child’s IQ (as

a result of a standardised intelligence test). A child which does not perform according to its

potential (operationalised as IQ score) should be identified as learning disabled (Hallahan,

Kauffman, & Pullen, 2012, p.140). This approach of learning disabilities has been criticised

for inaccurate judgments (Francis et al., 2005; Stuebing, Barth, Molfese, Weiss, & Fletcher,

2009) and can be seen as a “wait-to-fail” model (Hallahan et al, 2012). In an attempt to

overcome these difficulties with identification of learning disabilities and to provide more

early identification, the Response to Instruction model (RTI) has been developed in the

United States. According to Fuchs, Fuchs, and Vaughn (2008) the discrepancy between

estimated response to instruction and actual response to instruction is examined in RTI.

Progress and results of each student are monitored systematically, which provides an insight

to specific difficulties of the individual child. In this, teachers are able to estimate whether

falling performances of a student are due to either a deficit in the student or inappropriate

(i.e., not tailored to student’s capacities) instruction of the teacher. Although criticised by

several authors (e.g., Kavale, 2005; Reynolds & Shaywitz, 2009) because of its lack of

demonstrated empirical effectiveness, the RTI model can be seen as an alternative model

to identify learning disabilities. But, in line with Kavale (2005) together with Spaulding

and Beam (Kavale et al., 2009, p.39) the model does not provide a definition of learning

disabilities. However, the RTI model can officially be used to identify children as learning

disabled in the United States since 2004.

Consequently, the definition of learning disabilities needs to be adapted, as is the case

in the new edition of the Diagnostic and Statistical Manual of Mental Disorders (5th

CHAPTER 3

60

ed.; DSM-5; American Psychiatric Association, 2013). More attention will be drawn to

clinical based data instead of the discrepancy criterion. It remains, however, unclear

what the definition is exactly (Scanlon, 2013) and how it will be applied in practice.

Generally, a somewhat broader definition of the IQ-achievement discrepancy approach

is used to diagnose learning disabilities. This definition encompasses three criteria for

identifying learning disabilities (see also Tannock, 2013). Firstly, student’s performances

are below what is expected. In this, a measure of IQ is included, but in contrast to what

was mentioned before, background variables are taken into account when interpreting

IQ scores. Secondly, a criterion of didactic resistance is included. This means that

students who do not profit from (tailored) instruction show a certain persistence of

difficulties. And thirdly, the criterion of exclusivity, problems are not caused by other

impairments like hearing or visual impairments. This broader definition, which

has been adopted by the DSM-5, thus, reflects an extension to the endorsement of

individual differences in one classroom instead of general, one-pattern-fits-all, group

wise education. It remains, however, unclear how the above-mentioned three criteria

should be measured (Cavendish, 2013).

Didactic resistanceAs mentioned, the construct didactic resistance is one of the three main elements in

the determination of learning disabilities. This is an interesting construct, however,

little research has been published about the interpretation and measurement of this

construct as was revealed by a recent search in the EBSCO host database. It seems that

a grounding for a thorough understanding of this construct is lacking, as is the empirical

grounding for the psychometric characteristics: how should we interpret and measure

didactic resistance? In practice, students are regarded as didactically resistant if they are

unresponsive to quality instruction, i.e., the problems persist despite tailored teaching.

As such, the problem is stated at the student level, as are the measured outcomes. Since

an indication of the quality of the teaching provided remains unmentioned, this seems

a rather limited interpretation of the concept of didactic resistance. To illustrate, one

could state that didactic resistance not only implies a deficit in the student, but also


61

3

a deficit in the teacher (or the teaching method), since the teacher is not capable to

adapt instruction to a level that fits the level of the student. In practice, regrettably,

instruction is not as dynamically interpreted or implemented as assumed in the criterion

on didactic resistance. Instruction, after all, involves multiple partners, as is emphasised

by the theory of Vygotsky (1978), and should be regarded from a dynamic perspective in

which teacher and student play reciprocal roles (Steenbeek, Jansen, & Van Geert, 2012).

Timperly and Parr (2009) underlined the importance of a rather dynamic view on

instruction, as was demonstrated by their research on the influence of different teaching

strategies on students’ learning. Additionally, from a more general point of view, Chak

(2001) emphasised the importance of reflection upon adults’ sensitivity to children’s

learning in optimising one’s zone of proximal development. This applies for teacher-

student interactions as well. Hence, didactic resistance should not only be measured at

student’s level, but also at teacher’s level.

According to the above-mentioned, the criterion of didactic resistance could be

measured during a RTI approach. Due to regular adaptation of their instruction to

the specific needs of their students, teachers are able to monitor progress and highlight

problems of each student. Moreover, the careful progress monitoring provides an

insight in needs and, consequently, tailored instruction can be developed. In this way,

teachers should be more aware of their own teaching practices on students’ learning.

In contrast to intervention programs aiming at individual outcomes, for example a

certain instruction method to deal with learning difficulties, the RTI model can be seen

as an organisational model. It includes an adaptation of administering and teaching,

which, in turn, demands an adaptation of the whole organisation (in this: school). The

core assumption of the RTI model is that every child reacts differently to whole-class

teaching. Due to progress monitoring of each student, the extent to which each student

is sensitive to instruction can be estimated. When a child does not show enough progress

(compared to the mean progress of the group) this child is offered a kind of intervention.

The model consists of three levels of instruction (Fuchs & Fuchs, 2006). In the first tier,

students are offered general evidence-based classroom wide instruction. Students that

CHAPTER 3

62

do not show enough progress compared to their classmates receive extra instruction in

small groups several times a week; this is tier two instruction. If extra instruction does

not provide enough progress, an individualised route is set up for these students (tier

three). Other RTI models exist, though, with variations in the number of instruction-

levels. These models, however, are still based on the core model of Fuchs and Fuchs

(Kavale & Spaulding, 2008). The progress-monitoring is carried out with curriculum-

based measurements (CBMs; Espin, Wallace, Lembke, Campbell, & Long, 2010).

CBMs are closely related to the curriculum taught, and, therefore, provide insight into a

student’s response to current instruction in a specific domain. Due to several aspects of

the model, such as progress monitoring by CBMs and the differentiation of instruction

in tiers, the model delivers valuable information for planning instruction.

In practice, also when working with the RTI model, didactic resistance is often

interpreted as a static feature of the student. However, it should rather be regarded as

a dynamic concept of the interaction between student and teacher. In the RTI model,

theoretically, the interaction between teacher and student is monitored by the CBMs.

Since the CBMs are closely related to the current instruction, effects of the instruction

on student’s outcomes can be measured. Nevertheless, the instruction component in

the interpretation of the CBMs is often ignored and an emphasis is placed upon the

outcomes of the student solely. This results in, again, a rather static interpretation, and

resembles the IQ-discrepancy approach in identifying learning disabilities. Therefore,

contrary to the more standardised approach as favoured by many researchers, CBM

outcomes should not only be interpreted quantitatively, but also qualitatively. The

errors made in the CBM should not only be counted, but also analysed by the teacher

to gain insight in the mistakes that were made. Only in this way quality teaching can be

warranted, because instruction is tailored to a student’s needs.

Process of implementing the RTI modelIn theory, the RTI model seems a very promising instrument to measure didactic

resistance. However, when implementing a model to practice multiple factors come into


63

3

play (for an overview see Prince, Tiekstra, & Minnaert, 2014) that influence the outcomes

of the implementation process. A lot of (US) literature describes the core principles,

ideas, features, promises of, and experiences with the RTI model. It is still, however,

difficult to find literature about the effectiveness of implementation of the model (Fuchs

& Vaughn, 2012). This results in different models and different operationalisation of

each model in each context, which contributes to confusion surrounding RTI (Fuchs,

Mock, Morgan, & Young, 2003, p.159). Research often focuses only at one aspect of the

RTI model and the enormous amount of literature is difficult to overview in order to

get guidelines for practitioners with respect to its implementation (Jones & Ball, 2012).

Information for teachers on how to effectuate differentiated instruction in tiers remains

generalised and ambiguous (Fuchs & Vaughn, 2012). Moreover, specific information

about training or coaching during implementation of the RTI model is hard to find.

Ikeda (2012) stressed the fact that the focus is rather on decision making processes in the

model than on the content and effectuation of support for teachers.

The way information is implemented plays an determining role, next to the content

of what is implemented (see also Prince et al., 2014). In their qualitative case study,

Datnow and Castellano (2000) described the process of implementation of a school

wide program and illustrated the different outcomes between teachers. They touched

upon this important distinction between process of implementation and the content of the

model that is implemented. Generally it is underlined that engagement of teachers in

developing the program favours the implementation process. Although this was not the

case in Datnow and Castellano’s research, teachers still implemented the program since

they appreciated its content.

Furthermore, a main aim of implementation lies in the change of behaviour and

belief of participants. The Theory of Planned Behavior (Ajzen, 2005) describes the

interrelatedness of changing belief and behaviour. Beliefs have their influences on

intentions that precede behaviour. More particular to the classroom practice, Bandura

(1993) highlighted the importance of teachers’ beliefs and the influence of these beliefs

CHAPTER 3

64

on their behaviour in the classroom and quality of their teaching: “teachers . . . beset

by self-doubts construct classroom environments that are likely to undermine students’

sense of self-efficacy and cognitive development” (p. 140). Teachers’ beliefs, in turn, are

affected by experiences. This illustrates the interrelatedness of belief and behaviour

when implementing new models into the classroom. Moreover, as underlined by Van

de Ven, Polley, Garud, and Venkaraman (1999), implementation is a cyclic process. The

above-mentioned factors have an impact on the methodology used for this study. In the

Method section, therefore, the used methodology will be carefully described.

Aim of the present studyWhilst theoretically grounded the RTI model seems to be a promising instrument to

measure didactic resistance, this article deals with the issue whether the RTI model

can function as an instrument to measure didactic resistance in practice. Thence, the

implementation of the RTI model will be studied carefully, as the way of implementation

will have a major influence on the outcomes of the study. As was mentioned before, the

RTI model can be regarded as an organisational model. This has several consequences

for the implementation of the model. One could state that it can only be implemented

school wide and on all levels, from teacher to educational support staff. Since the goal

of this study is to provide detailed qualitative information about the possibility to assess

didactic resistance, rather than information about the implementation, the model has

been implemented at one academic grade only.

Method

Participants and settingIn 2011 a pilot study has been conducted which attempted to implement the RTI

model partially at a cluster of schools in the Netherlands. Care for children having

learning problems is organised in a stratified manner in the Netherlands, which implies

(theoretically) clear roles and responsibilities for all involved. In practice, decisions


65

3

are usually made democratically. In this particular cluster of schools the care process

involved one school psychologist who led 11 senior coaches, of whom each worked

at multiple schools. These senior coaches supported remedial teachers (one remedial

teacher per school). Remedial teachers supported the teacher and supplied information

and advice when the teacher asked for it. It was decided that one school per senior coach

participated in the study, resulting in a total of 11 schools that participated initially.

In the region where this cluster was situated, people generally speak a strong dialect,

which might influence students’ outcomes. The cluster of schools, however, can be

regarded as an ordinary cluster of public schools (comprising religious and non-religious

schools), with a non-deviant proportion of minority students. Participation to the pilot

study was voluntary. A notable fact was that educational staff (all participants) defined

themselves as being progressive teachers, since they attended professional courses

frequently. Attending professional courses is one of the requirements of inspectorate

and government. Next to this, quality of schools is assessed by students’ outcomes on

nationwide tests and visits in the classrooms by inspectorates.

InstrumentsThis study had a multi-method qualitative design. Questionnaires, learning reports,

interviews, and observations were qualitatively interpreted to provide data and are

described in detail below.

Questionnaires were constructed for senior coaches, remedial teachers and teachers.

The questionnaire for senior coaches and remedial teachers (SQ) was based on the

questionnaires that Meijer and Tijhaar (1995) developed in their studies around teacher

consultation and professionalisation. It described six phases in the implementation

process of the RTI model; introduction, problem identification, analysis of problems,

solution seeking, implementing solutions, evaluation. Participants had to rate on a

5-point scale to what extent they felt competent about each phase. The questionnaire

for teachers (TQ) differed, since items were more related to the classroom level, such as

CHAPTER 3

66

“at the beginning of the school year, I do a screening to assess each student’s reading

level.” Ratings were also on a 5-point scale. The aim of the questionnaire was to get an

idea of the current situation (with regard to RTI) at the schools and whether participants

knew how to translate knowledge.

The learning report (LR) consisted of five open questions. The learning report was

administered to senior coaches and remedial teachers during one training session to

detect possible bottlenecks in the implementation process. Moreover, we wanted some

insight in the opinions and experiences with the model until that moment.

Interviews took place with some of the senior coaches (SI), remedial teachers (SI) and

teachers (TI). Interviews were unstructured, but focused at gathering personalised

information about the implementation process and the strengths and weaknesses of the

RTI model. Some took place during training sessions, others in the schools, others at the

moment that a school dropped out of the project.

The first author (an educational psychologist) observed every training session and

additional meeting (OT) in order to monitor the implementation process closely.

She also visited three schools to observe how the RTI model was implemented. The

observations in the classroom (OC) took approximately 30 minutes in which the teacher

had planned reading instruction. In this way, some insight could be provided in how the

different tiers were organised in one class and how the RTI model was implemented

on a classroom level. The observations, thus, were only completed at the schools that

finished the project.

ProcedureThe planning was to provide participants with a training of two consecutive days at

the start of the academic year. Furthermore, several additional meetings were planned

during the project. For this, a training of several days was developed in order to provide

senior coaches and remedial teachers with knowledge about the RTI model and tools


67

3

to translate this knowledge to their teachers. During the six months project, every four

weeks a meeting was organized to coach, assist and encourage the senior coaches and

remedial teachers. In this way, the implementation process could be monitored carefully.

The questionnaire (SQ) was administered prior to the first (and after the last) training

session in order to measure the participants’ knowledge of the RTI model. Teachers’

questionnaires were distributed via remedial teachers and were collected between

second and third additional meeting. Next to the questionnaires, the learning report

was administered to remedial teachers in order to detect potential struggles during the

course of the project.

Data collection took place at certain fixed points in time (T1 to T6) that were planned

beforehand, but also in between (not planned). An overview is presented in Table 1.

Table 1 | Sequence of data collection during the pilot study

Instrument T1 T2 T3 T4 T5 T6

SQ x x

TQ x x

LR x

SI x x

TI x

OT x x x x x x

OC x x

Note. T1: Start training (2 days); T2: additional meeting 1; T3: additional meeting 2; T4: additional meeting 3; T5: additional meeting 4; T6: additional meeting 5

CHAPTER 3

68

Information input: trainingTraining was provided for senior coaches and remedial teachers, due to pragmatic reasons:

since it already took a great effort to assemble them, it would have been even more difficult

to assemble all teachers. It also resembles the way in which care is organised in schools in the

Netherlands: these people support teachers and provide them with new insights during the

school year. The goal of the training was to provide knowledge about the RTI model and

to provide tools and skills to facilitate the implementation process. The participants needed

to translate their knowledge to their teachers, which, again, resembles the normal process

in the Netherlands. Training took two days prior to the whole RTI project, and had five

additional meetings.

The RTI-training was developed by commercial training experts in collaboration with an

educational psychologist (the first author) to guard scientific input and to watch over the

implementation process as a whole. The RTI model is an abstract model and one of the

difficulties in developing the training was translating the information to such a practical

level that participants would understand the core ideas of the RTI model. This implied

the selection of one grade in which the model would be implemented instead of a school

wide implementation (multiple grades), since constructing CBMs for all grades would take

too much time. Grade 2 was selected to implement the RTI model, since severe learning

difficulties can be detected at these ages.

CBMs were constructed in advance by the trainers, to relieve teachers of another task. Once

a week, students were administered individually a CBM by their teacher or support staff

(i.e., remedial teacher or classroom assistant). The CBM consisted of a one-minute reading

out loud test and each week another part of the same story (with reading level appropriate

for end grade 2) was used. The number of correct words could then be calculated and

the errors made could be identified. A quantitative as well as qualitative analysis of the

errors was emphasised. An Excel-tool was developed to register progress and calculate dual

discrepancy.


69

3

Data analysisA qualitative data analysis was used, since the method should be sensitive to contextual and

situational characteristics in order to determine implementation effectiveness. In line with

the principles of Interpretative Phenomenological Analysis, data analysis took place in two

stages (Bazeley, 2013). First, data have been categorised with help of the ATLAS.ti program

(version 6.2) in order to establish thematic analysis. Second, from a more phenomenological

point of view, the researcher (first author) interpreted the data subjectively and in line with

participants’ experiences.

For the trustworthiness of data (Lincoln & Guba, 1985) the data has been triangulated

from the multiple data sources mentioned above. In each step of the data analysis peer

debriefing was established by the second and third author to obtain objectivity and reliability.

Subjectivity of first author, however, remains important, and has been highlighted in the

Results section when needed (Altheide & Johnson, 2011).

Results

In this section the researcher’s perception of the data and data of the researcher’s perception

will be listed with an emphasis on data and perceptions that are needed to answer the

research question Is the RTI model an instrument to measure didactic resistance? The results have

been structured according to the chronological sequence of events during the project.

Therefore, a division has been made between initial training, the five additional meetings

as follow up sessions, and eventually, observations at the schools that completed the project.

Each description of these phases comprises a short description of core events, highlighted

by quotations from the field, and the authors’ reflections based upon these. Generally, these

reflections relate to the division between implementation process and content of the RTI model

and the division between change in belief (point of view) and behaviour which appeared to be

core elements as revealed by the thematic analysis with ATLAS.ti.

CHAPTER 3

70

Initial trainingIt can be stated that the content of the RTI model and the process of implementation are

two separate factors in this pilot study. As pictured below, the training needed to be adapted

to the specific needs and situation of the targeted schools. This is a rather common situation

while implementing interventions. However, after the concessions to participants’ needs

(principally dictated by logistical situations), the implementation process of the RTI model

still appeared to be problematic during the follow ups.

Observations during the initial training revealed that participants did not agree with a new

screening at the beginning of the school year as several of them remarked: “this takes too

much time, and we have already done this at the end of previous grade”. Due to this refusal

of participants, the training reached at an impasse. Although the questionnaires showed

that there was little knowledge about the RTI model beforehand (scores 1 and 2, out of

5), there was no need for knowledge about the RTI model during the training session. In

order to continue the project, the implementation of the RTI model had to be adapted to

some requirements of the participants. The first condition of the participants was “no extra

screening at the beginning of the school year”. Secondly, “differentiation of instruction right

at the start of the school year instead of first 6 weeks of same instruction to all students”.

The researcher states that, as a consequence, the idea of differentiating between responding

and non-responding students to the current teacher’s instruction could not be determined.

However, she hoped that several weeks of implementation and working with the model

would change participants’ view and, thus, understanding of this idea.

After their first refusal and the concessions towards their requirements, some participants

started to see the additive value of the RTI model in such way that poor readers would

be detected earlier in the instruction process. These participants highlighted the need for

support of school administrators when implementing the model. The researcher thinks that

this illustrates rather a focus on implementation than on content of the training, since this

aspect never played an explicit role during the rest of the project. Although, 8 schools agreed

to continue the pilot study after this impasse, the researcher still had some doubts whether


71

3

the participants shared the view about the RTI model as was aimed for at the start of the

training.

Follow upsIn addition to the two days of initial training 5 meetings were planned in which, originally,

key moments during the process could be discussed in order to monitor the implementation

process. It appeared, however, that these meetings needed to be used for more explanations

and information input to the participants. Apparently, the core principles of the RTI model

were, still, not understood.

Process of implementation: unchanged behaviour and unchanged belief.

During the follow ups, serious discussions emerged about the general instruction and

screening of all students, and even caused some drop outs of the project. The researcher

thinks this is a remarkable fact, since decisions and agreements had been made about these

points during initial training. By the third additional meeting, already three schools had

refrained from participating in the project, of which only one school informed the trainers

officially. One of the reasons this school mentioned while quitting the project was: “we did

not want to stop our own instruction and intervention method”, and it appeared that after

four weeks of tier 1 instruction (and CBMs) they had already stopped the whole project.

Several weeks later, after additional meeting 4, another two schools dropped out, of which

one mentioned their problems with the general classroom instruction: “we’ll never going to

abstain children from extra instruction again”. According to the researcher, this reflected the

severe resistance to change and a certain fear to adapt their current instructions to outcomes

of the CBMs. It is likely that participants stick to their own (safe) intervention method. These

participants dropped out of the implementation process since they did not (want to) change

their behaviour. The quotations illustrate their unchanged beliefs about the role of the teacher

in didactic resistance. Since they continue with their current differentiation in reading instruction

and extra interventions, based on outcomes of the previous grade with instruction from another

teacher, it can be stated that they had a rather static view of didactic resistance.

CHAPTER 3

72

Process of implementation: changed behaviour but unchanged belief. First things

to implement were the CBMs, i.e. progress monitoring. Due to the precise instruction of one of

the trainers, the researcher expected that administration of CBMs could be effectuated easily.

However, a returning point in discussions with participants during the additional meetings and

interviews was the planning of the administration and the time it took. These issues relate to

the implementation process, rather than discussion regarding the content of the RTI model.

For instance, one participant remarked that “administering CBMs takes too much time, what

is the added value of this? We already knew which students needed extra instruction.” The

administration of the CBMs, appeared, thus, a rather negative assignment than an information

tool for teachers to adapt their instruction to the specific needs of their students. Later on in

the project participants became more positive about the CBMs, as the graph provided by the

(CBM-) tool was rated as “useful” and “motivating for students as well as teachers” according

to the learning reports and interviews. For the researcher it remained questionable whether the

discussions can be attributed to the real negative experiences in practice with the CBMs or rather

to a resistance to change or refusal of the participants. These participants implemented the

CBMs in their classroom practice, and changed their behaviour. The hesitations of participants

and quotations, however, reflect a lack of dynamic view on didactic resistance, since, again, the

decisions based on the previous grade remain very important. The changes in behaviour are more

related to the process of implementation, rather than to the content of the RTI model. These

behavioural changes will be highlighted in the next section, whereas belief remains unchanged.

RTI content: changed behaviour but unchanged belief. Aspects of the RTI model

that are crucial in determining whether it can be used for measuring didactic resistance

needed to be examined in detail. With regard to the interpretation of CBM outcomes, none

of the participants had problems, as was demonstrated in the learning reports and in the

observations during training. The researcher was surprised by this fact, since she estimated

that the selection moments would be crucial moments, and maybe might cause problems

as she experienced during the development of the training. As demonstrated in interviews

and observations, the interpretation of the CBM outcomes by the participants, however,

focused mainly on students. Besides, although never explicitly mentioned, the connection


73

3

to teacher’s instruction appeared to be very difficult. Observations, learning reports and

interviews revealed that it was unclear which interventions one should use, when to effectuate

these interventions next to the current (tier 1) instruction, and how to link these interventions

to the errors made in the CBMs. Participants were, for example, unsure about the content

of tier 2. They expected the RTI model to provide interventions, but the model rather aims

at activating the teacher to search for appropriate interventions that suit the outcomes of

the CBMs. The mentioned examples reflect a change in behaviour, but unchanged view on

didactic resistance. This is best illustrated by the focus on students’ outcomes solely instead

of the link to quality of instruction. The researcher thinks that participants (senior coaches

and remedial teachers), perhaps, tried to avoid any accountability on behalf of their teachers

or themselves, as a kind of self-protection. The change in belief, still, stays behind. Despite

these negative experiences, there were still some schools that continued the project and,

even, finished it in a very promising way.

Observations in practice Finally, three schools completed the whole project. These three schools have been paid

special attention to, manifesting in observations in the classrooms, and extra interviews with

participants in the field, i.e., teachers and remedial teachers. One school implemented the

RTI model, but still under some resistance. In the classroom it was observed that the teacher

had difficulties in combining the current reading method with different interventions, and,

thus differentiating between students. Afterwards the remedial teacher mentioned that she

did not see the value of the RTI model: “Our current reading method functions well, why

should we replace something that is going well? Why do we need to do this pilot study?” As

mentioned before, maybe this resistance has to do with the different implementation levels:

if the remedial teacher does not understand the core message of the RTI model, then, it will

not be communicated to the teacher and it will not be executed in the field properly. Bearing

the research question in mind, the RTI model does not function as an instrument to measure

didactic resistance in this school.

CHAPTER 3

74

Another school implemented the RTI model properly, but according to the observations

and interviews it can be noticed that there was only a change in behaviour and not yet

a change in belief. The quotation “we are very positive about the RTI model, since

students are more engaged during instruction”, indicates that outcomes are focused at

a student level, rather than at student and teacher level. The CBMs were administered

and interpreted by a classroom assistant. The remedial teacher made decisions, while

the teacher looked for interventions that match these outcomes herself. But, to what

extent are these outcomes representative for the instruction the teacher gives? In order

to provide real tailor-made teaching, the communication between these partners should

be good and regular. In this situation, according to the research question, the RTI model

can be a tool for didactic resistance, but communication should be open and honest,

in order to achieve congruence between CBMs interpretation and tailored instruction.

The third school was an excellent example of a well-implemented RTI model. The

teacher interpreted the CBMs herself, and was supported by the remedial teacher in

searching for appropriate interventions. Although, this had caused some uncertainty in

the teacher at first, she mentioned afterwards that she felt more capable in providing

instruction that fits her students. The teacher reported that she was more aware of what

students learn and how she effectuates her instruction. Furthermore she remarked “I

leave the instruction method in order to search for instruction that the specific student

fits.” Next to these outcomes on teacher level, the teacher observed “a large progress in

students and a greater engagement during instruction.” The teacher concluded “I am

going to use the graphical representations of student’s progress in parental consultations

and I will continue working with the model in future. I am going to develop CBMs by

myself.” This demonstrates a change in behaviour and belief, since the teacher related

the effects of her instruction to the progress of students. In this example, the RTI model

is used as an instrument to measure didactic resistance.


75

3

Discussion

This study investigated whether the RTI model could be an instrument to assess

a student’s didactic resistance. It appeared that it can be a valuable tool, if several

conditions are met. These conditions relate to the implementation of the RTI model,

and focus especially at the dynamic view which is needed in order to estimate a student’s

didactic resistance. As demonstrated by this pilot study, the implementation and the

content of the RTI model are two separate factors, but are intricately related. For

example, the schools that dropped out of the project did not support or understand

the ideas of the RTI model, but also showed a high resistance to change which occurs

frequently during implementation of any kind of new model or method. Next to these

factors, the change of behaviour and change of belief are important aspects to take

into account when implementing the RTI model. Belief and behavioural change can

be seen as separate factors, but at the same time are interrelated. This can best be

illustrated by great differences in the group of schools that finished the project. The

schools that implemented the RTI model only superficially did change their behaviour,

but did not change their belief in line with the RTI model. These participants still had

a strong focus on students’ outcomes and experienced great problems in linking the

CBM outcomes to their instruction. The change of behaviour and belief is of utmost

importance when implementing the RTI model, as demonstrated by the third school,

in which the teacher recognised her role during the teaching process and tried to reflect

on her teaching. In this case only, the RTI model can be used as an instrument to

assess didactic resistance. The interrelatedness between change of belief and change

of behaviour demands, however, further investigation. Longitudinal studies should

focus on the development and nature of the relationship between behaviour and belief

change. This is important when translating abstract models to practical skills and should

be taken into account when developing (RTI) training procedures.

The pilot study showed the necessity of a dynamic view on instruction and learning

when implementing the RTI model. Teachers can only adapt their instruction if they

CHAPTER 3

76

recognise their influences on learning of students. It is important not to rely only on

outcomes of standardised tests when identifying learning disabilities. The amount and

quality of teaching a student receives and has received in the past is a more determining

factor. After all, from a Vygotskian perspective, learning occurs when teaching takes

place in the zone of proximal development (ZPD). If teachers do not teach in a student’s

ZPD, one could state that instruction is not tailored to the student’s needs and learning

problems occur. Hence, the resistance criterion should not only be measured at the

student level, but also the received teaching should be assessed carefully. Consequently,

with respect to the interpretation and measurement of the concept didactic resistance, a

dynamic view on instruction is needed. Teachers influence the results of their students,

and students influence the way of teaching (Steenbeek et al., 2012). With Curriculum

Based Measurements these interactions can be monitored, on the condition that these

will be interpreted quantitatively and qualitatively.

The RTI model is often described as a theoretical, organisational model, since specific

intervention or implementation strategies remain unclear. As a consequence, different

versions of the model have been implemented. These versions differ in number of tiers

of instruction, but also in the way progress is monitored. Models exist in which progress

is monitored on a daily base, once a week or even once a month. Moreover, the way

CBMs are constructed can also be different: based on more standardised nationwide

tests, or constructed by the teacher. Bearing the research question in mind, if didactic

resistance needs to be measured, the CBMs should be administered at least once a

week, and the CBMs should be developed by the teacher in order to be able to assess

the effects of their current teaching. This is not in line with what Vaughn and Denton

(2008) proposed. They stated that “Teacher-developed measures are useful to inform

instruction, but they may lack the necessary precision to accurately identify students’

RTI” (p.66). Although this is completely true from a researchers’ point of view, it also

reflects a standardised view of didactic resistance. A qualitative interpretation of CBMs

constructed by the teacher, supported by a reading specialist or school psychologist,

should provide more reliable information about a student’s didactic resistance than


77

3

standardised measures interpreted only quantitatively. Besides, as they mention

“students’ RTI”, teacher-developed CBMs are important pre-eminently, since only in

this case an estimation of a student’s didactic resistance can be made.

Moreover, if teachers construct the CBMs themselves, their teaching becomes visible

and empowers them in their reflections upon their own teaching. In order to achieve

sensitivity to a student’s zone of proximal development (which is necessary to achieve

better learning outcomes due to a better fit to the student’s capacities) teachers should be

open for advice and reflective upon their teaching practices (as also slightly emphasized

by Chak, 2001). As a consequence, in teacher training and education teachers should

acquire skills to reflect upon their own teaching, instead of only focusing at the teaching

of a particular instruction method. Also, communication between partners in a team is

really important. To reflect upon one’s teaching peer consultation could be a valuable

tool. This is a premise for better implementation of the RTI model in future. The need

for good communication between partners with a strong focus on evaluation was also

demonstrated by the pilot study, in particular during the classroom observations.

One could state that the outcomes of this study are biased to subjectivity and are not

generalisable at all due to the small sample size. However, by triangulation of data,

extensive peer debriefing and openness to subjectivity, the outcomes became trustworthy

(Altheide & Johnson, 2011; Lincoln & Guba, 1985). Subjectivism was clearly indicated

when interpreting data. Triangulation allowed for reflection upon data from different

perspectives. The methodology used can, therefore, be regarded as a reliable method

to effectuate qualitative multi-method studies (Bazeley, 2013). Due to the fact that only

a small sample of schools participated in the pilot study, generalisability has never been

the main objective. Some remarks can be made, though, that do reflect general aspects.

Although the RTI model was only implemented partially (grade 2), several factors

emerged that can be regarded as general factors. For example, it appeared that teachers’

beliefs played an important role. However, background variables of teachers were not

CHAPTER 3

78

assessed beforehand. A causal relationship between background variables, belief and

behaviour could, thus, not be effectuated, as was also the case in the study of Datnow

and Castello (2000). Since this outcome played a role in all participants from different

schools (n = 8), this still can be regarded as a more general outcome of this study. If these

general factors already occur during the partial implementation of the RTI model, it

is estimated that a school-wide implementation will even show greater problems and

drop out. Moreover, the implications for the measurement of didactic resistance on

a micro level can be expected to be even more problematic. This underlines the need

for an implementation of the RTI model in stages. Maybe the greatest effects of the

RTI model will be effectuated while implemented bottom-up instead of the top-down

implementation as is the case in the United States. If teachers understand the core

principles of the model, and at the same time share the dynamic vision on instruction,

only then the RTI model can be an instrument to measure didactic resistance. And only

then, the RTI model is favourable in identifying learning disabilities.

With respect to the definition of learning disabilities, the concept of didactic resistance

should be explored further. This study underlined the need for a thorough understanding

of the concept and in-depth research into how it should be operationalised. When

interpreted from a Vygotskian perspective, both teacher and student should be taken into

account. As a consequence, the definition of learning disabilities should not only focus

on students’ characteristics, but also on external variables, such as quality of teaching

(or teaching method). After all, learning generally takes place in reciprocal situations.

This study is paving the way for the exploration of the concept and operationalisation

of didactic resistance by making use of the RTI model, and underlines the need for

more in-depth research into this concept.


79

3

This chapter is based on:Tiekstra, M., & Minnaert A. (under review).

At-risk students and the role of implicit theories of intelligence in educational professionals’ actions

Chapter 4At-risk students and the role of implicit theories of intelligence in

educational professionals’ actions

CHAPTER 4

82

Abstract

Implicit theories of intelligence play a role in teacher’s actions. Adaptive instruction

in and out of the classroom is important to optimise learning processes, especially

in the case of at-risk students. This study explored to what extent implicit theories

of intelligence play a role in the actions of educational professionals around at-risk

students. 44 teachers and 57 support professionals participated in this research. Data

were analysed separately for teachers and support staff. 34% of the actions of the

teacher can be explained by implicit theories. However, in denominational schools

this is 61%. SEM showed mediation effects of multiple belief factors in the actions

of support staff. Implicit theory of intelligence predicts the belief in IQ testing, which

precedes the belief in consequential validity of tests (i.e., link to actions according to test

outcomes). These results indicate a strong influence of implicit theories of intelligence

in educational practice.

AT-RISK STUDENTS AND THE ROLE OF IMPLICIT THEORIES OF INTELLIGENCE

83

4

Introduction

The behaviour of teachers in the classroom has an impact on students’ learning

outcomes (Pakarinen et al., 2010), students’ behavioural outcomes (Skinner & Belmont,

1993) and students’ social wellbeing (Jennings & Greenberg, 2009). Moreover, the impact

sustains into later grades (Konstantopoulos & Chung, 2011) and, thus, the ultimate

development of students. Especially in the case of at-risk students, teachers’ behaviour

might have an enormous impact on learning outcomes. Students classified as at-risk in

their learning progress often show learning delays as a consequence of environmental

causes (such as immigrant or low SES families) or caused by learning difficulties (such

as reading or arithmetic problems). Consequently, at-risk students need more coaching

than the rest of their peers, resulting in more one-to-one contacts in which the teacher

plays a crucial role. In practice, this often results in challenging situations where the

teacher needs to guard the whole classroom and to coach at-risk students at the same

time. Furthermore, in order to achieve as best learning results as possible, teachers

should adapt instruction to the level of their students. According to Vygotsky (1978)

the teacher should teach in the Zone of Proximal Development of each student. This

Zone of Proximal Development (ZPD) is the difference between the actual independent

achievement of the student and the performance level of the student when tailored

help is provided (Lidz, 1995). Teaching in the ZPD helps fulfilling the student’s learning

potential, which contributes to the development of the student.

The above-mentioned aspects can be seen as the core principles of teachers’ actions

in the classroom. However, teacher’s actions in the classroom are presumed to be

influenced by implicit theories of intelligence. Dweck and colleagues demonstrated in

their extensive research that implicit theories affect people’s judgments and reactions

(Dweck, Chiu, & Hong, 1995). The influences of teachers’ beliefs has been studied for

longtime (Wiedl, 1980) and has been demonstrated to influence behaviour (Azjen, 2005).

More specifically, Bandura (1993) described the influence of teachers’ beliefs to quality

of teaching and their behaviour in the classroom, which in turn has its impact on the

CHAPTER 4

84

school career of students (Gibbs & Powell, 2012). Beliefs that have been demonstrated

to play a role during teaching processes refer to a broad terminology (Pajares, 1992), for

example teachers’ self-efficacy beliefs (e.g., Tschannen-Moran & Woolfolk Hoy, 2001),

teachers’ perceptions (e.g., Pelletier, Seguin-Levesque, & Legault, 2002), teachers’

attributions (Gibbs & Gardiner, 2008), or teachers’ feeling of responsibility (e.g.,

Lauermann & Karabenick, 2011). Next to these examples, the influence of people’s

implicit theories on their behaviour has been investigated. Implicit theories can be seen

as beliefs that have been constituted subconsciously within an individual, thus, people

are often unaware of the theories that they support.

People’s implicit theories can be general, but are mostly domain-specific. In this article

we focus on the implicit theories on intelligence, referring to the extent to which an

individual considers intelligence as malleable or not (Dweck, et al., 1995). Studies in

the domain of implicit theories of intelligence mainly focused on the impact of an

individual’s implicit theories on the achievement goals of this individual (e.g., Dinger

& Dickhäuser, 2013), as was originally initiated by studies of Dweck (see also Dweck

& Leggett, 1988). In turn, achievement goals determine an individual’s behaviour.

According to Minnaert (2013) achievement goals are considered to contribute to school

success of students substantially, but the way in which achievement goals influence

one’s learning outcomes remains complex. He underlined the need for more research

in this area. According to Leroy, Bressoux, Sarrazin, and Trouilloud (2007) teachers’

implicit theories influence the atmosphere in the classroom. Ames (1992) pointed at

the influence of classroom structure to students’ goals. This gives an indication for the

role that implicit theories play during learning processes of students. However, studies

that explicitly address the link between teachers’ implicit theories and its influences on

students’ learning outcomes are executed to a much lesser extent (Wolters & Daugherty,

2007). A first step in this issue is to investigate the assumed relationship between implicit

theories and behaviour of teachers. Following this assumption, classroom practices of

teachers are supposed to affect students’ learning outcomes.


85

4

People adhere to either an entity view or an incremental view on intelligence (Dweck

et al., 1995). The entity theorists hold the opinion that intelligence is a stable trait and

something that can hardly be changed. Individuals that support the incremental view,

consider intelligence rather as malleable and subjective to change. These different views

imply differences in achievement goals, since achievement situations are experienced

contrastingly (Dweck & Leggett, 1988). An entity theorist assumes an achievement

situation as a situation to demonstrate skills oriented towards evaluation, resulting

in performance goals. Whereas incremental theorists are oriented towards learning

goals, since achievement situations are regarded as learning opportunities (Dinger

& Dickhäuser, 2013). Consequently, these views have their influence on classroom

practices. For instance, teachers with an entity view of intelligence are more likely to be

focused on performance and direct instruction practices. Whereas incremental theorists

envision mastery-oriented learning and focus on the learning process instead of the

learning results solely.

Teaching in the ZPD is, thus, influenced by these implicit theories of intelligence. This,

in turn, has its impact on educational opportunities of students. For instance, in the case

of low performing students, an entity teacher would only adapt instruction in such way

that these students get supplementary practice or extra assignments, or opt in favor of

exclusion, since low scores on achievement tests indicate a low ability to learn. Entity

teachers, do not expect these students to learn, resulting in more stigmatisation and

self-fulfilling prophecies. On the contrary, an incremental teacher would provide instruction

that is adapted to the level of the student, since low scores on achievement tests indicate

that these students needs more (or other) opportunities to learn. Exploring a student’s

ZPD is, likewise, more common among incremental theorists.

Due to deviating results from the rest of their classmates, at-risk students are frequently

eligible to assessment procedures. These assessment procedures often include IQ tests.

Accordingly, outcomes of intelligence testing are interpreted differently by entity or

incremental theorists. Moreover, entity theorists will attach great importance to the

CHAPTER 4

86

outcomes of an IQ test, whereas incremental theorists will approach these outcomes as

a starting point for adaptive support.

The influences of teacher’s implicit theories on particular groups of at-risk students have

been investigated previously. Literature can be found about the group of gifted students

(Baudson & Preckel, 2013; García-Cepero & McCoach, 2009). However, information

about SEN students seems to be scarce. Jordan and Stanovich (2001) underlined in their

research that teachers’ beliefs affect the instructional processes, and had even greater

influences in the group of at-risk students. Teachers who had an incremental view, and

therefore felt responsible for their students’ learning processes, were more frequently

involved with at-risk students than teachers with an entity view. The need for more in-

depth examination of the influence of teachers’ beliefs on their educational actions is

underlined by several researchers (e.g., García-Cepero & McCoach, 2009).

Aim of the present studySince the content and manifestation of instruction is extremely important for the

group of at-risk students, a closer look is needed in the process among educational

professionals involved with teaching these children. The current study aims at the

exploration of a relationship between implicit theories of intelligence and actions of

educational staff with regard to the care process of at-risk students. At-risk students

often undergo assessment procedures eligible for individualised education plans (IEP),

resulting in interventions and adapted instruction. Therefore, next to teachers, support

professionals were taken into account. These professionals administer the assessments

and assist teachers in their instructional processes. But in what way play implicit theories

part in the educational professionals’ actions in practice? As mentioned above, entity

and incremental theorists approach at-risk children differently. However, the extent to

which these implicit theories play their role in daily practice remains unclear. Therefore,

this study addresses this issue from a practical point of view. The role of implicit theories

will be investigated with respect to actions in practice.


87

4

There is plenty of evidence that background variables such as years of teaching

experience, and level of grade play a role in teachers’ beliefs (Bol, Stephenson, O’connell,

& Nunnery, 1998; Leroy et al., 2007; Wolters & Daugherty, 2007). Therefore, these

variables have been taken into account. If actions can indeed be explained by implicit

theories, then, do amount of experience, and the grade in which the educational

professionals work, interact in this relationship? Moreover, the influence of school ethos

on teachers’ beliefs has been highlighted by Gibbs and Powell (2012). They underlined

the importance of collective beliefs: Teachers are often influenced by collective beliefs

that are prominent in a school. As illustrated by Opdenakker and Van Damme (2006),

the type of school and denomination contribute to collective beliefs inside schools,

which affects students’ outcomes. Therefore, it was also investigated whether the type

of school (defined as denominational or public school1) influenced the relationship

between implicit theory and behaviour.

Method

SampleThe study was carried out in the western part of The Netherlands. This specific area

of the Netherlands is confronted with a large population of ethnic minorities. In

comparison to other parts of The Netherlands, teachers in this area are confronted

with more at-risk students in their classrooms having more serious problematic home

situations. The issue of adapting instruction to the level of students is very common

in schools in this area. Due to the high incidence of immigrant children, schools often

receive financial aids from the government, which can be used for extra assistance inside

or outside the classroom.

1 In this, denominational schools refer to schools that hold a Christian point of view in their education, whereas public schools are not explicitly grounded in an ideology. Students in public schools are not restricted to a certain religion. The division does not account for differences in financial aid.

CHAPTER 4

88

In The Netherlands elementary school begins with Kindergarten (at age 4) and continues

until grade 6 (at age 12), after which children attend secondary education. The ‘care

process’ at regular education is organised in different levels where school psychologists,

special services coordinators, remedial teachers, and teachers are involved (Imants, Van

der Aalsvoort, De Brabander, & Ruijssenaars, 2001). Generally, when teachers remark

problems with a student, the school’s special services coordinator assists the teacher in

these problems. In the case of persistency of problems, the special services coordinator

asks help from a school psychologist working at an educational advisory agency.

For this study, school psychologists working for an educational advisory agency were

asked to participate. Each school psychologist would then distribute the questionnaire

to one special services coordinator with whom they worked and three teachers with

whom the special services coordinator worked. One teacher represented the first two

years of elementary school (Kindergarten), another teacher grade 1 to 3, and the third

teacher represented grade 4 to 6 of elementary school to provide a representative

sample of teachers. It was aimed for to obtain a response of 50 school psychologist, 50

special services coordinators, and 150 teachers. A total of 44 teachers and 57 support

professionals (21 special services coordinators and 36 school psychologists) participated

in this research, which reflects a response rate of 40,4% in total.

Measures Implicit theories of intelligence (ITI) were assessed by three items that indicated the view

on intelligence, either as an entity or incremental theory. The original items (Dweck et

al., 1995) were converted to teacher items and translated to Dutch. Next to these items,

the behaviour of teachers and support staff in case of an at-risk student was assessed

by a content-driven questionnaire. Since the study was explorative by nature, and the

items needed to address behaviour that was adapted to the level of each professional,

a different version for teachers and support staff was used. Depending on the version,

the questionnaire consisted of 17 or 18 statements that had to be answered on a 1 to 5

Likert scale.


89

4

Teacher questionnaire. The ITI- items proved to be reliable in our sample, as

indicated by a Cronbach’s alpha of .77 in this version. A principal component analyses

(PCA), graphical representations and other ad-hoc descriptive data analyses revealed

that in the teachers’ questionnaire one factor emerged (α.73 ; 9 items) explaining 33%

of the total variance. Examples of items are I observe problems in learning progress of a student

when the student obtains low scores on achievement tests that I administered, and I evaluate instruction

I provide to the at-risk student with the special services coordinator. This factor was called “actions

in case of an at-risk student”. A higher score on this factor reflects more consistency in

actions undertaken by the teacher in case of an at-risk student.

Support professional questionnaire. The ITI- items had an Cronbach’s alpha

of .75 for support staff. The PCA’s in the support professionals’ questionnaire showed

two factors that could be distinguished. The first factor was composed of three items (α.

58) and reflected the belief that IQ testing is a correct estimator of learning capacities,

called “belief in IQ testing”. One of the items was e.g., The administration of an IQ test is

enough for the estimation of a child’s learning capacity. The higher the factor score, the more

faith to IQ testing (i.e., the score reflects learning capacity of students perfectly).

The second factor was related to three items (α .55) describing the link from test score to

actions in practice, shortly the “belief in consequential validity of tests”. For example,

Test outcomes have a binding effect upon the advice I give to teachers. A high score was indicative of

more value attributed to results of testing, implying a more performance oriented view

on capacities. The authors note that several items were not taken into account, since

these did not load on any of the factors, though the other items had very high loadings

on one of the two factors.

Both factors could, thus, clearly be separated according to the data. Moreover, both

factors explained 55% of the total variance.

CHAPTER 4

90

Analyses Hierarchical regression analyses were carried out to investigate the possible influences

of background variables on the relationship between the ITI- items and the other factor

for teachers’ questionnaire. Careful interpretation of the two factors that emerged from

the PCA’s in the support professionals’ questionnaire showed that one factor reflected

upon belief in IQ testing and the other upon belief in consequential validity of tests.

Therefore, a strong grounding for testing mediational effects appeared. By making use of

the Lisrel program, Structural Equation Modeling (SEM) was used to test the mediation

effects between different components of the support professionals’ questionnaire.

Results

TeachersThe correlation between the implicit theory of intelligence factor and the other factor

was .58 (p ≤ .01; N = 38). Ergo, 34% of the actions of a specific teacher in this sample

can be explained by the implicit theory of intelligence this teacher has. Hierarchical

regression analyses investigated the moderating effects of background variables in the

relationship between implicit theories of intelligence on the one side and “acting in case

of at-risk student” on the other side. Experience of teacher, and the group in which

the teacher works, did not significantly moderate this relationship. More interestingly,

it appeared that the variable “type of school (public vs. denominational)” did play a

moderating role in this relationship. As can be seen in Table 1, the analysis resulted

in a significant additive 9% in the variance explained by adding the interaction effect

between type of school and implicit theory in the relationship between implicit theory

and actions of the teacher (∆R2 = 0.92, ∆F = 5.45, p = .03).


91

4

Table 1 | Hierarchical regression analyses predicting ‘acting of teacher’ from implicit theories of intelligence, and type of school, and interaction between both predictors

Model R ∆R2 ∆F Sign. ∆F

Implicit theory .58 .34 18.13 .00

Implicit theory; Type of school .58 .00 .02 .88

Implicit theory; Type of school; Implicit theory * type of school

.65 .09 5.45 .03

Note. N = 38, distributed over 10 public and 10 denominational schools

Actually, as showed in Figure 1, this interaction effect between ITI and actions in case

of an at-risk student appears to be much stronger in denominational schools than in

public schools. In denominational school implicit theories of intelligence explain 61%

(∆R2 = 0.61, ∆F = 20,6, p = .001, N =15) of actions of the teacher, compared to only

20% (∆R2 = 0.20, ∆F = 5.2, p = .03, N = 23) of actions in public schools. This effect was

not subjective to outliers, as was tested by the Leverage and Cook criteria.

CHAPTER 4

92

Teac

hter

act

ions

5,0 7,5 10,0

Implicit theory of intelligence

12,5 15,015,0

20,0

25,0

30,0

35,0

40,0

publicdenominationalpublicdenominational

Figure 1 | The difference in influence of implicit theory of intelligence on actions of the teacher between teachers at public schools and denominational schools.

Support professionalsHierarchical regression analysis revealed no moderating effects of background variables

in this group on the relationship between implicit theories and both factors. Structural

equation modeling showed that there was a full mediational model between these

variables. The model tested the effect of implicit theories of intelligence, via belief in IQ

tests, on the belief in consequential validity of tests, as dependent variable. This model

had a very good fit, since the margin of error was small (90% CI RMSEA [0.00, 0.00],

p = .998) and the sources for unexplained variance in the model were trivial (SRMR=

.004). There were no direct effects of implicit theories on the belief in consequential

validity. As can be seen in Figure 2, implicit theory of intelligence predicts the belief in

IQ testing, which, in turn predicts the consequential validity of tests.


93

4

Figure 2 | Mediational model of belief factors in support staff; * significance at a 5% level. N = 56.

Discussion

Since the influence of implicit theories of intelligence (either entity or incremental)

in practice needed to be investigated, this study aimed at an in-depth exploration of

actions of educational professionals in the case of at-risk students. These students are

often confronted with the consequences of stigmatisation and self-fulfilling prophecy,

need more coaching from teachers and support professionals, and are often eligible

for assessment procedures. This study explored the underlying principles for the

educational professionals’ actions to give an insight in quality of care processes around

at-risk students. Support staff (i.e., school psychologists and remedial teachers) and

teachers were asked to fill in a questionnaire. The ad hoc constructed questionnaire was

composed of items that reflected upon practice and items that reflected one’s implicit

theory of intelligence (from Dweck et al., 1995). Data analysis showed clear distinctions

between the level of the teacher and the level of support staff. Therefore, results will be

listed separately for both groups of participants.

In respect of teachers, results show that implicit theories play a prominent role in their

actions in the classroom in the case of an at-risk student. In this sample, 34% of the

variance on the factor “actions of a teacher” can be explained by either an entity view or

Implicit theory ofintelligence Belief in IQ testing Consequential

validity of testing

.25* .33*

.08

CHAPTER 4

94

an incremental view on intelligence. Moreover, this study showed that this relationship

is moderated by type of school. Analyses showed that the prediction of teachers’ actions

by their implicit theories is more robust at denominational (Christian) schools than at

public schools. Differences between both types of schools have been found previously in

educational studies. However, as Opdenakker and Van Damme (2006) underline, these

differences are often related to other variables.

The positive correlation coefficients for both type of schools indicate that the more

a teacher supports the entity view of intelligence, the more consistency is showed in

acting in the case of an at-risk student. Thus, teachers know better what they need to

do in case of an at-risk student if they support the entity view. However, their actions

are in line with the entity view, reflecting a more performance oriented approach as

explained in the introduction. Consequently, stigmatisation and self-fulfilling prophecies

could occur in these situations. On the other hand, as was shown in Figure 1, there

is a lack of teachers that do support the incremental view on intelligence showing a

high consistency in their actions. The more incremental view on intelligence, the less

consistency in actions of the teacher. Hence, some teachers do support a more dynamic

(i.e., incremental) view on intelligence, but they do not know how to act accordingly.

This indicates a lack of declarative knowledge or a lack of procedural knowledge in the

actions of teachers. These effects are more robust at denominational schools.

With regard to support professionals, i.e. special services coordinators and school

psychologists, implicit theories do also play a role, though differently. Next to the

factor describing implicit theory of intelligence, two other factors emerged from the

questionnaires. One factor reflecting the belief in IQ testing and another factor reflecting

the belief in consequential validity of testing. Background variables appeared to be of

less importance in this group, since these did not moderate on relationships between the

factors. The SEM modeling showed that there was a full mediational model between

the factors, implying that implicit theories predict the belief one has about IQ tests,

and, in turn, explains the way people act according to these tests. The consequential


95

4

validity of tests is, thus, caused by implicit theories people have. This means that the

way people think about intelligence affects their opinion about the quality of these tests,

and thus, their interpretations. If one supports the more static view on intelligence (an

entity theorist), this individual will hold more account to IQ testing results, which will

result in more static decision making processes. One might regard this model rather as

logical, but when applied to the educational practice, it is rather disastrous. Due to a

stronger focus on IQ testing results, learning problems will be regarded as a problem of

the students, instead of a reciprocal effect between teacher and students. Less attention

will be drawn upon the role of the teacher in adapting to the level of the student.

Consequently, the teacher will not teach in one Zone of Proximal Development.

As demonstrated by this study, implicit theories of intelligence play a substantive

role in the educational care process around at-risk students. In this group of students

the identification of learning disabilities, or the admission to remedial activities, is a

frequently occurring issue. In each of these decision making events, implicit theories play

a role. In turn, the impact on educational opportunities of at-risk students is immense.

These findings suggest that educational professionals should develop awareness of

the influence of these implicit theories on all their actions. Educational professionals

should strive towards a model of constant reflecting upon one’s own actions and of

those professionals around them. Teachers should be supported in their instructional

practices. School directors should create opportunities for their teams in order to consult

each other’s work. An incremental view on intelligence should be supported in order to

create learning opportunities and exploration of a students’ ZPD rather than a focus on

performances (as is often the case with entity theorists).

There are, however, some limitations to the results of this study. The relatively low

response rate (40,4%) may have caused biased results. Moreover, the sample size

was rather small (44 teachers and 57 support professionals). Although the statistical

grounding of some results might be regarded as thin, the authors would like to emphasise

this small sample size which probably caused a problem of power. The alpha’s of the

CHAPTER 4

96

factors would have been higher if the test would have been lengthened. According to

Carmines & Zeller (1979) and Cortina (1993) interpretation of alphas should be done

with caution, as it is subjective to the number of items used. The reliability of the

scores on the separate factors is, therefore, above threshold, even in the case of a low

average inter-item correlation, since only three items have been used in some factors.

This implies that evidence might have been even stronger when the study would have

been carried out with more items, and in a greater sample distributed nationally instead

of a specific area in the western part of the Netherlands. Moreover, due to the high

incidence of at-risk students in this specific area, it is estimated that people hold a more

flexible approach to these students than in the rest of The Netherlands. A nation-wide

sample would probably have proved an even more persisting role of implicit theories

of intelligence in educational practice. Furthermore, results are based on self-reported

questionnaires which are always sensitive to socially acceptable answers.

To conclude, these results show that we should be very attentive to implicit theories on

intelligence of educational professionals. They play a prominent role in processes in

the classroom, and outside the classroom, in case of at-risk students. These results have

clear implications for the implementation of educational interventions. If the aim of

a specific educational intervention is to change behaviour of educational professionals,

this study indicated that more processes come into play when interpreting the effects

of the intervention. Behaviour might not be changed due to persistent implicit theories

people have. Accordingly, this study paved the way for interventions that intervene

on beliefs and behaviour at the same time. Since both aspects cannot be seen apart

from each other, one should strive toward a change in both aspects in order to achieve

sustainable effects in behaviour.

With regard to the consequential validity of testing procedures, the study also

had implications. The study demonstrated that implicit theories play a role in the

interpretation of, and actions according to test outcomes. This means that not so

much the outcome itself, but rather the interpretation play a prominent role in psycho-


97

4

educational practice. More focus upon the interpretation of test outcomes and the

translation to practical guidelines, in other words the consequential validity of tests, is

requested for. After all, the future of individual at-risk students often depends on few

assessment outcomes.

“The unexamined life is not worth living.”(Socrates)

Chapter 5Voices from practice:

when is the gap between diagnosis and intervention apparent?

Acknowledgement:I thank Lotte Bergwerff for her contribution to the studies in this chapter.

CHAPTER 5

100

Abstract

In this chapter an overview is provided of the current state of psycho-educational

practice in the Netherlands, in particular the role of test outcomes considered. In order

to detect clues for bridging the gap between diagnosis and intervention, one should

investigate the ecology in which this gap is apparent. Two in vivo studies have been

carried out. In a first study, a questionnaire has been administered to a total of 36 school

psychologists, 21 special care coordinators, and 44 teachers. A qualitative analysis of the

answers, by means of a classification scheme (κ .82), revealed questions about the specific

role of intelligence tests and its consequences to educational practices. Therefore, a

case study has been carried out in a second study. The case study provided in-depth

information about the targeted care process around a grade 1 student. Results indicated

a gap between diagnosis and intervention that followed after the administration of

the intelligence test. Suggestions are proposed for improvement, and the need for

interventions at the level of educational professionals is highlighted.

VOICES FROM PRACTICE: WHEN IS THE GAP BETWEEN DIAGNOSIS AND INTERVENTION APPARENT?

101

5

Introduction

The results of a recent literature review (see chapter 2 of this dissertation) showed

that more attention should be drawn to consequences of test administrations and test

results in the educational field. In other words, the consequential validity (Messick,

1995) of assessment procedures is insufficiently warranted at the moment (Tiekstra,

Minnaert, & Hessels, 2016). As a consequence, the well-known gap between diagnosis

and intervention remains present (Haywood, 2012; Shapiro & Kratochwill, 2000). If

we want the attempts in bridging this gap to be successful, one should not only focus on

the outcomes of the assessment procedures, but also focus upon educational practice.

As demonstrated in chapter four of this dissertation it appears that the way people

think about intelligence influences the way people act in practice. Implicit theories of

intelligence, thus, play a prominent role in educational practice (Dweck, Chiu, & Hong,

1995; Tiekstra, & Minnaert, submitted). Accordingly, test scores influence teachers’

ideas and beliefs about students, and are often input for decisions about a student’s

academic career (i.e., special education eligibility, targeted care). Therefore, the need

for valid outcomes of (intelligence) tests is urgent.

One of the core elements in this dissertation is the exploration of ways to enlarge

consequential validity of test procedures. A next step in this exploration is to investigate

the ecology in which the studies in this dissertation are situated. This chapter reports on

the results of an in vivo study where the educational field in case of at-risk students was

subject to a profound examination. Only in this way ecologically valid conclusions can

be drawn about the state of consequential valid test outcomes that might bridge the gap

between diagnosis and intervention.

Role of test scoresWhen diagnosing children with severe learning disabilities, test scores play a prominent

role. Generally, school achievement tests are used to detect learning delays, and

standardised intelligence tests often come into play when students show persisting

CHAPTER 5

102

learning delays. As highlighted in chapter three, in the newest definition proposed

by the DSM-5, an IQ score is still regarded as an important parameter to diagnose

learning disability. Therefore, an IQ measure is a prerequisite to get subsidisation from

the government. As a consequence, if a child displays difficulties in learning reference is

often made to an IQ test. To summarise, IQ tests play a prominent role in daily psycho-

educational practice.

The use of IQ tests in individual assessment procedures is, however, questionable.

Growing evidence shows that IQ tests do not provide reliable and valid measures in at-

risk children (for more information see e.g. Tiekstra et al., 2014). Besides, intelligence tests

have been criticised for their lack of information for practice (Gresham & Witt, 1997;

Lebeer et al., 2011), often referred to as the gap between diagnosis and intervention

(Shapiro & Kratochwill, 2000). It is worth mentioning though, that, originally,

standardised tests had been developed for classification issues instead of educational

intervention planning. Theoretically, this means that IQ tests are not purposefully

developed to provide information for educational practice. Subsequently, the question

arises whether and why we should administer IQ tests as frequently as is done in current

educational practice? Lebeer and colleagues (2011) demonstrated in their large-scale

study among European countries that there is an urgent need for practical information

according to test outcomes. Moreover, if we want to prevent teachers from planning

their instruction intuitively, a focus upon accurate translation of test outcomes to daily

practice is required.

Alternative formats of assessment should be used to bridge the gap between diagnostic

assessment and classroom practice. Bosma and Resing (2008) proposed to use dynamic

assessment. They investigated the opinions of teachers about diagnostic reports in

which a measure based on either a traditional or a dynamic test was indicated. Teachers

appreciated the dynamic reports in which clues for classroom practice were provided.

Moreover, classroom observations revealed that some of the teachers even adapted their

instruction. Research shows, however, that dynamic assessment is not yet implemented


103

5

in educational practice (e.g., Elliott, 2003). Currently, the gap remains present in Dutch

educational practice in two ways: the used assessment does not provide guidelines for

the process after the assessment, furthermore dynamic assessment (which provides such

guidelines) is not yet implemented at schools.

Professionals involvedIn terms of reference, generally, individual test administrations belong to school

psychologists. Thurlow and Ysseldyke (1982) underlined the difference between school

psychologists and teachers with regard to their opinions about information that is

needed for educational intervention planning. They indicated that school psychologists

rely on the use of standardised test outcomes solely, while teachers also supported the

use of behavioural observations in instructional planning. Rouse and Agbenu (1998)

underlined the problems teachers have with all kinds of assessment and suggest that

“many of the problems results from confusion about the nature and purpose of

assessment, the teachers’ role in this process and how to assess pupils with learning

difficulties.” In short, more insight is needed into the educational practice with regard

to processes around students showing problems in learning and information processing.

In the Netherlands another professional comes into the picture when a child has

learning difficulties: next to the remedial teacher, a special care coordinator operates

within the care system of the school. In Dutch regular education, the teacher generally

detects learning problems of students (as caused by learning delay, learning disability or

behavioural problem) in the classroom. The special care coordinator’s task is to support

the teacher in resolving these learning delays. If the learning problems exist, the school

psychologist will provide specialised assistances. In effect, usually, the care process is

organised by different professionals involved. Theoretically, this implies clear roles, but

in the educational practice these roles are less clear. As demonstrated by Imants, Van

der Aalsvoort, De Brabander, and Ruijssemaars (2001) special care coordinators have

different functions in different schools, and rarely provide coaching to teachers.

CHAPTER 5

104

Aim of the current research If the consequential validity of assessment procedures needs to be improved, then

we should know how the care processes in schools are effectuated and what role test

outcomes play in these care processes. The aim of this research is to thoroughly explore

the processes in regular education in case of an at-risk student. What role do the

professionals involved play, and what type and kind of information are decisions based

upon? Therefore, the goal of the study was to provide situational information about this

issue by interpreting information from multiple voices from practice. This demanded a two-

step approach; a questionnaire was distributed first. Second, a case-study was needed to

provide answers to questions resulting these questionnaires.

General Method

The in vivo study aimed to describe experiences and needs of educational professionals

in the Netherlands. Several studies were carried out. Both quantitative and qualitative

data was gathered. For the purpose of clarity, this chapter focuses on the results of

the qualitative part solely. The results of the quantitative part are included within the

previous chapter (chapter 4) of this dissertation, only some descriptive quantitative

(frequency) analysis will be displayed in this chapter when needed to illustrate the results

of this study.

A multiple qualitative approach was used in order to compare and explore the needs

and experiences in greater depth. A questionnaire comprising statements, vignettes,

and open-ended questions was administered to explore the current state in providing

care to at-risk students. In addition, a case study was carried out which allowed for in-

depth information. A case study describes a case within a context, resulting in detailed

information that is closely related to practice (Flyvbjerg, 2006; Swanborn, 2010).


105

5

ParticipantsThe study was set up in collaboration with an educational advisory agency in the

Western part of the Netherlands. In comparison to other parts of The Netherlands,

teachers in this area are confronted with more at-risk students in their classrooms

having more serious problematic home situations. Due to the high incidence of

immigrant children, schools often receive financial aid from the government, which

can be used for extra student oriented assistance inside or outside the classroom.

ProcedureTwo separate studies have been carried out. A questionnaire has been distributed in

the second semester of the 2012 school year. Next, a case study has been conducted

in the same semester during a 4 month period. The school selected for the case

study did not participate in the questionnaire study, nor had any knowledge of it.

Study 1: In Vivo Questionnaire

MethodA questionnaire was developed which addressed several aspects of the care process

around children showing any kind of learning problems in regular education. These

aspects were the identification of problems (related to intelligence and intelligence

tests), educational plans, communication, and the effectuation of care processes.

Examples of items can be found in the appendix.

Sample. It was aimed for to obtain a response of 50 school psychologists, 50

special care coordinators and 150 teachers. First, school psychologists attached

to the educational advisory agency completed the questionnaire. Second, school

psychologists distributed the questionnaire to one special care coordinator with

whom they worked regularly and three teachers with whom the special care

coordinator worked. A total of 44 teachers, 21 special care coordinators, and 36

CHAPTER 5

106

school psychologists participated in this research, which reflects a response rate of

40,4% in total.

Instruments. The questionnaire comprised of three divisions: statements, a vignette, and

open-ended questions. Logically, these questions needed to be closely related to the daily

practice of participants, so three different versions were developed.

The statements had to be answered on a Likert-scale, varying from 1 (completely disagree)

to 5 (completely agree). Participants needed to respond to 17 or 18 statements, depending

on the version. An example of an item in the teachers’ version is The information I get from the

special care coordinator is sufficient to adapt my instruction to the specific needs of student. And an example

which was used in all versions is Administering an IQ test is essential for setting up educational plans.

The vignette was composed of a situational description in which a student with severe

learning problems needed to be identified. The participants were asked about their role

in the care process in two multiple choice questions (special care coordinator, and school

psychologist) or one multiple choice and one open question (teacher), formulated as What

would you do if you would observe problems of student X, as in the given case?

The section with open-ended questions also focused on the care process of students showing

learning problems. In the version of school psychologist 3 questions were asked. Special

care coordinators had to answer 5 questions. The teachers’ version comprised another

multiple choice question (where the last option was: something else, namely…) and 4 open-

ended questions.

Procedure. The questionnaire was distributed to school psychologists in the second

semester of the 2012 school year. Then, school psychologists were asked to distribute the

questionnaire to special care coordinators and teachers with whom they worked. These

completed questionnaires were sent to the researcher.

Analysis. A descriptive analysis took place of the statements (frequency analysis)


107

5

and the multiple choice questions. Answers to the open-ended questions have been

analysed by means of a classification scheme. This classification scheme was developed

in two phases. First, 10% of the questionnaires was randomly selected to formulate

classifications. The selected questionnaires did not provide sufficient variation in

answers, therefore classifications based on researchers’ experiences were added to the

scheme. Several examples of items and the categories in the classification scheme can

be found in the appendix. It must be mentioned that multiple classifications could be

scored per item. The classification scheme was tested for its reliability, resulting in an

overall interrater agreement of κ.82. The Cohen’s kappa for the version of teachers was

κ.74, for special care coordinators κ.90, and κ.87 for school psychologists. Because of

these high Kappa values, the classification scheme can be considered reliable, and not

subject to chance.

Results and interpretationIdentification of problems. The vignette study showed that teachers generally

observe learning problems based on the results of the previous school year (45%), or

based on observations in the classroom (46%).

The majority of teachers thinks that administering IQ tests is essential for advice with

regard to the educational practice, whereas a minority of school psychologists and special

care coordinators supports this statement. In any case, teachers are informed about the

results of the individual test administration by the school psychologist. Less experienced

special care coordinators (0-3 years of experience) indicate that their advice to teachers

is often to very often based on these test results, whereas more experienced special care

coordinators ( ≥ 3 years) indicate sometimes this is the case. Special care coordinators do

not administer IQ tests themselves. 44% of school psychologists think that the current

intelligence tests are appropriate enough to base their advice on. Whether this advice

is rather needs-based or classification focused remains unclear. Moreover, special care

coordinators and school psychologists value the outcomes of intelligence tests to a wide

range (min. 1 max. 4), implying large individual differences.

CHAPTER 5

108

The value of information that intelligence tests provide for educational practice is

ambiguously evaluated by school psychologists. On the one side, school psychologists

often provide advice for the classroom practice based on test outcomes. On the other

side, they indicate that test outcomes do not provide such information exactly to the

specific context of the assessed student. Here, too, the type of advice remains unclear.

Half of the teachers indicated to detect learning problems when a student shows low

scores on curriculum related achievement tests. There is no consensus in the answers

about the use of these tests in identification of learning problems. 40% of teachers

consult their more experienced colleagues in case of at-risk students.

And finally it should be noted that clear differences were observed with regard to the

knowledge of dynamic assessment: only 10% of teachers knew what dynamic assessment

is, compared to 30% of special care coordinators, and 80% school psychologists. It

must be mentioned, though, that these percentages are elevated compared to average

Dutch schools. The questionnaires were administered to a group of school psychologists

who worked at the same educational advisory agency. This agency provided their staff

with training in dynamic assessment which is rather exceptional in the Netherlands.

Therefore, these percentages are not representative to the Dutch population. However,

the amount of teachers and special care coordinators is, still, very small. Moreover, only

80% of school psychologists indicated to have knowledge of dynamic assessment.

Individual Educational Plan (IEP). 70% of the special care coordinators thinks that

the psychological reports provide enough information for the instruction in the classroom.

It appeared that teachers set up IEP, sometimes in consultation with the special care

coordinator. 60% of special care coordinators and 86% of school psychologists indicate

setting up IEP for teachers only rarely. The content of the IEP is often discussed between

special care coordinators and teachers (80%). Strikingly, no special care coordinator reported

that the educational plans are discussed with school psychologist.


109

5

With regard to the psychological reports, 33 of 36 school psychologists need test results to

write these reports. Besides, 25 school psychologists would like to receive information from

the teacher, and 19 indicate information about home situation of the student.

Communication. In most cases, teachers report immediately to special care coordinators

when observing learning problems (41%). However, teachers did not answer clearly to the

question “What happens after you consulted the special care coordinator?” The majority of teachers

indicates that the information they get from special care coordinators is sufficient for adapting

their instruction. Special care coordinators and teachers evaluate the adapted instruction

with regard to at-risk student regularly. However, school psychologists are not informed of

the adaptations in the teaching process.

According to the answer to the open-ended question for teacher “What is the school psychologist’s

role?” the role of school psychologist is not clear to teachers. According to some school

psychologists, on the other hand, information they provide about the students’ results and

advice is not acted upon by the teacher. Evaluation of the care process only takes place

between teacher and special care coordinator. School psychologists are rarely involved with

evaluation, as one of them remarked “I suspect that my advice is put into a drawer and never looked at

again”. This demonstrates that there might be discontinuity in targeted care.

Effectuation of care processes. As mentioned before, teachers report to special

care coordinator in case of an at-risk student. The steps undertaken by special care

coordinators differ very much. Answers like I administer several tests (24%), or consultation

to other experts (19%) were mentioned. They seldom assist in adapting instruction: half

of them indicate never inventing exercises for the classroom instruction, the other half

rarely.

According to teachers, the school psychologist comes into play when the school is unable

to provide appropriate intervention. 17% of the school psychologists immediately

administer an IQ test when a child is referred to them, 83% works differently. Several

CHAPTER 5

110

aspects, like classroom observation, parental or teacher consultation, are mentioned.

School psychologist, thus, work very differently with respect to at-risk students.

Future needs. Participants were asked about information they would need, and

currently did not receive. Teachers strongly suggest that school psychologists provide

information about care or interventions. Special care coordinators indicate the need for

a specialist within the school. School psychologists would like to have opportunities to

evaluate and take care of continuity of the care process. Currently, they cannot perform

a follow-up of the students after their reports.

ConclusionDue to the rather ambiguous answers to questions around the value of intelligence

tests for advice in educational practice, questions remained unanswered. Although

participants were positive overall about care processes, discontinuity in the care

processes was shown. Roles are unclear, and evaluation hardly takes place. Moreover,

questions arose about the type and format of advice (remedial) teachers get from school

psychologists who interpret the outcomes of an intelligence test. Why is an individualised

assessment of intelligence administered? What happens afterwards? These questions

needed to be answered in a second study. Therefore, a case study was carried out to

unfold the process between assessment and intervention.


111

5

Study 2: Case Study

MethodThe aim of the case study was to provide answers to the questions produced by the first

study, as mentioned above. The answers to these questions will provide an insight in the

gap between diagnosis and intervention.

Sample. Subject to the case study was the care process of a grade 1 student showing

persistent learning problems. The school in which the case study was carried out

received financial aid from the government, due to the diverse multicultural population

of students. This resulted in an additional grade 1 teacher for several days a week,

which allowed for teaching in smaller groups and individual assistances for students that

needed it the most.

Instruments. Observations during all key elements in the care process (such as

meetings, consultations, and test administrations), and observations in and outside

the classroom were undertaken. Moreover, all professionals involved, i.e. teacher and

additional teacher, school psychologist and special care coordinator, were interviewed

regularly and at specific moments (such as parental consultation) during the process

in order to gain insight in their beliefs and reasons for acting in the classroom. Finally,

student’s files and school’s policy documents were studied in depth.

Procedure. During a 4 month period the care process of the student has been followed

carefully. The study started in the second semester of grade 1 at the moment that the

school psychologist was consulted for screening the student (begin March 2012). The

study ended at the end of grade 1 (end of June 2012).

Analysis. For the trustworthiness of data (Lincoln & Guba, 1985) peer debriefing and

member checking techniques were carried out. During each step of the data collection

one or two other researchers critically reviewed subjective statements of the initial

CHAPTER 5

112

researcher. Information was checked with all participants involved. Moreover, data

triangulation from the multiple data sources mentioned above contributed to reliability

of results.

Results and interpretationWhy is an individualised assessment of intelligence administered in this case? The teacher, assistant

teacher, special care coordinator and the school psychologist decided together to start

an “individual assessment of intelligence” (IAI) to get more information about the

student’s problems. In the present case, the IAI comprised not only the administration

of a static intelligence test, but also an observation in the general classroom by the school

psychologist, and an analysis of the information provided by the teacher and special care

coordinator. However, in the school (i.e. teacher, special care coordinator, and assistant

teacher) the IAI is seen as “an estimation of intelligence of the student, and what to

expect from her” and is seen as a very valuable instrument. The IAI appeared to be a

tool to gain insight in the possible causes for learning delays of this student, the school

had high expectations for the outcomes of the IAI. The special care coordinator and

teacher expect advice from the school psychologist “in order to eliminate the student’s

learning delays, and how they should act with this student”.

Later on in the process, the IAI appeared to play a less important role in problem

identification. Student’s results to other instruments, such as curriculum related tests

and scores on nationwide screening instruments, and teacher’s observations appeared

to be more indicative for the ‘care’ process. The outcomes of the IAI were only additive

to these results. In this case, results of the IAI were surprising to the school, since

they “could expect more from the student than she shows at the moment”. At this

moment, the participants seem to have different ideas about the usefulness and goal

of the intelligence test administration. The assistant teacher indicates the influence of

contextual effects during the test administration by the school psychologist: “it is different

from the classroom situation”. Participants seem to pay less attention to the outcomes

of the test, since it is not in congruence with what they experienced in the classroom. In


113

5

sum, the diagnosis of the problems of this student seems not to be executed in a cyclic

process. Systematic evaluation of the outcomes to the care process of the child is, apart

from the meeting with the school psychologist, not evaluated between participants.

The reasons for the IAI are multiple: what can we expect from the student? What

are the causes for the problems? But specific answers to these questions do not seem

to be provided by the IAI. Other information sources (such as curriculum related

test, observations) seem to provide information that is needed to give answers to these

questions.

What happens afterwards? Already ahead of the results of the IAI it was clear that

the student needed more care in future: intensive individual remedial teaching was

planned. Currently, the student already experienced intensive support with regard to

Dutch (school language), and this support should be continued. In the care process,

however, no structure or vision was observed, nor did the care process change after the

administration of the IAI during the period of this study. Although the IAI showed that

they could expect better results from the student, the school did not change their ‘care’

after the IAI outcomes. Thus, a gap between diagnosis and intervention was observed.

Summary. Before the administration of the IAI, high expectations concerning the

outcomes of the IAI were set by the educational staff. At the end of grade 1, however,

the results of the IAI did not play a significant role anymore. The researcher observed

an overrated value of the IAI before the administration, and the gap between diagnosis

and intervention was not bridged by the IAI. On the one hand, the IAI seemed to

deliver important information for the school with regard to their expectations of the

student. On the other hand, however, the school could not deduce practical guidelines

from the IAI to develop a tailored program adapted to this particular student. While a

lot was expected from the results of the IAI, the effects of the test administration did

not translate to extra care and time for this student in practice, as could be observed

during our study.

CHAPTER 5

114

ConclusionTraditionally, an IQ test is regarded as the best instrument to find causes of learning

problems. One could state that a blind faith in intelligence testing could be observed

in the present case study. A lot was expected beforehand, but after the administration

nothing changed in the ‘care’ process around the student, which reflects a certain

discontinuity in the process. Different opinions about the usefulness of the IAI were

noticeable. Educational professionals act rather intuitively, on short term vision, with

respect to the care provided in this case.

General discussion

An overview of the current state of care to at-risk students in Dutch educational

practice is provided by both studies described above. The results of the first study, in

which questionnaires to school psychologists, special care coordinators, and teachers

were distributed, indicated an important role of intelligence tests. However, answers

were often ambiguous and large differences between participants were observed,

especially with regard to the gap between diagnosis and intervention. Some schools

use the information from an intelligence test solely, others use alternative methods

such as classroom observations. Moreover, different views about the role of the school

psychologists indicate a difference in ‘care’ processes between schools.

The significance of intelligence tests to the development of interventions remained

unknown. Therefore, a case study was carried out in the second study. The focus

of the case study was the use of intelligence tests and its consequences for the ‘care’

process of a grade 1 student. It appeared that before the administering took place, a

lot was expected: participants aspired to get a solution for, and clues for actions with,

the student and her specific problems. However, the care provided to the student did

not change after the administration of the intelligence test. In summary, there is a need

for guidelines, but these guidelines are not provided by the individualised assessment


115

5

of intelligence. Besides, during the period of time of our case study, evaluation of the

‘care’ process between professionals was not undertaken, which also indicates a gap

between diagnosis of problems and the interventions provided.

The studies underlined the need for other formats of assessment that provide specific

guidelines for practice. Generally, assessment procedures focus on estimation of cognition

of the at-risk student. It can be questioned, however, whether learning problems of at-

risk students always originate in cognition. Next to cognitive aspects, other aspects that

play a role in learning processes, such as metacognitive and motivational aspects, should

be taken into account when providing guidelines for adaptive education in the general

classroom. In the case study the school psychologist not only used information from

the intelligence test, but also information from the teacher and classroom observation.

Still, a gap between his advice and the actions of the educational professionals in the

school was apparent. This indicates the need for enhancement of professionals’ skills

with regard to interpretations of tests scores. Even though this school was known for its

quality of care, gaps in the process could be detected, for example evaluation was not

undertaken during the case study. As already highlighted by Rouse and Agbenu (1998),

teachers should develop their skills in observation and assessment in order to bridge the

gap between assessment and intervention.

It should be noted that both studies were carried out in a rather small, specific sample

which could have biased outcomes. The educational advisory agency has shown interest

in dynamic assessment, resulting in more knowledge among their school psychologists

compared to school psychologists working at other educational advisories. Moreover,

the case study has been carried out at a school which is able to provide more care due to

financial aids by the government. This is not representative of schools in other regions

of the Netherlands. It is estimated that the results described in this chapter are rather

positive compared to other Dutch schools.

CHAPTER 5

116

With respect to the questionnaires it must be mentioned that answers to questions

could have been subjected to individual cases and experiences of participants. When

answering, they might have thought about specific students, resulting in differences in

answers to the questions. Individual beliefs, thus, play a significant role in answers to these

questions. This relates to the issues of self-efficacy and implicit theories of intelligence

in educational practice. As described in the previous chapter of this dissertation, beliefs

about the malleability of intelligence have their impact on the actions of educational

professionals. Therefore, next to knowledge about (dynamic) assessment, these beliefs

should be taken into account when intervening in educational practice.

Several improvements are needed in current Dutch psycho-educational practice. Other

formats of assessment, knowledge about assessment and diagnosis, awareness of one’s

own beliefs and their influence to one’s own acting, and the need for evaluation between

significant others in the learning process of student, are examples of aspects to improve

in order to bridge the gap between diagnosis and intervention. Therefore, in the next

chapter, the results of an intervention study that aimed at the development of these

skills will be reported.


117

5

Appendix | Examples of open-ended items and categories in coding scheme

Item of questionnaire for teachers

Which information do you want from the school psychologist?

Assessment results

Advice on specific needs of student

Information about learning potential / intelligence of student

Information about level and manner in which instruction should be effectuated

I don’t need any information

Unclear or no answer

Otherwise specified, namely:

Item of questionnaire for special care coordinators

In what way do you provide support to the teacher when encountering students with learning problems?

Via conversations

I work with the student myself (e.g., assessments, observations)

I provide advise (about e.g., intervention programs)

I evaluate students’ results



Item of questionnaire for school psychologists

Which information do you need to formulate your actions-based advise?

Information from teacher

Classroom observation

Information from parents (home situation)

Insight in opportunities at school

Test outcomes (information about student)



Chapter 6The next step in bridging the gap: a pilot study

to enhance teacher’s skills with at-risk children

This chapter is based on:Tiekstra, M., Minnaert, A., Hessels, M., & Verdel, R. (in prep.).

The next step in bridging the gap: a pilot study to enhance teacher’s skills with at-risk children.

CHAPTER 6

120

Abstract

It has been demonstrated that teachers rarely provide instruction tailored to specific

needs of students. Teachers seem to lack these tailored oriented skills, especially in the

case of at-risk students. In this pilot study, an intervention was developed to enhance

teacher’s skills in the general classroom in order to foster students’ learning potential.

Since implicit theories of intelligence influence teacher’s behaviour these were also

taken into account. A short intervention of four sessions was implemented at a school

team (N=21) of one primary school. The sessions focused on mediational techniques,

learning potential, the use of (dynamic) assessment, and translation to classroom

practice. The results showed a significant change in beliefs of intelligence, but not

yet in teacher’s behaviour. Multilevel growth curve analysis showed the significant

contribution of self-efficacy to the increase of incremental views on intelligence. It is

recommended that in order to change teacher behaviour, beliefs and self-efficacy should

be taken into account.

THE NEXT STEP IN BRIDGING THE GAP: A PILOT STUDY

121

6

Introduction

Students that are at-risk for learning difficulties, or seen to be eligible for special education,

often lack problem solving skills. These students, therefore, need more guidance in the

classroom. Several studies demonstrated that it seems difficult for teachers to provide

guidance which is tailored to the students’ needs (Steenbeek, Jansen, & Van Geert, 2012;

Sutherland, Lewis-Palmer, Stichter & Morgan, 2008). Dymond and Russell (2004) showed

in their observational research that teachers rarely adapt their instruction to the needs of

students with mild intellectual disabilities, while they do for students with severe disabilities.

In the latter case, teachers are often supported by an educational psychologist or other

professional, and consequently, feel less responsible for the learning processes of these

students in the classroom. It is a striking conclusion that students with learning problems

are submitted to general classroom instruction without any adaptation. Generally, teachers’

actions should be based upon Individualised Educational Plans (IEP). However, problems

emerge when translating IEPs to the classroom. Clearly, teachers lack skills to adapt their

instruction to mild problems, which are eminent in the daily inclusive classroom practice.

Since teachers play a leading part in learning processes of students (e.g., Hattie, 2009),

there is a need for teacher professionalisation or interventions at teacher level that aim at

improving their skills for adequately teaching students with mild intellectual disabilities.

However, a focus on skills solely is not in congruence with theory, since teachers’ beliefs have

been demonstrated to also play a role in the instruction teachers provide. Teachers’ self-

efficacy has an impact on behaviour in the classroom (e.g., Tschannen-Moran & Woolfolk

Hoy, 2001). Self-efficacy refers to the belief in one’s own capacity to achieve a certain

performance, and is context dependent (Bandura, 1997). Imants and De Brabander (1996)

define teachers’ sense of efficacy as “the extent to which teachers believe they can affect

student learning” (p.179). More specifically, as demonstrated by Gibbs and Powell (2012),

teachers’ self-efficacy is composed of their belief in efficacy for classroom management,

children’s engagement, and instructional strategies. Previous experiences and feedback

from the environment influence self-efficacy.

CHAPTER 6

122

Furthermore, teachers’ actions in daily classroom practice are influenced by implicit

theories of intelligence (Tiekstra & Minnaert, submitted). Dweck and colleagues have

shown in their extensive research that people hold implicit theories of intelligence, with

either an incremental or entity view, and that these views affect people’s behaviour (e.g.

Dweck, Chiu, & Hong, 1995). In the incremental view intelligence is considered as

malleable, which is in contrast to the entity view in which intelligence is seen as something

stable and not subjective to change. Implicit theories of intelligence influence the

quality of interactions, as was demonstrated by Moorman and Pomerantz (2010) with

regard to mothers and their children. This also holds for teachers (Tiekstra & Minnaert,

submitted). For instance, teachers supporting the entity view tend to practice direct

instruction with, essentially, a focus on performances. Teachers with an incremental

view, on the contrary, support mastery-oriented learning and focus on the learning

process instead of the learning result. Jordan and Stanovich (2001) demonstrated that

teachers with an incremental view were more frequently involved with at-risk students

than teachers holding an entity view, since these teachers feel more responsible for

their students’ learning processes. Implicit theories of intelligence, thus, affect the way

in which instruction is effectuated in the classroom. Accordingly, students’ learning is

influenced by different teaching strategies (Rozendaal, Minnaert, & Boekaerts, 2005;

Timperly & Parr, 2009). This means that an incremental approach is important for the

classroom situation: intelligence should be seen as malleable, and every individual has

potential which could be fostered by teachers’ actions.

The intervention (theoretical background)The aim of this study was to explore the effectiveness of an intervention aiming at the

enhancement of teachers’ skills and incremental views in order to foster students’ learning

potential. An incremental view on intelligence is a prerequisite for fostering learning potential

in the classroom. Blackwell, Trzesniewski and Dweck (2007) demonstrated the effectiveness

of intervening on theories of intelligence. More specifically, students who were taught that

intelligence is malleable (incremental view) showed more positive motivational patterns

which, in turn, caused higher grades. This underlines the need for incremental views in


123

6

education, and that intervening on such core beliefs can have huge impacts on students’

learning. The intervention in the study of Blackwell et al. (2007), however, was targeted at

students. In our intervention, in line with the above, we self-evidently intervene on the level

of teachers, since they play a crucial role in fostering a student’s learning potential.

To realise a student’s potential, teachers should explore and, accordingly, teach in the

student’s zone of proximal development (ZPD). According to Vygotsky (1978), only in this

way learning occurs. The teacher has a mediating role, and should be sensitive to the specific

needs of students. Feuerstein’s Instrumental Enrichment is an operationalisation of these

Vygotskian principles, and was originally developed for individualised intervention situations

(Feuerstein, Rand, & Hoffman, 1979). It is possible, though, to translate the intervention

principles to general classroom education. By asking mediating questions and focus upon

students’ learning processes, general cognitive teaching in the ZPD can be fostered.

Currently, teachers lack the skills and knowledge to execute such cognitive education, as

highlighted by Hessels and Hessels-Schlatter (2013).

Since teachers play a paramount role in students’ learning processes, they should be

reflective upon their own actions (Jennings & Greenberg, 2009; Steenbeek et al., 2012).

Insight in their own learning and acting has a positive influence on teachers’ effective

behaviour in individual differences of students (Rosenfeld & Rosenfeld, 2008). Moreover,

it has been demonstrated that reflecting and evaluating is an important feature of teacher

professionalisation (Meirink, Meijer, Verloop, & Bergen, 2009; Runhaar, Sanders, & Yang,

2010). Therefore, an important aspect of our intervention was to emphasise the role teachers

play in learning processes and to help reflect collaboratively on this issue.

In sum, the rationale of our intervention was that in order to develop a dynamic, (meta)

cognitive teaching style in teachers that fosters the students’ learning potential, one

should support an incremental view of intelligence. Since Meirink et al. (2009) showed

that changing teachers’ beliefs is a complex task, the focus of the intervention was

more on teachers’ activities in the classroom with at-risk children, than on changing

CHAPTER 6

124

their implicit theory of intelligence. In this way, the intervention was closely tailored to

teachers’ experiences in the classroom, accounting for the benefit of the implementation

of the intervention.

Interventions in educational contextsWhen implementing interventions one should take into account the ecology in which

the intervention takes place. Prince, Tiekstra, and Minnaert (2014) showed that the

educational context is composed of a web of agents who all have their own beliefs

and intentions, which causes a complex ecology for interventions. To eliminate

implementation bias, these authors stressed the necessity for collaboration, and a shared

vision and language among participants, including the researcher.

Yeager and Walton (2011) also emphasised the importance of taking contextual effects

into account when generalising effects of educational interventions. Several authors

(e.g., Boekaerts & Minnaert, 2003; Levin & O’Donnell, 1999; Sternberg et al., 2006)

recommend testing interventions in pilot studies first, before executing large scale

randomised control trials (for more information, see Prince et al., 2014). Accordingly,

the relevance and effectiveness of the content of our intervention was studied for the

first time by implementing it in one school only. In this way, the context remained

controllable over time and we could tailor the intervention to exclude contextual

influences and improve implementation. Although school-specific, the theoretical

principles of the intervention remained the same. The main research question in this

study addressed the extent to which the intervention accounted for a change in belief

and behaviour of the participants, taking teachers’ self-efficacy into account.


125

6

Method

Participants The school that was selected for this study was a primary school in the western part

of the Netherlands. This area accommodates more immigrant families compared to

other parts of the Netherlands, implying a very diverse school population. This school,

however, can be characterised as a typical, average school.

Our emphasis on a “shared language” among school members demanded the

participation of the whole school team. The school board endorsed this viewpoint

and agreed to participate in the first training session to show their support. Generally,

support of directors is in favour of implementation (see Prince et al., 2014). Every

educational professional working at the school participated, which resulted in a total of

N= 21, all females.

The school team consisted of three elementary school teachers, seven middle school

teachers (grade 3 – 5), four teachers for the upper-level classes (grade 6 – 8), and for one

teacher it was unknown in which grade she taught. Next, six participants mentioned

that they taught in several classes, one of whom was a remedial teacher, one school

psychologist, the general coordinator (assistant director), and the special care coordinator

(assistant director). The mean working experience of the school team was 10,3 years.

Nine participants were very experienced (> 10 years), five had average experience (4 –

10 years), and five were less experienced (0 – 3 years). All participants will be referred to

as “teachers” in the rest of the article.

ProcedureThe intervention comprehended four two- hour sessions over a period of two weeks at

the beginning of the school year (September/October 2013). The study comprised a pre-

test, post-test, and a follow-up measure. The baseline measurement (pre-test, N =21) was

administered before starting the first training session (week 0). Immediately after the last

CHAPTER 6

126

training session, the post-test (N =15) was conducted (week 2). The follow-up measurement

(N =19) was carried out four weeks after the last training session (week 6).

Training. Several studies of A. Minnaert and colleagues (personal communication, March

11, 2013; May 23, 2014) showed that a short period of intensive training with a maximum

of two weeks provides more robust longitudinal effects than a training which lasts several

months. Moreover, Meirink et al. (2009) underlined the importance of a combination

of collaborative reflection with colleagues and immediate utilisation of new ideas in the

classroom when changing teacher’s beliefs. Therefore, four two-hour training sessions

distributed over a period of two weeks were carried out. In this way, learned material could

be tested in the classroom and each session started with a reflection on participants’ own

experiences. Sessions focused on mediational principles, reflection upon role of the teacher

in the learning processes, the use of assessment outcomes for setting up IEPs, and placed

emphasis on learning potential and the malleability of intelligence. See the appendix for a

more detailed description of the individual training sessions.

Implementation. In anticipation of the research, to warrant implementation, a series of

meetings was organised with a local educational advisory agency to develop the training.

In a next step, a meeting with the school board took place to tailor the intervention to the

specific needs of the school. As mentioned, the intervention aimed at the whole school team

to warrant a shared language within the school. The training was conducted by the first

author (who is an educational psychologist) together with two professional trainers from the

educational advisory agency. This assured control over the exact content of the training.

Moreover, observations and evaluations of each training session provided the feedback

needed to adjust the training where needed to guarantee its full applicability. The two

members of the school board who participated in the training provided the researchers with

feedback about the reactions within the school team after each training session, allowing

us to further adjust, or to attend to certain aspects more thoroughly.


127

6

Measurements A questionnaire was administered to measure the effects of the intervention on several

variables listed below. Additionally, observations were carried out during the training

sessions and interviews after each training guided the implementation process.

Teacher behaviour. Teachers’ behaviours in the classroom were evaluated by

means of a questionnaire comprising statements with a vignette. Six questions followed

a vignette that described a specific classroom situation around a student in grade 3

with a learning problem. These focused on daily classroom practices, and had anchors

indicating either a dynamic (incremental) approach (4) or a static (entity) approach (1) of

the case. For example, item 2 addressed the individual approach to the specific student:

During a one-to-one moment with student M.,… I explain the exercise once again (static approach),

or …I ask the student how she is going to tackle the problem in the exercise (dynamic approach). The

other items referred to teacher’s behaviour in group wise instruction, student’s learning

problem, consultation of the special care coordinator, and setting up and evaluation of

an IEP. The higher the total score, the more dynamic is teacher behaviour. In this way

we could investigate whether teachers’ intentions in the daily classroom practice had

changed after our intervention. The vignette showed a reliability of Cronbach’s α .68 at

pre-test. A vignette similar to the pre-test (same classroom situation, different problems)

was administered six weeks later at the follow-up (α .61).

Implicit theories of intelligence. Next to behaviour, implicit theories of intelligence

(ITI) were assessed. The validated and widely used ITI-scale of Dweck et al. (1995) was

converted into items appropriate for Dutch teachers. The scale, comprising three items,

was administered at pre-test (α .85), post-test (α .82), and follow-up (α .88). A higher

score indicates a greater entity (static) view on intelligence (min. 3, max. 15).

Self-efficacy. Five items (pre-test, α .82) assessed self-efficacy of teachers. This scale

was retrieved from Opdenakker (2008) who tested the scale in different educational

contexts. Mean scale scores (min. 5, max. 25) were calculated to be used in the analysis.

CHAPTER 6

128

A higher score indicates a greater sense of self-efficacy.

Observations and interviews. Each training session has been observed and

evaluated by the trainers (including the researcher). Moreover, each session was

evaluated by the school board. They informed the researcher about reactions and

changes within their team during the intervention. After the administration of the

follow-up questionnaire (week 6) a final interview took place with several participants

of the training, and the school board in particular. This interview focused on perceived

differences within the school team.

AnalysisIn order to measure the effect of the training, descriptive analyses of scale and data

distributions were carried out first. Effect sizes (Cohen’s d) were calculated per item

measuring teacher behaviour. To minimise capitalisation on chance, it was decided that

only Cohen’s d’s of ≥ .50 accounted for a change.

The change in participants’ beliefs could be measured at post-test and follow-up.

Unfortunately, at post-test, we were confronted with a dropout rate of 33%. A detailed

analysis revealed that participants with a rather static view at pre-test did not participate

at post-test. A mixed model approach was employed to explore whether results were

independent of this dropout. Mixed models do not simply impute missing values, but

calculate a function to estimate these missing values, based on the functions that the

model estimated for the rest of the data. We conducted a specific kind of mixed models,

that is multilevel growth curve analyses in which measurements (time) were units within

participants, on the implicit theories variable. In this way, a trend analysis of the ITI

measure on pre, post and follow-up could be established independently of dropout.

It was further investigated whether self-efficacy influenced the outcomes in a next

multilevel growth curve model.


129

6

The qualitative information from the final interview has been used to complement and

illustrate the results of these analyses.

Results

Change in behaviourVignette. The descriptive analysis of the pre-test showed that scores on the vignette-

scale were quite high already (M= 20,6 out of max. 24), and the effect size of difference

between pre-test and follow-up of the vignette as a scale appeared to be relatively

low (Cohen’s d -.09). However, detailed analysis of each item separately showed some

substantial differences in distributions of scores on pre-test and follow-up. Therefore,

effect sizes were calculated for each item separately. As showed in Table 1, these ranged

from -.43 to .65. The considerable effect of d. = .65 was found for item 2 referring to

individual instruction. Since a higher score implied more dynamic behaviour of the

teacher, a positive Cohen’s d was desired.

Table 1 | Overview of effect sizes per item measuring teacher behaviour

Item Cohen’s d

Group wise instruction -.43

Individual instruction .65

Student’s learning problem .09

Special care coordinator consultation -.41

Setting up IEP -.06

Evaluation of IEP -.27

CHAPTER 6

130

Interview. The final interview with the school board and several teachers revealed

that the school team had re-evaluated their group and individualised educational plans

several weeks after the intervention. Moreover, the school board remarked a difference

in some teachers with respect to their approach of students, and in the format of their

meetings about students. According to the school board, a trend was noticeable in the

change of behaviour of the school team.

Change in beliefImplicit theories of intelligence. Descriptive analysis revealed a trend in the mean

scores of the participants: from a relatively strong entity view on intelligence at pre-test

(M = 7,6) to a rather incremental view at post-test (M = 5,6). This trend flattened out at

follow-up (M = 6,4). We investigated whether these results were biased by the dropouts

at post-test, which turned out not to be the case. The multilevel quadratic growth curve

model fitted the data best. The decline in static view appeared to be significant (Time p

<.001), as was the upward trend between post-test and follow-up (Time2 p .003). Table

2 shows the modelling results. A picture of the trend in implicit theories of intelligence

can be seen in Figure 1.


131

6

Table 2 | Multilevel growth curve model of implicit theories over time (N = 54)

Implicit theories of intelligence

Coefficient SE

Fixed effects

Intercept 7.619* .466

Time -2.600* .811

Time2 .755* .256

Random effects

Level 2 variance

Intercept 2.086 1.455

Time .049 .279

Intercept*time .080 .472

Time2 .000 .000

Intercept*time2 .000 .000

Time*time2 .000 .000

Level 1 variance

Residual 2.476* .938

Deviance 229.486

Note. * p ≤ .01

CHAPTER 6

132

Figure 1 | The development of implicit theories of participants at pre-test, post-test and follow-up (with confidence interval), independent from dropouts at post-test. The higher a score, the more entity oriented a view.

Dw

eck

fact

or

0,0 1,1 2,2

time

3,6

5,4

7,2

9,0Implicit theory of intelligence

Self-efficacy and implicit theories of intelligence. The question arose whether

self-efficacy influenced the curve of the model. Another quadratic multilevel growth

curve model, including self-efficacy intercept, and interactions of self-efficacy with time

and time2 (quadratic part), appeared to be the best fitting model to the data. Table 3

shows the outcomes of this analysis. The fixed effect of self-efficacy is not significant.

However, the interaction with time is nearly significant (self-efficacy*time p = .051) at

5% level. The upward trend is also significant at 10% level (self-efficacy*time2 p = .095).

Figure 2 provides more insight into this issue. The decline between pre-test and post-test

is different for participants with more or less self-efficacy.


133

6

Table 3 | Multilevel growth curve model of implicit theories over time, taking self-efficacy into account (N = 51)

Implicit theories of intelligence

Coefficient SE

Fixed effects

Intercept 8.583** 3.214

Time 7.491 5.224

Time 2 -1.971 1.638

Self-efficacy -.221 .797

Self-efficacy * Time -2.549 † 1.31

Self-efficacy * Time 2 .686 † .411

Random effects

Level 2 variance

Intercept 1.791* .908

Time .000 .000

Intercept*time .000 .000

Time 2 .000 .000

Intercept*time 2 .000 .000

Time*time 2 .000 .000

Level 1 variance

Residual 2.404** .605

Deviance 210.494

Note. * p ≤ .05, and ** p ≤ .01, and † p ≤ .10

CHAPTER 6

134

Figure 2 | The influence of self-efficacy on scores on implicit theory of intelligence (the higher a score, the more entity oriented a view) on pre-test, post-test, and follow-up. The line at the top indicates a relative low mean score (3) on self-efficacy, whereas the line at the bottom implies a high mean score (5) on self-efficacy.

The higher self-efficacy at the start of the intervention, the more decline in static views

(nearly at 5% alpha level), and the more maintenance effect of this decline in the follow-

up measure. It should be noticed, however, that the interaction effect in the quadratic part

(upward line between post-test and follow-up) was significant at 10 % alpha level. This is

related to power problems: the small sample size affects the level of alpha (Carmines &

Zeller, 1979). The influence of self-efficacy in the enhancement of an incremental view is

shown in figure 2. These results indicate a differential effect in which self-efficacy makes a

difference in the long term in favour of the incremental view.

Dw

eck

fact

or

0,0 1,1 2,2

time

3,6

5,4

7,2

9,0SEgem.pred = 3SEgem.pred = 3,5SEgem.pred = 4SEgem.pred = 4,5SEgem.pred = 5


135

6

DiscussionThe goal of this pilot study was to investigate whether teachers’ skills to effectively adapt

their instruction in order to foster student’s learning potential in the classroom could

be enhanced. A short, intensive, intervention was implemented in one school team (N=

21) to explore the extent to which their behaviour and beliefs would change during

the intervention. The results showed a substantial change in participants’ beliefs about

intelligence, but did not provide enough proof for a change in teacher’s behaviour.

Change in beliefMultilevel growth curve analysis showed a significant decline in the trend of static

implicit theories of intelligence after the training: the rather static (entity) views at the

beginning changed in more dynamic (incremental) views immediately after the training,

which remained present at follow-up, although to a lesser extent. The intervention

provided more incremental views on intelligence among participants which sustained

over (at least) six weeks of time.

The effectiveness of the training, however, was moderated by self-efficacy of participants.

More self-efficacy at the start of the project seemed to produce more decline in static

beliefs about intelligence over time. This illustrates the need for emphasis on self-efficacy

in teacher professionalisation, implying that actions in the classroom should be a key

aspect in interventions in order to achieve effects. Teachers learn from their actions,

which demands for a recursive design of studies (see also Boekaerts & Minnaert, 2003).

In this pilot study, teachers were trained in metacognitive teaching and reflection upon

their own teaching. Although the change in beliefs of participants was a promising

result, reflection, an explorative attitude towards students’ learning problems, and

metacognitive teaching should be embodied in teacher education. The teacher’s belief

in own actions in the classroom may be an important contributor to incremental views

on intelligence in educational practice, as well as it might be to skills for metacognitive

teaching.

CHAPTER 6

136

Change in behaviourDue to the restriction in measurement of change in behaviour in this pilot study, it remained

unclear whether these changes in beliefs affect changes in classroom situations. Some

outcomes indicated a change in behaviour. The vignette study showed a significant change

with a large effect size in a more dynamic approach of individual learning situations in

this school. Due to an unfortunate ceiling effect, however, the vignette can be regarded as

not discriminative enough. Moreover, it was a self-report measure, and there was a lack

of observations in the classroom that could have been used to assess change in teacher’s

behaviour. Nevertheless, the final interview with the school board and several teachers

revealed some promising indications for change of behaviour taking place in the school

team. Perhaps, the information of the intervention needed more time to be assimilated

by the teachers, which implies that the results in change of behaviour should have been

measured after a longer period of time.

The change in belief about IQ testing is an important finding with respect to the

implementation of dynamic assessment (DA) in practice. As highlighted by several

authors (e.g., Deutsch & Reynolds, 2000; Elliott, 2003), dynamic assessment is only

scarcely implemented in practice, regardless of many advantages over static testing in

at-risk populations. As a next step in bridging the gap between assessment and classroom

instruction, this pilot study showed that specific training is needed in order to change the

beliefs about IQ testing within school teams. In order to effectuate full implementation

of DA in educational practice, specific training of educational psychologists in dynamic

assessment is needed. Yet, being another important aspect to this issue, teachers too,

should embody a general “incremental attitude” towards learning (Hessels & Hessels-

Schlatter, 2013).

Future directionsThe current pilot study revealed some important aspects to consider in future research.

As Yeager and Walton (2011) underlined “an incremental mindset intervention might

have no effect if students believe that the person who tells them about their potential


137

6

for growth and improvement does not believe this himself or herself.” (p.290). Although

our intervention focused on behaviour of teachers in the classroom, a positive change

at student level was desired. Therefore, we recommend taking students’ outcomes into

account in future research.

Furthermore, since the pilot study was conducted at one single school, the results could

be subject to specific characteristics of this school. Studies at multiple schools should

be performed to investigate the school effects. Schools in which the intervention is

implemented should be compared to schools in a control group. Observations in the

classroom context should be carried out instead of self-report measures. Moreover,

results should be measured in a long term perspective, to investigate the maintenance

effect of change in belief and behaviour.

This pilot study showed that a short intervention changed participants’ static beliefs

about intelligence into more incremental (dynamic) views. Despite the fact that a change

in one’s own beliefs is influenced by one’s individual pre-existing beliefs (Meirink,

et al., 2009), the whole school team demonstrated a decline in static views. This is

a promising outcome and proves ground for future research in the issue of fostering

student’s potential in the classroom. The decline in static views over time was subject

to participants’ self-efficacy. In order to change teacher behaviour, the contribution

of beliefs and self-efficacy to this issue should not be disregarded. Therefore, when

trying to change behaviour in interventions, beliefs and self-efficacy should be taken

into account.

CHAPTER 6

138

Appendix |Description of the training sessions

Training session 1

Goal To experience the connection between vision on individual differences and one’s own teaching actions in the classroom

Contents Participants’ views on intelligence How do you act upon individual differences in intelligence?

Intended aims

belief

behaviour

Individual differences demand an incremental view on intelligence

A shared language among members school team

Training session 2

Goal To learn about the role of the teacher in student’s learning process

Contents Zone of Proximal Development Role of the teacher as mediator in individual learning processesMetacognitive teaching in individual learning situations

Intended aims

belief

behaviour

Every student can learn (= incremental view on intelligence)

Enhancing skills in metacognitive teaching


139

6

Training session 3

Goal To learn about the role of the teacher in the general classroom and at-risk students

Contents Steps in cognitive and motivational developmentSetting up group vs. individual educational plansSetting up “dynamic” educational plansAt-risk students in the classroom and the role of the teacher

Intended aims

belief

behaviour

Teachers play central role in intellectual development student

Setting up educational plans as guidance to tailored instruction; focus on student’s outcomes & teacher behaviour when reflecting upon plans

Training session 4

Goal To learn how to use (meta)cognitive teaching strategies (“mediation”) in daily classroom practice

Contents Mediational techniques during dynamic assessmentLink to classroom situation: mediational techniques in practice

Intended aims

belief

behaviour

Incremental view on learning

Enhancing metacognitive skills in group wise and individual learning situations

“I cannot teach anybody anything. I can only make them think.”(Socrates)

Chapter 7General Discussion

CHAPTER 7

142

GENERAL DISCUSSION

143

7

The first chapter of this dissertation started with the case of Yamina. This example from

daily classroom practice illustrated the gap that presently exists between assessment and

instruction. In an attempt to bridge this gap, more information was needed about the

consequential validity of assessment procedures. The aim of this thesis was to explore how

the consequential validity of (dynamic) assessment could be increased. This can be seen as a

methodological and theoretical approach to the “bridging the gap-issue” which is present in

educational practice. Therefore, the outcomes will be reviewed in this chapter by means of

another description of the case of Yamina:

When the problems of Yamina worsened, the teacher and remedial teacher decided to take action. The teacher

developed a series of measurements that linked with his instruction, and started to monitor Yamina’s progress to

gain more insight in her response to his instruction. He learned a lot with respect to his own teaching when he

interpreted Yamina’s results to these curriculum based measurements. Based on these evaluations the remedial

teacher in turn decided to provide reading instruction in small groups of students that make the same kind of

mistakes as Yamina. Although the adaptations in the instructions seemed to improve Yamina’s reading results,

she still had problems with finishing her tasks and keeping up with her classmates. The teacher and remedial

teacher put in a request for an individualised assessment by the school psychologist. The school psychologist

listened carefully to the experiences of the teacher and remedial teacher and decided to administer a dynamic

assessment procedure to gain more insight in Yamina’s problems. The learning potential test estimated Yamina’s

potential as much higher than expected. Moreover, the school psychologist remarked that Yamina responded well

to his metacognitive hints during the assessment. He also noticed that Yamina revived when she experienced her

successful responses to the test. The school psychologist was able to write a detailed report from the outcomes of

the dynamic assessment procedure. His advice was to especially support Yamina in the metacognitive aspects

of learning, such as planning and monitoring, and not to focus on her cognitive learning outcomes alone. Since

Yamina has a good potential for learning, the support in the metacognitive area may help her to further develop

her reading skills. The school psychologist predicted that Yamina would be much happier, and insisted upon

evaluation with the teacher and remedial teacher in three weeks.

This can be seen as a mindful and ideal situation based on a grounded assessment in which

the consequences of test outcomes serve as a guiding principle for classroom instruction and

CHAPTER 7

144

intervention: the gap between diagnosis and instruction has been bridged! This dissertation

explored the issues that are needed in order to achieve such an ideal situation in educational

practice.

Summary of findings As explained in chapter one bridging the gap not only involves a critical review of current (dynamic)

assessment procedures, but also a critical view on the significant others in educational practice.

Both sides have been investigated in this thesis. Chapters two and five focused at the consequences

of current (dynamic) assessment, chapters three, four, and six focused at significant others in the

learning processes of at-risk students.

Since dynamic assessment can be seen as of added value to traditional IQ testing in at-risk students

(see chapter 1), and the learning phase during administration could provide useful guidelines for

practice, the literature review in chapter two focused on current dynamic assessment procedures.

A critical review of 31 articles that met the inclusion criteria, in which 29 different assessment

procedures were described, revealed that, although promising, these procedures did not yet

provide explicit indications for practice. While proximal consequential validity was warranted

(i.e., test administration can be adapted to the responses of the student), it remained questionable

to what extent distal consequential validity was fostered: the majority of tests resulted in a number

(raw score) and only few also included an indication of the level of help needed, but no explicit

guidelines resulted from the procedures to underpin interventions. To assess the potential value

for bridging the gap, we focused at the learning phase of each procedure. It appeared that during

most of the procedures the examiner addressed cognitive and/or metacognitive aspects, whereas

motivational aspects never played an explicit role. This is a striking conclusion, since dynamic

assessment is known for the attention to non-intellective factors. In sum, although dynamic

assessment procedures have potential, they do - in practice - not contribute to bridging the gap

between assessment and instruction in their current formats.

In chapter three we elaborated on the extent to which a student responded to quality

instruction. During dynamic assessment qualitative one-to-one instruction is provided,

GENERAL DISCUSSION

145

7

and the extent to which a student profits from this instruction is being measured. In

other words, the extent to which a student is didactically resistant, one of the criteria for

the diagnosis of learning disabilities, is being assessed. But how can this be translated

to general classroom practice? The RTI model seemed to be an instrument to make

this translation possible. In a qualitative multi-method study, the RTI model was

implemented in Dutch educational practice. The results showed that the RTI model

can be a valuable tool, but that the way in which the model is implemented is the

decisive factor. Not only behaviour of participants needed to change (i.e., they executed

the model), but more importantly, their beliefs about teaching and didactic resistance

needed to change to make meaningful interpretations about student’s responses to their

instruction. We concluded that the RTI model can be a valuable tool to measure didactic

resistance, but only if teachers have a dynamic view on learning and instruction, and

thus, acknowledge their role in student’s learning processes.

We regarded this division between the change of belief and the change of behaviour

of participants as an important aspect in improving the consequential validity of

assessment procedures. Therefore, we wanted to explore the influence of beliefs on

teaching behaviour of professionals in the Dutch educational context more in-depth.

More specifically, the extent to which implicit theories of intelligence influence behaviour

in the case of at-risk children has been investigated in a quantitative study in chapter

four. It appeared that implicit theories of intelligence play a prominent role in daily

classroom practice. In this sample, 34% of the total variance in the actions of teachers

can be explained by their implicit theories of intelligence, and in denominational

schools this percentage is even 61%. This means that teaching in the Zone of Proximal

Development is influenced by teachers’ implicit theories of intelligence. Furthermore,

a full mediational structural equation model was found in the group of support

professionals: implicit theories predict the belief about IQ tests, and, in turn, explains

the way people act according to the outcomes of these tests. The consequential validity

of tests is, thus, caused by people’s implicit theories of intelligence.

CHAPTER 7

146

Since implicit theories of intelligence, and herewith the faith in testing results, seemed to be

important for improving consequential validity, we wanted to investigate what role tests and

testing results play in practice to date. Therefore, chapter five reported on two qualitative in-

vivo studies. In the first study, an open-ending questionnaire amongst 36 school psychologists,

21 special care coordinators, and 44 teachers, revealed ambiguous answers about the use of

test results, and the tasks of significant others in educational practice. Therefore, a case study

has been set up in the second study. This study also confirmed the gap between assessment

and instruction: the teacher as well as the special care coordinator indicated a need for

guidelines. However, the results of the IQ test did not provide specific information to guide

their instruction. More specifically, the care for and instruction to the student did not change

after the administration of the IQ test. To conclude, these in-vivo studies underlined the

necessity for other formats of assessment in current Dutch educational practice.

The above-mentioned studies showed the need for more knowledge and skills among

educational professionals in order to foster students’ learning potential in the general

classroom. The influence of implicit beliefs concerning the malleability of intelligence

should be recognised. In chapter six we, therefore, reported on a pilot study in which a short

intensive intervention was implemented with one school team (N=21). This pilot study

provided insight in the processes that are needed to change current psycho-educational

practice in order to bridge the gap between assessment and instruction. It appeared that

the short intervention accounted for a change in beliefs of participants: a decline in static

views was visible among the whole school team. Besides, the multilevel growth curve model

showed that the decline in static views was subject to participants’ self-efficacy: the more

self-efficacy one has, the more decline in static views, even in the long term. A change in

behaviour could, unfortunately, not yet be proven due to restrictions in the measurements. It

was concluded that if we want to change behaviour of educational professionals, we should

also take beliefs and self-efficacy into account.

All studies underlined the complexity and interrelatedness of several factors with respect to

fostering student’s potential to learn in the general classroom. In the next sections theoretical

GENERAL DISCUSSION

147

7

and methodological considerations will be addressed. Furthermore, implications for

practice and policy will be highlighted. Finally, conclusions will be listed. The ideal case

of Yamina described above will be used to illustrate these considerations.

Theoretical considerationsMany types of, and approaches to, validity exist (Kerlinger & Lee, 2000). This thesis is

centred on the concept of ‘consequential validity’. Although Messick (1989) underlined

the fact that this concept should not be seen as a separate type of validity, but rather

as an aspect of construct validity, emphasis should be put on the importance of his

message, and we therefore used the term separately. Messick (1995) stated that “validity

is not a property of the test or assessment as such, but rather the meaning of test scores”

(p. 5). The application, and herewith the consequence(s), of test scores is a type of

validity that should not be neglected. Especially within the context of learning and

development, where the ultimate purpose of assessment is to guide actions in the

classroom, consequences are of paramount importance in validity research. Therefore,

consequential validity can be seen as the ultimate type of validity, while predictive and

content validity can be considered as conditional stages to be reached before one can

say something about the extent of consequential validity. Therefore, test developers

should never disregard the consequences in validity studies.

Currently, consequential validity does not play a central role in validity studies of

assessment. This is a missed opportunity, since consequential validity can be seen as

the key element in bridging the gap between assessment and instruction in practice.

This gap, as shown in chapter five, is still present in daily classroom practice. Dynamic

assessment seems promising for bridging this gap. During dynamic assessment the

proximal consequential validity is warranted. After all, the examiner adapts hints

and feedback to the responses of the examinee in the learning phase: instruction is

influenced by test performance. A careful administration of these adaptations could

enhance the distal consequential validity of the test procedure. The administration

of hints and feedback that the student needs to solve the task relate to adaptations

CHAPTER 7

148

that could be made in instruction to optimise student’s learning. In short, the learning

phase during dynamic assessment plays a prominent role in bridging the gap between

assessment and instruction.

If dynamic assessment procedures strive to measure learning potential, then significant

others need to develop skills in fostering the learning potential. The beliefs of these

significant others play a prominent role in fostering learning potential. People’s implicit

theories of intelligence influence the way test scores are used. Furthermore, test scores

affect the way people think about abilities of the student. With respect to the case of

Yamina, the consequences of the individual assessment by the school psychologist could

have been different to the individual assessment (same test, same student) administered

by another school psychologist. It is, therefore, important that tests provide valid

scores. Moreover, educational professionals who work with (at-risk) students should

have an incremental view on intelligence, i.e., intelligence being seen as malleable. An

incremental view on intelligence can be seen as a prerequisite for the implementation

of dynamic assessment in current educational practice.

Moreover, the incremental mindset is a prerequisite for the operationalisation of

‘didactic resistance’. The concept should be measured at both student and teacher level.

After all, it is hard to say anything meaningful about the extent to which a student is

didactically resistant without knowing what kind of instruction has been offered by the

teacher or remedial teacher. In the ideal situation described above, Yamina’s teacher

decided to undertake action in order to examine Yamina’s didactic resistance. The

teacher evaluated his own teaching by curriculum based measurements, the remedial

teacher tried to adapt small group instruction to Yamina’s needs, and an evaluation

moment with the school psychologist will take place in three weeks. In other words,

an incremental view of educational professionals accounts for the acknowledgment of

their influence to the learning processes of their students. Only in this way, the extent

of ‘didactic resistance’ can be measured.

GENERAL DISCUSSION

149

7

Methodological considerationsThe two-sided approach to the issue of consequential validity, i.e. the focus upon tests on

the one hand, and significant others on the other hand, demanded for multiple studies to

be carried out. This explorative approach can be regarded as a methodological strength

of this thesis. The combination of qualitative and quantitative methods accounted for

a strong focus upon contextual influences. Therefore, the conclusions can be seen as

ecologically sound. Moreover, the different studies are strongly connected, accounting

for the internal validity of the thesis. Each study was anchored on the previous one,

exploring an issue more in-depth, or using a different approach to the problem.

A recurrent problem, however, was the attrition of participants. Apparently, the studies

addressed emotive subjects for participants in the field. After all, views on learning and a

critical view on actions undertaken with at-risk children come very close to participants’

actions and their sense of self-efficacy. In some participants this might have provoked

resistance, resulting in an avoidance of interest in the studies undertaken. As addressed

in chapter three and six, the group that dropped out of the studies did not adhere to

or understand a dynamic view on learning. I will further address this issue in the next

section.

The strengths of the thesis, the multiple explorative studies and a strong focus on the

context, leads us to the weaknesses of the thesis. Since the studies were carried out in

small sample sizes, one could question the generalisability of the outcomes to educational

practice in the Netherlands, or even abroad. The tension between statistics (focus on

generalisability) and need for individual approach in the case of at-risk children is

emphasised in this thesis. Although quantitative information provides knowledge about

general processes in at-risk students (e.g., the role of working memory in many learning

deficits), it does not provide information that is needed in at-risk cases that are often

surrounded by individual and specific characteristics. Qualitative information is needed

to reveal the specific processes that cause the problems of individual at-risk cases: for

example, one dyslexic student can be very different from another, and consequently,

CHAPTER 7

150

different interventions are needed. Accordingly, a mixed method approach, in which

both quantitative and qualitative methods are applied, is needed when studying consequential

validity. After all, the context plays an important role with respect to consequential validity,

which can be best studied by qualitative methods.

The explorative studies showed the importance of beliefs when changing behaviour. This

can be seen as a first step in ameliorating the implementation of dynamic assessment in

educational practice. The matter is, however, not yet settled. If we wanted to improve

consequential validity, the change in behaviour should have been measured, too. The

next step is, therefore, to focus on the transfer of beliefs to behaviour. More importantly,

future research should focus on the measurement of teaching behaviour of educational

professionals, especially in the case of at-risk children. However, unmanageable processes

that are characteristic for educational practice make it hard to study behavioural change.

In this light, observations could be combined with other, more generalisable, instruments.

Hence, the use of mixed methods is the best way to investigate the complex system of beliefs

and behaviour.

Implications for practice and policyCurrent educational policies demand for inclusion of students with all kind of disabilities in

the regular classroom (UNESCO, 2005). In the Netherlands, more particularly, the policy

of “Adaptive Education” forces schools to provide targeted care (a term in line with the

DSM-V) for all their students. If we want this targeted care to be effectuated properly, then

changes are needed in current educational practice in the Netherlands. In this process,

bridging the gap between assessment and instruction is an absolute requirement to improve

educational practice. After all, adaptive education demands for an individual approach to

students’ learning processes. Moreover, adaptive instruction demands for a critical approach

to the instruction provided. Besides, as demonstrated by this dissertation, if we want to

change behaviour, beliefs should be addressed first. Next to educational professionals in

the schools, such as teachers, remedial teachers, and care coordinators, this involves school

psychologists, test developers, and policy makers.

GENERAL DISCUSSION

151

7

The process from assessment to instruction is composed of several steps. Test scores

result from the test administration. These need to be interpreted and translated to

the educational practice accurately. In order to provide quality instruction, finally,

instruction should be adapted to the outcomes of the measurements. Several stakeholders

interact in these different steps. If we want to bridge the gap between assessment and

instruction, and accordingly, increase the consequential validity of assessment, the next

recommendations should be followed.

Firstly, recommendations at the level of test developers will be addressed. Test developers

should think about the consequences strived for when their test is administered. With

this, they should consider as well how to prevent bias in interpretations of the results of

their tests. With respect to the development of dynamic assessment procedures, progress

can be made to result in explicit clues for practice by administering the processes

during learning phases carefully. There should be more emphasis on motivational and

metacognitive aspects. A qualitative analysis of the aspects deployed and the hints being

given will provide explicit guidelines for the significant others around the student being

tested. The focus upon these consequences in educational practice will contribute to an

improvement of the quality of tests, and should, therefore, have a prominent place in

test evaluation reports (COTAN).

This paves the way for the second level of recommendations. School psychologists

have the responsibility to communicate the outcomes to educational professionals

in the schools, such as (remedial) teachers. They should be able to make accurate

interpretations of the test scores, and more prominently, to translate these outcomes to

educational practice, in such way that instruction can be adapted to individual needs.

This leads us to a critical review of the role of the school psychologist in practice.

Currently, tasks and actions of school psychologists are rather focused on classification,

diagnosing, and decision making. The focus is not on providing specific guidelines for

practice, whereas professionals “IN” the schools urgently need this information. This

indicates a need for specific training in translating test results to guidelines during early

CHAPTER 7

152

career, as well as professional courses later on. Professional organisations at national

level (such as the NVO in The Netherlands) should take a leading role in this. Moreover,

school psychologist should be aware of their own beliefs about intelligence and learning

when administering and interpreting tests.

Next to school psychologists, all professionals in the schools (i.e., teachers, special care

coordinators, and even school administrators) should acquire a disposition focused

on the malleability of intelligence. During initial teacher training emphasis should

be placed upon the crucial role teachers play in the learning processes of students.

Teachers’ implicit theories of intelligence influence the amount of teaching in, and

exploration of, the Zone of Proximal Development of a student. This implies that the

quality of teaching a student receives depends on which teacher provides the instruction.

The instruction is influenced by implicit theories of intelligence. The third level of

recommendations, thus, addresses the educational professionals who are responsible for

adapting their instruction regularly to the individual needs of students. Skills need to

be developed of all these professionals. The case of Yamina can be helpful to illustrate

this statement. Since Yamina appeared to have a good potential to learn, and currently

does not show enough progress, it should be evident that instruction should be adapted.

It might be the case that the remedial teacher and the teacher respond differently to

this advice. It could be that, after a while, the teacher is not convinced about Yamina’s

learning potential anymore since her school results do not improve despite several

adaptations he made. It could be that the remedial teacher does not know how to

include metacognitive teaching during the sessions with Yamina. It might be that the

teacher knows how to do this, but is not sure about his own skills. If they both do not

acknowledge their own expertise in the learning processes of Yamina, and do not feel

responsible enough, then nothing will change in the instruction provided to Yamina. As

a result, her future learning outcomes will not progress, despite her potential to learn.

This dissertation underlined the urgent need for more emphasis on beliefs and

behaviour of educational professionals in the case of at-risk students. Only in this way

GENERAL DISCUSSION

153

7

the consequential validity of (dynamic) assessment will be warranted. Eventually this

will lead to bridging the gap between assessment and instruction, resulting in quality

instruction, especially to at-risk students in the general classroom.

References

156

Altheide, D. L., & Johnson, J. M. (2011). Reflections on interpretive adequacy in qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (4th ed., pp. 581-594). Los Angeles, CA: Sage.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of Educational Psychology, 84, 261-271. doi:10.1037/0022-0663.84.3.261

Antón, M. (2009). Dynamic assessment of advanced second language learners. Foreign Language Annals, 42, 576-598.

ATLAS.ti (Version 6.2) [Computer software]. Berlin: Scientific Software Development GmbH.

Azjen, I. (2005). Attitudes, personality and behavior (2nd ed.). Milton Keynes, England: Open University Press.

Bandura, A. (1993). Perceived self-efficacy in cognitive development and functioning. Educational Psychologist, 28, 117-148. doi:10.1207/s15326985ep2802_3

Bandura, A. (1997). Self-efficacy: The exercise of control. New York: Freeman.

Baudson, T. G., & Preckel, F. (2013). Teachers' implicit personality theories about the gifted: An experimental approach. School Psychology Quarterly, 28, 37-46. doi:10.1037/spq0000011

Bazeley, P. (2013). Qualitative data analysis: Practical strategies. London: Sage.

Beckmann, J. F. (2006). Superiority: Always and everywhere? On some misconceptions in the validation of dynamic testing. Educational and Child Psychology, 23(3), 35-49.

Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and intervention. Child Development, 78, 246-263. doi:10.1111/j.1467-8624.2007.00995.x

Boekaerts, M., & Minnaert, A. (2003). Measuring behavioral change processes during an ongoing innovation program: scope and limits. In E. De Corte, L. Verschaffel, N. Entwistle, & J. Van Merriënboer (Eds.), Powerful learning environments: Unravelling basic components and dimensions. (pp. 71- 87). Oxford, England: Pergamon/Elsevier Science.

Bol, L., Stephenson, P. L., O'Connell, A. A., & Nunnery, J. A. (1998). Influence of experience, grade level, and subject area on teachers' assessment practices. Journal of Educational Research, 91, 323- 330. doi:10.1080/00220679809597562

REFERENCES

157

Ref

eren

ces

Borkowski, J. G., Carothers, S. S., Howard, K., Schatz, J., & Farris, J. R. (2007). Intellectual assessment and intellectual disability. In J. W. Jacobson, J. A. Mulick, J. Rojahn, J. W. Jacobson, J. A. Mulick & J. Rojahn (Eds.), Handbook of intellectual and developmental disabilities. (pp. 261-277). New York: Springer Publishing Co. doi:10.1007/0-387-32931-5_14

Bosma, T., Hessels, M. G. P., & Resing, W. C. M. (2012). Teachers’ preferences for educational planning: Dynamic testing, teaching’ experience and teachers’ sense of efficacy. Teaching and Teacher Education, 28, 560-567. doi:10.1016/j.tate.2012.01.007

Bosma, T., & Resing, W. C. M. (2006). Dynamic assessment and a reversal task: a con tribution to needs- based assessment. Educational and Child Psychology, 23(3), 81-98.

Bosma, T., & Resing, W. C. M. (2008). Bridging the gap between diagnostic assessment and classroom practice. Journal of Cognitive Education and Psychology, 7, 174-198. doi:10.1891/194589508787381854

Büttner, G., & Hasselhorn, M. (2011). Learning disabilities: Debates on definitions, causes, subtypes, and responses. International Journal of Disability, Development and Education, 58, 75-87. doi:10.1080/1 034912X.2011.548476

Carlson, J. S., & Wiedl, K. H. (1992). Principles of dynamic assessment: The application of a specific model. Learning and Individual Differences, 4, 153-166. doi:10.1016/1041-6080(92)90011-3

Carmines, E. G., & Zeller R. A. (1979). Reliability and validity assessment. In J. L. Sullivan & R. G. Niemi (Series Ed.), Sage University Paper series on Quantitative Applications in the Social Sciences, 07-017. Beverly Hills, CA: Sage Publications.

Cavendish, W. (2013). Identification of learning disabilities: Implications of proposed DSM-5 criteria for school-based assessment. Journal of Learning Disabilities, 46, 52-57. doi:10.1177/0022219412464352

Chak, A. (2001). Adult sensitivity to children's learning in the zone of proximal development. Journal for the Theory of Social Behaviour, 31, 383-395. doi: 10.1111/1468-5914.00166

Compton, D. L., Fuchs, D., Fuchs, L. S., Bouton, B., Gilbert, J. K., Barquero, L. A., . . . Crouch, R. C. (2010). Selecting at-risk first-grade readers for early intervention: Eliminating false positives and exploring the promise of a two-stage gated screening process. Journal of Educational Psychology, 102, 327-340. doi:10.1037/a0018448

Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98-104. doi:10.1037//0021-9010.78.1.98

Daniel, M. H. (1997). Intelligence testing: Status and trends. American Psychologist, 52, 1038-1045. doi:10.1037/0003-066X.52.10.1038

158

Datnow, A., & Castellano, M. (2000). Teachers' responses to success for all: How beliefs, experiences, and adaptations shape implementation. American Educational Research Journal, 37, 775-799. doi:10.3102/00028312037003775

de Beer, M. (2010a). Longitudinal predictive validity of a learning potential test. Journal of Psychology in Africa, 20, 225-231.

de Beer, M. (2010b). A modern assessment psychometric approach to dynamic assessment. Journal of Psychology in Africa, 20, 241-246.

Delclos, V. R., Burns, M. S., & Kulewicz, S. J. (1987). Effects of dynamic assessment on teachers’ expectations of handicapped children. American Educational Research Journal, 24, 325-336. doi:10.3102/00028312024003325

Den Brok, P. J., Van Tartwijk, J. W. F., Wubbels, T., & Veldman, I. M. J. (2010). The differential effect of the teacher-student interpersonal relationship on student outcomes for students with different ethnic backgrounds. British Journal of Educational Psychology, 80, 199-221. doi:10.1348/000709909X465632

Deutsch, R., & Reynolds, Y. (2000). The use of dynamic assessment by educational psychologists in the UK. Educational Psychology in Practice, 16, 311-331. doi:10.1080/02667360020006381

Dina, F., & Efklides, A. (2009). Metacognitive Experiences as the Link between Situational Characteristics, Motivation, and Affect in Self-Regulated Learning. In M. Wosnitza, S. Karabenick, A. Efklides, & P. Nenniger (Eds.), Contemporary Motivation Research: from global to local perspectives (pp. 117-146). Cambridge, MA: Hogrefe & Huber.

Dinger, F. C., & Dickhäuser, O. (2013). Does implicit theory of intelligence cause achievement goals? Evidence from an experimental study. International Journal of Educational Research, 61, 38-47. doi:10.1016/j.ijer.2013.03.008

Dolores Calero, M., Belen, G. M., & Auxiliadora Robles, M. (2011). Learning potential in high IQ children: The contribution of dynamic assessment to the identification of gifted children. Learning and Individual Differences, 21, 176-181. doi:10.1016/j.lindif.2010.11.025

Dweck, C. S., Chiu, C., & Hong, Y. (1995). Implicit theories and their role in judgments and reactions: A word from two perspectives. Psychological Inquiry, 6, 267-285. doi:10.1207/s15327965pli0604_1

Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95, 256-273. doi:10.1037//0033-295X.95.2.256

Dymond, S. K., & Russell, D. L. (2004). Impact of grade and disability on the instructional context of inclusive classrooms. Education and Training in Developmental Disabilities, 39(2), 127-140.

REFERENCES

159

Ref

eren

ces

Elleman, A. M., Compton, D. L., Fuchs, D., Fuchs, L. S., & Bouton, B. (2011). Exploring dynamic assessment as a means of identifying children at risk of developing comprehension difficulties. Journal of Learning Disabilities, 44, 348-357. doi:10.1177/0022219411407865

Elliott, J. G. (2003). Dynamic assessment in educational settings: Realising potential. Educational Review, 55, 15-32. doi:10.1080/0013191022000037830

Espin, C., Wallace, T., Lembke, E., Campbell, H., & Long, J. D. (2010). Creating a progress-monitoring system in reading for middle-school students: tracking progress toward meeting high-stakes standards. Learning Disabilities Research & Practice, 25, 60-75. doi:10.1111/j.1540- 5826.2010.00304.x

Fabio, R. A. (2005). Dynamic assessment of intelligence is a better reply to adaptive behavior and cognitive plasticity. Journal of General Psychology, 132, 41-64.

Feuerstein, R., Rand, Y., & Hoffman, M. B. (1979). The dynamic assessment of retarded performers. The Learning Potential Assessment Device: Theory, instruments, and techniques. Jerusalem, Israel: ICELP Press.

Flyvbjerg, B. (2006). Five misunderstandings about case-study research. Qualitative Inquiry, 12, 219-245. doi:10.1177/1077800405284363

Francis, D., Fletcher, J., Stuebing, K., Lyon, G., Shaywitz, B., & Shaywitz, S. (2005). Psychometric approaches to the identification of LD: IQ and achievement scores are not sufficient. Journal of Learning Disabilities, 38, 98-108. doi:10.1177/00222194050380020101

Freeman, L., & Miller, A. (2001). Norm-referenced, criterion-referenced, and dynamic assessment: What exactly is the point? Educational Psychology in Practice, 17, 3-16. doi:10.1080/02667360120039942

Fuchs, D., Compton, D. L., Fuchs, L. S., Bouton, B., & Caffrey, E. (2011). The construct and predictive validity of a dynamic assessment of young children learning to read: Implications for RTI frameworks. Journal of Learning Disabilities, 44, 339-347. doi:10.1177/0022219411407864

Fuchs, L. S., Compton, D. L., Fuchs, D., Hollenbeck, K. N., Craddock, C. F., & Hamlett, C. L. (2008). Dynamic assessment of algebraic learning in predicting third graders' development of mathematical problem solving. Journal of Educational Psychology, 100, 829-850.

Fuchs, L. S., Compton, D. L., Fuchs, D., Hollenbeck, K. N., Hamlett, C. L., & Seethaler, P. M. (2011). Two- stage screening for math problem-solving difficulty using dynamic assessment of algebraic learning. Journal of Learning Disabilities, 44, 372-380. doi:10.1177/0022219411407867

Fuchs, D., & Fuchs, L. S. (2006). Introduction to response to intervention: What, why, and how valid is it? Reading Research Quarterly, 41, 93-99. doi:10.1598/RRQ.41.1.4

Fuchs, D., Fuchs, L. S., & Vaughn, S. (Eds.) (2008). Response to Intervention: A framework for reading educators. Newark, DE: International Reading Association

160

Fuchs, D., Mock, D., Morgan, P. L., & Young, C. L. (2003). Responsiveness-to-intervention: Definitions, evidence, and implications for the learning disabilities construct. Learning Disabilities Research & Practice, 18, 157-171. doi:10.1111/1540-5826.00072

Fuchs, L. S. , & Vaughn, S. (2012). Responsiveness-to-Intervention: A decade later. Journal of Learning Disabilities, 45, 195-203. doi:10.1177/0022219412442150

García-Cepero, M.C., & McCoach, D.B. (2009). Educators' implicit theories of intelligence and beliefs about the identification of gifted students. Universitas Psychologica, 8, 295-310.

Ghesquière, P. (2002). Perspectives in learning potential assessment: Conclusive remarks. In G. M. van der Aalsvoort, W. C. M. Resing, & A. J. J. M. Ruijssenaars (Eds.), Advances in cognition and educational practice Volume 7. Learning potential assessment and cognitive training (pp. 385-388). New York: Elsevier Science Ltd.

Gibbs, S., & Gardiner, M. (2008). The structure of primary and secondary teachers' attributions for pupils' misbehaviour: A preliminary cross-phase and cross-cultural investigation. Journal of Research in Special Educational Needs, 8, 68-77. doi:10.1111/j.1471-3802.2008.00104.x

Gibbs, S., & Powell, B. (2012). Teacher efficacy and pupil behaviour: The structure of teachers' individual and collective beliefs and their relationship with numbers of pupils excluded from school. British Journal of Educational Psychology, 82, 564-584. doi:10.1111/j.2044-8279.2011.02046.x

Gielen, S., Dochy, F., & Dierick, S. (2003). Evaluating the Consequential Validity of New Modes of Assessment: The Influence of Assessment on Learning, Including Pre-, Post-, and True Assessment Effects. In M. Segers, F. Dochy, & E. Cascallar (Eds.), Optimising New Modes of Assessment: In search of Qualities and Standards (pp. 37-54). Dordrecht, The Netherlands: Kluwer Academic Publishers.

Gillam, S. L., Fargo, J., Foley, B., & Olszewski, A. (2011). A nonverbal phoneme deletion task administered in a dynamic assessment format. Journal of Communication Disorders, 44, 236-245. doi:10.1016/j. jcomdis.2010.11.003

Gresham, F. M., & Witt, J. C. (1997). Utility of intelligence tests for treatment planning, classification, and placement decisions: Recent empirical findings and future directions. School Psychology Quarterly, 12, 249-267. doi:10.1037/h0088961

Grigorenko, E. L., & Sternberg, R. J. (1998). Dynamic Testing. Psychological Bulletin, 124, 75-111. doi:10.1037/0033-2909.124.1.75

Gummersall, D. M., & Strong, C. J. (1999). Assessment of complex sentence production in a narrative context. Language Speech and Hearing Services in Schools, 30, 152-164. doi:10.1044/0161- 1461.3002.152

REFERENCES

161

Ref

eren

ces

Hallahan, D. P., Kauffman, J. M., & Pullen, P. C. (Eds.) (2012). Exceptional Learners. An introduction to special education. Boston, MA: Pearson.

Hallahan, D. P., & Mock, D. R. (2003). A brief history of the field of learning disabilities. In H. Swanson, K. Harris, & S. Graham (Eds.), Handbook of Learning Disabilities (pp. 16-29). New York: The Guilford Press.

Hamers, J. H. M., Hessels, M. G. P., & Pennings, A. H. (1996). Learning potential in ethnic minority children. European Journal of Psychological Assessment, 12, 183-192.

Hattie, J. (2009). Visible Learning. A synthesis of over 800 meta-analyses relating to achievement. London: Routledge.

Haywood, H.C. (2012). Dynamic assessment: A history of fundamental ideas. Journal of Cognitive Education and Psychology, 11, 217-229. doi:10.1891%2F1945-8959.11.3.217

Haywood, H. C., Tzuriel, D., & Vaught, S. (1992). Psychoeducational assessment from a transactional perspective. In H. C. Haywood & D. Tzuriel (Eds.), Interactive as sessment. New York: Springer Verlag. Pp. 38-63.

Hessels, M.G.P. (1993). Leertest voor ethische minderheden: Theoretische en empirische verantwoording [Learning potential test for ethnic minorities: Theoretical and empirical foundation]. Rotterdam, The Netherlands: RISBO.

Hessels, M. G. P. (2000). The learning potential test for ethnic minorities (LEM): A tool for standardized assessment of children in kindergarten and the first years of primary school. In C. S. Lidz, & J. Elliott (Eds.), Dynamic Assessment: Prevailing Models and Applications (pp.109-131). New York: Elsevier.

Hessels, M. G. P., & Hessels-Schlatter, C. (2013). Current views on cognitive education: A critical discussion and future perspectives. Journal of Cognitive Education and Psychology, 12, 108-124. doi:10.1891/1945-8959.12.1.108

Hessels, M. G. P., & Tiekstra, M. (2010). Evaluation des capacités de raisonnement et prédiction des apprentissages dans un domaine scolaire nouveau chez des élèves présentant une déficience intellectuelle légère [Evaluation of reasoning capacity and prediction of learning in a novel school domain of students with mild mental retardation]. In M. G. P. Hessels, & C. Hessels- Schlatter, (Eds.), Evaluation et intervention auprès d'élèves en difficultés [Evaluation and intervention with students with difficulties] (pp. 51-66). Bern, Switzerland: Peter Lang.

Hessels-Schlatter, C., & Hessels, M. G. P. (2009). Clarifying some issues in dynamic assessment: Comments on Karpov and Tzuriel. Journal of Cognitive Education and Psychology, 8, 246-251. doi:10.1891/1945- 8959.8.3.246

Ikeda, M. J. (2012). Policy and practice considerations for Response to Intervention: Reflections and commentary. Journal of Learning Disabilities, 45, 274-277. doi:10.1177/0022219412442170

162

Imants, J. G. M., & De Brabander, C. J. (1996). Teachers’ and principals’ sense of efficacy in elementary schools. Teaching and Teacher Education, 12, 179-195. doi:10.1016/0742-051X(95)00053-M

Imants, J., Van der Aalsvoort, G. M., De Brabander, C. J., & Ruijssenaars, A. J. J. M. (2001). The role of the special services coordinator in Dutch primary schools: A counterproductive effect of inclusion policy. Educational Management Administration & Leadership, 29, 35-48. doi:10.1177/0263211x010291003

Jeltova, I., Birney, D., Fredine, N., Jarvin, L., Sternberg, R. J., & Grigorenko, E. L. (2011). Making instruction and assessment responsive to diverse students' progress: Group-administered dynamic assessment in teaching mathematics. Journal of Learning Disabilities, 44, 381-395. doi:10.1177/0022219411407868

Jennings, P. A., & Greenberg, M. T. (2009). The prosocial classroom: Teacher social and emotional competence in relation to student and classroom outcomes. Review of Educational Research, 79, 491-525. doi:10.3102/0034654308325693

Jitendra, A. K., & Kameenui, E. J. (1996). Experts' and novices error patterns in solving part-whole mathematical word problems. Journal of Educational Research, 90, 42-51.

Jones, R. E., & Ball, C. R. (2012). Introduction to the special issue: Addressing response to intervention implementation: Questions from the field. Psychology in the Schools, 49, 207-209. doi:10.1002/ pits.21590

Jordan, A., & Stanovich, P. (2001). Patterns of teacher–student interaction in inclusive elementary classrooms and correlates with student self-concept. International Journal of Disability, Development & Education, 48, 33-52. doi:10.1080/10349120120036297

Kanevsky, L., & Geake, J. (2004). Inside the zone of proximal development: Validating a multifactor model of learning potential with gifted students and their peers. Journal for the Education of the Gifted, 28, 182-217.

Katz, N., Golstand, S., Bar-Ilan, R. T., & Parush, S. (2007). The dynamic occupational therapy cognitive assessment for children (DOTCA-ch): A new instrument for assessing learning potential. American Journal of Occupational Therapy, 61, 41-52.

Kavale, K. A. (2005). Identifying specific learning disability: Is responsiveness to intervention the answer? Journal of Learning Disabilities, 38, 553-562. doi:10.1177/00222194050380061201

Kavale, K. A., & Spaulding, L. S. (2008). Is response to intervention good policy for specific learning disability? Learning Disabilities Research & Practice, 23, 169-179. doi:10.1111/j.1540- 5826.2008.00274.x

Kavale, K. A., Spaulding, L. S., & Beam, A. P. (2009). A time to define: Making the specific learning disability definition prescribe specific learning disability. Learning Disability Quarterly, 32, 39-48.

REFERENCES

163

Ref

eren

ces

Kerlinger, F. N., & Lee, H. B. (2000). Foundations of Behavioral Research (4th edition). New York: Harcourt Brace College Publishers.

Kirkwood, M. W., Weiler, M. D., Bernstein, J. H., Forbes, P. W., & Waber, D. P. (2001). Sources of poor performance on the rey-osterrieth complex figure test among children with learning difficulties: A dynamic assessment approach. Clinical Neuropsychologist, 15, 345-356.

Konstantopoulos, S., & Chung, V. (2011). The persistence of teacher effects in elementary grades. American Educational Research Journal, 48, 361-386. doi:10.3102/0002831210382888

Larsen, J. A., & Nippold, M. A. (2007). Morphological analysis in school-age children: Dynamic assessment of a word learning strategy. Language, Speech, and Hearing Services in Schools, 38, 201-212.

Lauchlan, F., & Elliott, J. (2001). The psychological assessment of learning potential. British Journal of Educational Psychology, 71, 647-665.

Lauermann, F., & Karabenick, S. A. (2011). Taking teacher responsibility into account(ability): Explicating its multiple components and theoretical status. Educational Psychologist, 46, 122-140. doi:10.1080/ 00461520.2011.558818

Lebeer, J., Birta-Székeley, N., Demeter, K, Bohács, K., Araujo Candeias, A., Sønnesyn, G., . . . Dawson, V. (2011) Re-assessing the current assessment practice of children with special education needs in Europe. School Psychology International, 33, 69-92. doi:10.1177/0143034311409975

Leroy, N., Bressoux, P., Sarrazin, P., & Trouilloud, D. (2007). Impact of teachers' implicit theories and perceived pressures on the establishment of an autonomy supportive climate. European Journal of Psychology of Education, 22, 529-545. doi:10.1007/BF03173470

Levin, J. R., & O’Donnell, A. M. (1999). What to do about educational research’s credibility gaps? Issues in Education, 5, 177-229. doi:10.1016/S1080-9724(00)00025-2

Lidz, C. S. (1995). Dynamic assessment and the legacy of L. S. Vygotsky. School Psychology International, 16, 143-153.

Lidz, C. S., & Macrine, S. L. (2001). An alternative approach to the identification of gifted culturally and linguistically diverse learners: The contribution of dynamic assessment. School Psychology International, 22, 74-96.

Lincoln, Y. S., & Guba, E. G. (1985). Establishing Trustworthiness. In Y. Lincoln & E. Guba (Eds.), Naturalistic Inquiry (pp. 289-331). Beverly Hills, CA: Sage.

Meijer, J. (2001). Learning potential and anxious tendency: Test anxiety as a bias factor in educational testing. Anxiety, Stress and Coping, 14, 337-362.

164

Meijer, J., & Elshout, J. J. (2001). The predictive and discriminant validity of the zone of proximal development. British Journal of Educational Psychology, 71, 93-113.

Meijer, W., & Tijhaar, M. (1995). Begeleidingsvaardigheden bij consultatieve leerlingbegeleiding [Support skills in teachers’ consultation on student behavior]. School en Begeleiding [School and Counseling], 13(1), 26-29.

Meirink, J. A., Meijer, P. C., Verloop, N., & Bergen, T. C. M. (2009). Understanding teacher learning in secondary education: The relations of teacher activities to changed beliefs about teaching and learning. Teaching and Teacher Education, 15, 89-100. doi:10.1016/j.tate.2008.07.003

Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5-11.

Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14(4), 5-8.

Minnaert, A. (2013). Goals are motivational researchers’ best friend, but to what extent are achievement goals and achievement goal orientations also the best friend of educational outcomes? International Journal of Educational Research, 61, 85-89. doi:10.1016/j.ijer.2013.08.002

Minnaert, A., & Vermunt, J. D. (2006). 25 jaar Onderwijspsychologie in Nederland en Vlaanderen in de periode 1980 tot 2005: Trends, pendels en grensverleggers [25 years of Educational psychology in The Netherlands and Flanders during 1980 till 2005]. Pedagogische Studiën, 83, 260-277.

Moorman, E. A., & Pomerantz, E. M. (2010). Ability mindsets influence the quality of mothers’ involvement in children’s learning: An experimental investigation. Developmental Psychology, 46, 1354-1362. doi:10.1037/a0020376

Nirmalakhandan, N. (2009). Use of computerized dynamic assessment to improve student achievement: Case study. Journal of Professional Issues in Engineering Education and Practice, 135, 75-80.

Opdenakker, M.-C. (2008). Vragenlijst over de leeromgeving – versie docenten [Questionnaire about learning environment – teacher version]. Groningen, The Netherlands: University of Groningen.

Opdenakker, M., & Van Damme, J. (2006). Differences between secondary schools: A study about school context, group composition, school practice, and school effects with special attention to public and catholic schools and types of schools. School Effectiveness and School Improvement, 17, 87-117. doi:10.1080/09243450500264457

Pajares, M. F. (1992). Teachers' beliefs and educational research: Cleaning up a messy construct. Review of Educational Research, 62, 307-32. doi:10.3102/00346543062003307

REFERENCES

165

Ref

eren

ces

Pakarinen, E., Kiuru, N., Lerkkanen, M., Poikkeus, A., Siekkinen, M., & Nurmi, J. (2010). Classroom organization and teacher stress predict learning motivation in kindergarten children. European Journal of Psychology of Education, 25, 281-300. doi:10.1007/s10212-010-0025-6

Pelletier, L. G., Seguin-Levesque, C., & Legault, L. (2002). Pressure from above and pressure from below as determinants of teachers' motivation and teaching behaviors. Journal of Educational Psychology, 94, 186-196. doi:10.1037//0022-0663.94.1.186

Peltenburg, M., van den Heuvel-Panhuizen, M., & Doig, B. (2009). Mathematical power of special-needs pupils: An ICT-based dynamic assessment format to reveal weak pupils' learning potential. British Journal of Educational Technology, 40, 273-284.

Peltenburg, M., van den Heuvel-Panhuizen, M., & Robitzsch, A. (2010). ICT-based dynamic assessment to reveal special education students' potential in mathematics. Research Papers in Education, 25, 319- 334. doi:10.1080/02671522.2010.498148

Perry, N., Turner, J.C., Meyer, D.K. (2006). Classrooms as contexts for motivated learning. In P. A. Alexander & P. H. Winne (Eds.), Handbook of Educational Psychology (2nd ed., pp. 327-348). Mahwah, NJ: Lawrence Erlbaum.

Prince, A., Minnaert, A., & Opdenakker, M.-C. (2014). Motivation in educational interventions: A systematic review exploring the concept in relation to intervention outcomes. In A. Prince, Motivation: How to tame the elephant in the classroom? (Unpublished doctoral dissertation). Groningen, The Netherlands: University of Groningen.

Prince, A., Tiekstra, M., & Minnaert, A. (2014). When theory meets practice becomes political: Can we combine scientific credibility and societal credibility? In A. Prince, Motivation: How to tame the elephant in the classroom? (Unpublished doctoral dissertation). Groningen, The Netherlands: University of Groningen.

Resing, W. C. M. (1990). Intelligentie en leerpotentieel. Een onderzoek naar het leerpotentieel van jonge leerlingen uit het basis- en speciaal onderwijs [Intelligence and learning potential. A study on the learning potential of young children from primary schools and special education]. Lisse, The Netherlands: Swets & Zeitlinger.

Resing, W. C. M. (2006). Using children’s ‘ability to learn’ in diagnosis and assessment. Educational & Child Psychology, 23(3), 6-11.

Resing, W. C. M., Tunteler, E., de Jong, F. M., & Bosma, T. (2009). Dynamic testing in indigenous and ethnic minority children. Learning and Individual Differences, 19, 445-450. doi:10.1016/j. lindif.2009.03.006

Reynolds, C. R., & Shaywitz, S. E. (2009). Response to intervention: Ready or not? Or, from wait-to-fail to watch-them-fail. School Psychology Quarterly, 24, 130-145. doi:10.1037/a0016158

166

Rosenfeld, M., & Rosenfeld, S. (2008). Developing effective teacher beliefs about learners: the role of sensitizing teachers to individual learning differences. Educational Psychology, 28, 245-272. doi:10.1080/01443410701528436

Rouse, M., & Agbenu, R. (1998). Assessment and special educational needs: teachers’ dilemmas. British Journal of Special Education, 25, 81-87. doi:10.1111/1467-8527.t01-1-00062

Rozendaal, J. S., Minnaert, A., & Boekaerts, M. (2005). The influence of teacher perceived administration of self-regulated learning on students’ motivation and information-processing. Learning and Instruction, 15, 141-160. doi:10.1016/j.learninstruc.2005.04.011

Ruijssenaars, A. J. J. M. (2001). Leerproblemen en leerstoornissen. Remedial teaching en behandeling. Hulpschema’s voor opleiding en praktijk [Learning difficulties and learning disabilities. Remedial teaching and treatment. Frameworks for education and practice]. Rotterdam, The Netherlands: Lemniscaat.

Runhaar, P, Sanders, K., & Yang, H. (2010). Stimulating teachers’ reflection and feedback asking: An interplay of self-efficacy, learning goal orientation, and transformational leadership. Teaching and Teacher Education, 26, 1154-1161. doi:10.1016/j.tate.2010.02.011

Ryba, K. (1998). Dynamic assessment and programme planning for students with intellectual disabilities. New Zealand Journal of Psychology, 27(2), 3-10.

Scanlon, D. (2013). Specific learning disability and its newest definition: Which is comprehensive? And which is insufficient? Journal of Learning Disabilities, 46, 26-33. doi:10.1177/0022219412464342

Schlatter, C., & Büchel, F. P. (2000). Detecting reasoning abilities of persons with moderate mental retardation: the analogical reasoning learning test (ARLT). In C. S. Lidz, & J. Elliott (Eds.), Dynamic Assessment: Prevailing Models and Applications (pp.155-186). New York: Elsevier.

Shamir, A., & Tzuriel, D. (2004). Children’s Mediational Teaching Style as a Function of Intervention for Cross-Age Peer-Mediation. School Psychology International, 25, 59-78.

Shapiro, E. S. (1987). Behavioral assessment in school psychology. Hillsdale, NJ: Lawrence Erlbaum.

Shapiro, E. S., & Kratochwill, T. R. (2000). Behavioral assessment in schools. Theory, research, and clinical foundations. New York: Guilford Press.

Skinner, E. A., & Belmont, M. J. (1993). Motivation in the classroom: Reciprocal effects of teacher behavior and student engagement. Journal of Educational Psychology, 85, 571-581. doi:10.1037/0022- 0663.85.4.571

Steenbeek, H., Jansen, L., & Van Geert, P. (2012). Scaffolding dynamics and the emergence of problematic learning trajectories. Learning and Individual Differences, 22, 64-75. doi:10.1016/j.lindif.2011.11.014

REFERENCES

167

Ref

eren

ces

Sternberg, R. J. (2004). Culture and intelligence. American Psychologist, 59, 325-338. doi:10.1037/0003- 066X.59.5.325

Sternberg, R. J., Birney, D., Jarvin, L., Kirlik, A., Stemler, S., & Grigorenko, E. L. (2006). From molehill to mountain: The process of scaling up educational interventions In M. A. Constas, & R. J. Sternberg (Eds.), Translating Educational Theory and Research into Practice: Developments in content domains, large-scale reform, and intellectual capacity. (pp. 205-222). Mahwah, NJ: Lawrence Erlbaum Associates.

Sternberg, R. J., Grigorenko, E. L., Ngorosho, D., Tantufuye, E., Mbise, A., Nokes, C., . . . Bundy, D. A. (2002). Assessing intellectual potential in rural Tanzanian school children. Intelligence, 30, 141- 162.

Stuebing, K. K., Barth, A. E., Molfese, P. J., Weiss, B., & Fletcher, J. M. (2009). IQ is not strongly related to response to reading instruction: A meta-analytic interpretation. Exceptional Children, 76, 31-53.

Sutherland, K. S., Lewis-Palmer, T., Stichter, J., & Morgan, P. L. (2008). Examining the influence of teacher behavior and classroom context on the behavioral and academic outcomes for students with emotional or behavioral disorders. The Journal of Special Education, 41, 223-233. doi:10.1177/0022466907310372

Swanborn, P. (2010). Case study research: What, why and how? London: SAGE

Swanson, H. L. (1995a). Effects of dynamic testing on the classification of learning-disabilities - the predictive and discriminant validity of the Swanson cognitive processing test (S-cpt). Journal of Psychoeducational Assessment, 13, 204-229.

Swanson, H. L. (1995b). Using the cognitive processing test to assess ability: Development of a dynamic assessment measure. School Psychology Review, 24, 672-693.

Swanson, H. L. (2011). Dynamic testing, working memory, and reading comprehension growth in children with reading disabilities. Journal of Learning Disabilities, 44, 358-371. doi:10.1177/0022219411407866

Swanson, H. L., & Howard, C. B. (2005). Children with reading disabilities: Does dynamic assessment help in the classification? Learning Disability Quarterly, 28, 17-34. doi:10.2307/4126971

Tannock, R. (2013). Rethinking ADHD and LD in DSM-5: Proposed changes in diagnostic criteria. Journal of Learning Disabilities, 46, 5-25. doi:10.1177/0022219412464341

Teo, P., & Roodenburg, J. (2013). Extensions of cognitive ability assessment with dynamic testing using self-regulated learning. Journal of Cognitive Education and Psychology, 12, 230-250. doi:10.1891 %2F1945-8959.12.2.230

Thorndike, E. L. (1924). An introduction of the theory of mental and social measurements. New York: Wiley.

168

Thurlow, M. L., & Ysseldyke, J. E. (1982). Instructional planning: Information collected by school psychologists versus information considered useful by teachers. Journal of School Psychology, 20, 3-10.

Tiekstra, M., Hessels, M. G. P., & Minnaert, A. E. M. G. (2009). Learning capacity in adolescents with mild intellectual disabilities. Psychological Reports, 105, 804-814. doi:10.2466/PR0.105.3.804-814

Tiekstra, M., & Minnaert A. (2015). At-risk students and the role of implicit theories of intelligence in educational professionals’ actions. Manuscript submitted for publication.

Tiekstra, M., Minnaert, A., & Hessels, M. G. P. (2016). A review scrutinising the consequential validity of dynamic assessment. Educational Psychology, 36, 112-137. doi:10.1080/01443410.2014.915930

Timperley, H. S., & Parr, J. M. (2009). What is this lesson about? Instructional processes and student understandings in writing classrooms. Curriculum Journal, 20, 43-60. doi:10.1080/09585170902763999

Tschannen-Moran, M., & Woolfolk Hoy, A. (2001). Teacher efficacy: Capturing an elusive construct. Teaching and Teacher Education, 17, 783-805. doi:10.1016/S0742-051X(01)00036-1

Tzuriel, D. (2000). The seria-think instrument: Development of a dynamic test for young children. School Psychology International, 21, 177-194.

Tzuriel, D., & Kaufman, R. (1999). Mediated learning and cognitive modifiability: Dynamic assessment of young Ethiopian immigrant children to Israel. Journal of Cross-Cultural Psychology, 30, 359-380. UNESCO (1994). The Salamanca statement and framework for action on special needs education. Paris: UNESCO.

UNESCO (2005). Guidelines for Inclusion: ensuring access to education for all. Paris: UNESCO.

Van de Ven, A. H., Polley, D., Garud, R., & Venkaraman, S. (1999). The innovation journey. New York: Oxford University Press.

Van der Aalsvoort, G. M., Resing, W. C. M., & Ruijssenaars, A. J. J. M. (Eds.). (2002). Advances in cognition and educational practice Volume 7. Learning potential assessment and cognitive training. New York: Elsevier Science Ltd.

Vaughn, S., & Denton, C. A. (2008). Tier 2: The role of intervention. In D. Fuchs, L. Fuchs, & S. Vaughn (Eds.), Response to Intervention: A framework for reading educators (pp. 51-70). Newark, DE: International Reading Association.

Vermunt, J. D. (1996). Metacognitive, cognitive and affective aspects of learning styles and strategies: A phenomenographic analysis. Higher Education, 31, 25-50.

REFERENCES

169

Ref

eren

ces

Viljoen, J. L., Odgers, C., Grisso, T., & Tillbrook, C. (2007). Teaching adolescents and adults about adjudicative proceedings: A comparison of pre- and post-teaching scores on the MacCAT-CA. Law and Human Behavior, 31, 419-432.

Vygotsky, L. S. (1978). Mind in society: The development of Higher Psychological Processes. In M. Cole, V. John- Steiner, S. Scribner, & E. Souberman (Eds.). Cambridge, MA: Harvard University Press.

Wiedl, K. H. (1980). Paradigmatische Orientierungen von Lehrern zum Problem psychischer Normalität and Störung: theoretische Explikation und empirische Analyse [Paradigm orientations of teachers about psychological normality and disorder: theoretical explication and empirical analysis]. Sonderpädagogik und Therapie, 286-295.

Wigfield, A., Byrnes, J. P., & Eccles, J. S. (2006). Development during early and middle adolescence. In P. A. Alexander & P. H. Winne (Eds.), Handbook of Educational Psychology (2nd ed., pp. 87-113). Mahwah, NJ: Lawrence Erlbaum.

Wilson, B. (1998). Matching Student Needs to Instruction. In S. A. Vogel & S. Reder (Eds.), Learning Disabilities, Literacy and Adult Education (pp. 191-212). Baltimore, MD: Brookes.

Woide, N., Beckmann, J. F., Elliott, J. G., & Guthke, J. (2005, July). Effects of feedback: individual differences in the ability to benefit from feedback. Paper presented at the 10th Conference of the International Association on Cognitive Education and Psychology, Durham, England.

Wolters, C. A., & Daugherty, S. G. (2007). Goal structures and teachers' sense of efficacy: Their relation and association to teaching experience and academic level. Journal of Educational Psychology, 99, 181-193. doi:10.1037/0022-0663.99.1.181

Yeager, D. S. , & Walton, G. M. (2011). Social-psychological interventions in education: They’re not magic. Review of Educational Research, 81, 267-301. doi:10.3102/0034654311405999

Ysseldyke, J. (2001). Reflections on a research career: Generalizations from 25 years of research on assessment and instructional decision making. Exceptional Children, 67, 295-309. doi:10.1177/001440290106700301

SamenvattingSummary in Dutch

NEDERLANDSE SAMENVATTING

173

Het leerpotentieel benutten van risicoleerlingen in de klasEen onderzoek naar de consequentiële validiteit van dynamische tests

In de reguliere onderwijspraktijk is het afnemen van tests gebruikelijk geworden wanneer

een leerling aanhoudende (leer-)problemen in de klas vertoont. Bij problemen in het

cognitieve gebied (dat wil zeggen, de leerling loopt wat betreft schoolresultaten voor of

achter op klasgenoten) wordt veelal uitgeweken naar het afnemen van een intelligentietest.

Deze testafnames zijn bedoeld om juiste diagnoses1 te stellen, met als uiteindelijk doel dat

er een passend onderwijsaanbod aan de leerling geboden kan worden. Echter, de vraag

blijft in hoeverre instructiegedrag van de leerkracht nou daadwerkelijk verandert na

resultaten van deze testafnames. Het blijkt voor leerkrachten moeilijk om de uitkomsten van

intelligentietests te vertalen in concrete veranderingen in hun onderwijs (Freeman & Miller,

2001; Ysseldyke, 2001). Kortom, er bestaat een kloof tussen het stellen van een diagnose

(assessment) en het aanpassen van leerkrachtgedrag op deze diagnose (Haywood, 2012;

Shapiro & Kratochwill, 2000).

De onderzoeken in deze dissertatie richten zich op deze kloof. Om uiteindelijk gerichte

interventies toe te kunnen passen die recht doen aan het leerpotentieel van de leerling, is

er onderzoek nodig dat inzicht geeft in de wijze waarop de uitkomsten van tests vertaald

worden in concrete acties. Met andere woorden, de consequenties van testafnames zouden

beter bestudeerd moeten worden (zie Messick, 1989, 1995). De problematiek wordt vanuit

een methodologisch perspectief benaderd: als we de kloof willen overbruggen, dan zal de

consequentiële validiteit van (dynamische) testafnames verhoogd moeten worden. Op deze

manier zal de instructie aan leerlingen bij wie het leren niet vanzelfsprekend is gebeuren op

een wijze die overdacht en gefundeerd is. Tests met consequentiële validiteit hebben immers

als doel het onderwijskundige en orthopedagogische handelen van docenten te verbeteren

om op deze wijze het leren bij leerlingen weer vlot te trekken of, in geval van hardnekkige

problematiek, verdere achterstand te voorkomen.

1 Met het stellen van een diagnose wordt bedoeld dat de problematiek van een leerling onderzocht wordt, en niet het etiketteren van (leer-)problemen van de leerling.

174

Het verhogen van de consequentiële validiteit van tests, en daarmee het overbruggen

van de kloof tussen assessment en instructie in de klas, vraagt om een tweezijdige

benadering: niet alleen de tests zijn hierin belangrijk, maar ook de context waarin de

tests afgenomen en geïnterpreteerd worden. In deze context spelen de zogenoemde

“significante anderen” een grote rol in het ontwikkelen van het potentieel van leerlingen

en zijn zij degene die testuitslagen interpreteren.

Op dit moment vindt de afname van tests meestal plaats op een geprotocolleerde

manier om aan criteria zoals betrouwbaarheid en validiteit te voldoen. Dit betekent bij

IQ tests dat de testafnemer wel af en toe een vraag mag stellen, maar geen additionele

uitleg mag geven tijdens de testafname. In het geval van risicoleerlingen kunnen hierbij

vraagtekens gezet worden, en een dynamische manier van testen is in dit soort gevallen

een beter alternatief. In een studie voorafgaand aan deze dissertatie is aangetoond dat

een dynamische test onderscheid kan maken tussen leerlingen die op een standaard

IQ test allen laag scoren (Tiekstra, Hessels, & Minnaert, 2009). Tijdens de dynamische

testafname vindt er een fase plaats waarin feedback en/of hints gegeven worden. Deze

“leerfase” lijkt een belangrijke rol te kunnen spelen bij het overbruggen van de kloof

tussen assessment en instructie, omdat deze inzicht verschaft in de processen die de

leerling aanwendt om taakjes op te lossen. Ondanks deze voordelen worden dynamische

tests nauwelijks in de dagelijkse praktijk gebruikt. Er dient onderzocht te worden hoe

we de kloof tussen assessment en instructie kunnen overbruggen en of dynamische tests

daarbij kunnen helpen.

Resultaten Omdat dynamische testprocedures gezien kunnen worden als een waardevol alternatief

voor de standaard IQ tests, richtte het eerste onderzoek (zie hoofdstuk 2) zich op deze

vorm van tests. Een literatuurstudie van 31 wetenschappelijke artikelen waarin in totaal

29 verschillende dynamische testprocedures werden beschreven, leverde een aantal

interessante bevindingen op. Na een grondige analyse kon vastgesteld worden dat

dynamische tests wellicht voldoen aan het criterium van validiteit voor consequenties


175

direct volgend op de testafname (immers, de aanwijzingen van de testafnemer

worden aangepast op de resultaten van de zojuist gemaakte test van de leerling), maar

richtlijnen voor instructie of interventie na de testafname worden nauwelijks gegeven;

de consequentiële validiteit die verder van de test af zit, wordt amper benut bij de

huidige tests. Daarom was het belangrijk om een grondige analyse te maken van de

leerfases die bij ieder instrument beschreven waren. Het bleek dat de testafnemer hints

en feedback geeft die voornamelijk op het cognitieve gebied, en in mindere mate ook op

metacognitief gebied, gericht zijn. Echter, motivationele aspecten worden nooit expliciet

genoemd in het afnemen van deze tests. Dat is opmerkelijk, omdat dynamische tests

zich ook richten op de “warme” kant van het leren. Kortom, ondanks het potentieel van

dynamische tests om de kloof tussen assessment en instructie te overbruggen, voegen ze

op dit gebied in de praktijk nog niets extra’s toe.

Tijdens een afname van dynamische tests wordt de mate waarin een kind didactisch

resistent is gemeten. Immers, er wordt nagegaan in hoeverre een kind reageert op de

kwaliteits- één-op-één instructie die geboden wordt tijdens de testafname. De vraag blijft

hoe men dit zou kunnen vertalen naar de praktijk in de reguliere klas. Het Response to

Instruction/Intervention model (beide termen worden gebruikt) lijkt een manier te zijn

om deze vertaling mogelijk te maken. Het kwalitatieve multipele methode onderzoek dat

in hoofdstuk drie beschreven wordt, richtte zich op de implementatie van het RTI-model

in de Nederlandse context. In het bijzonder werd er gekeken naar de mogelijkheid om

met dit model didactische resistentie te meten van leerlingen in de reguliere klas. De

resultaten wezen uit dat het een instrument kan zijn om didactische resistentie te meten,

mits het model zodanig geïmplementeerd wordt dat naast de resultaten van de leerling

ook de resultaten (geboden instructie) van de leerkracht in kaart gebracht worden. Dit

vraagt om een aanpassing in hoe men denkt over het leren (namelijk: de leerkracht

speelt een net zo grote rol in de mate van didactische resistentie) en niet alleen een

verandering in gedrag (d.w.z. het model uitvoeren).

176

De scheiding tussen gedrag en opvattingen van professionals die zo duidelijk uit

dit onderzoek naar voren kwam, leek ons belangrijk voor het verbeteren van de

consequentiële validiteit. In hoofdstuk 4 wordt er verder ingegaan op het verschil tussen

gedrag en opvattingen van onderwijsprofessionals. In een kwantitatieve studie werd de

invloed van impliciete intelligentietheorieën op het gedrag van leerkrachten in de klas

en onderwijsprofessionals buiten de klas bij risicoleerlingen onderzocht. Het bleek dat

deze impliciete intelligentietheorieën een grote rol spelen in de dagelijkse praktijk van het

onderwijs. Ongeveer 34% van de totale variantie in handelingen van de leerkrachten kon

voorspeld worden door impliciete theorieën over intelligentie, bij leerkrachten van christelijke

scholen was dit zelfs 61%. Dit betekent dat het onderwijzen in de Zone van de Naaste

Ontwikkeling beïnvloed wordt door ideeën van leerkrachten over de veranderbaarheid van

intelligentie. In de groep van onderwijsprofessionals buiten de klas (zoals intern begeleiders

en schoolpsychologen of orthopedagogen) bleek dat er een model uit de data afgeleid kon

worden: impliciete intelligentietheorieën voorspellen de waarde die wordt gehecht aan IQ-

tests, en dit voorspelt vervolgens de handelingen die worden gedaan naar aanleiding van

de testuitkomsten. In het kort gezegd, de consequentiële validiteit van tests wordt voorspeld

door de impliciete intelligentietheorie van de professional.

Omdat impliciete intelligentietheorieën, en hiermee het op waarde schatten van testresultaten,

een dusdanig grote rol bleken te spelen in de mate van consequentiële validiteit, wilden

we uitzoeken welke rol tests en testgegevens spelen in de dagelijkse praktijk. In hoofdstuk vijf

wordt gerapporteerd over twee kwalitatieve studies die gericht zijn op een beschrijving van

de praktijk. In de eerste in-vivo studie zijn de antwoorden van 36 schoolpsychologen, 21

intern begeleiders en 44 leerkrachten op open vragen geanalyseerd. De resultaten wezen

uit dat er geen eenduidig beeld bestond over het gebruik van testuitkomsten en de rol van

alle onderwijsprofessionals in de dagelijkse praktijk. Daarom is er in de tweede in-vivo studie

een casestudy uitgevoerd. Ook in deze studie werd de kloof tussen testafnames en instructie

bevestigd. Samenvattend laten deze studies zien dat er in de huidige onderwijspraktijk

behoefte is aan een andere soort van tests die (passende) richtlijnen voor het handelen in

de praktijk kunnen geven.


177

Alle bovengenoemde studies toonden aan dat onderwijsprofessionals meer kennis en

vaardigheden nodig hebben om het leerpotentieel van leerlingen te kunnen benutten in

de reguliere klas. De invloed van impliciete intelligentietheorieën moet hierbij zeker niet

vergeten worden. Als laatste studie hebben we een kortstondige interventie ontwikkeld

om te kijken of we deze vaardigheden en kennis kunnen verhogen en de impliciete

theorieën zouden kunnen beïnvloeden. De resultaten zijn beschreven in hoofdstuk zes.

Een pilotstudie onder een schoolteam (N=21) heeft meer inzicht kunnen verschaffen in

welke processen nodig zijn om de huidige onderwijspraktijk te veranderen en de kloof

tussen assessment en instructie te kunnen overbruggen. Het bleek dat de kortdurende

interventie een effect had op de mate waarin mensen geloven in de veranderbaarheid

van intelligentie: binnen het gehele schoolteam namen de “statische” opvattingen over

intelligentie af. Bovendien lieten groeicurvemodellen zien dat deze afname beïnvloed

wordt door de mate van het geloof in eigen handelen van leerkrachten (“self-efficacy”).

Hoe zekerder men is over het eigen handelen, des te sterker de afname in statische

opvattingen, zelfs op de lange termijn. Echter, hoewel de training zorgde voor een

verandering in het denken over intelligentie, kon de verandering in handelen niet

gemeten worden door restricties in de meetinstrumenten (plafondeffect). Dit aspect zou

dan ook in een vervolgstudie beter gemeten moeten worden. Concluderend kan men

stellen dat als we gedrag van leerkrachten willen veranderen, dat opvattingen en de

zekerheid over het eigen handelen meegenomen moeten worden in de interventies.

Implicaties en aanbevelingen Tests worden afgenomen met als doel het handelen van leerkrachten te veranderen.

Daarom kan ‘consequentiële validiteit’, waarin de validiteit van consequenties centraal

staat, gezien worden als ultieme vorm van validiteitseisen van tests. Helaas wordt deze

vorm van validiteit op dit moment niet meegenomen in oordelen over betrouwbaarheid

en validiteit van tests. De studies in deze dissertatie laten zien dat juist deze vorm van

validiteit de brug kan slaan tussen diagnose en interventie. De studies hebben laten zien

dat dynamische tests een groot potentieel hebben om meer consequentiële validiteit te

waarborgen dan de huidige statische tests.

178

Daarnaast onderschrijven de studies de behoefte aan een dynamisch perspectief op

leren van alle betrokkenen: van leraren tot schooldirecteuren, van orthopedagogen tot

beleidsmakers. In Nederland is recent het Passend Onderwijs beleid ingevoerd. Met dit

beleid is de zorgplicht voor zorgleerlingen verschoven naar de scholen. Dit betekent dat

onderwijsbetrokkenen op een juiste manier zouden moeten kunnen handelen in het geval

van risicoleerlingen. Als we willen dat zit zal slagen, zijn er veranderingen nodig.

Ten eerste moet er meer nadruk komen te liggen op de overtuigingen en het geloof in

eigen handelen (self-efficacy) van alle onderwijsprofessionals die met risicoleerlingen werken.

Meerdere studies in deze dissertatie hebben aangetoond dat opvattingen voor een groot

deel bepalen hoe professionals handelen, en grote gevolgen kunnen hebben voor het

optimaal benutten van het leerpotentieel van leerlingen. In initiële opleidingen van deze

professionals zou een dynamische opvatting over intelligentie en leren aangeleerd dan wel

ontplooid moeten worden, zodat er recht gedaan kan worden aan het (leer)potentieel van

alle leerlingen.

Ten tweede zou er in plaats van de huidige overwaardering voor (statische) intelligentietests

in de onderwijspraktijk meer aandacht besteed moeten worden aan alternatieven,

zoals dynamische tests. Ondanks het grote potentieel van dynamische tests voor de

onderwijspraktijk, blijven kennis en kunde van onderwijsprofessionals achter. Indien

onderwijsprofessionals de bovengenoemde dynamische opvatting over intelligentie

ontwikkelen, zal de implementatie van dynamische tests in de praktijk soepeler verlopen.

Ten derde zou consequentiële validiteit centraal moeten komen te staan in testvalidering.

Naast de aanbevelingen voor onderwijsprofessionals, hebben ook testontwikkelaars

de verantwoordelijkheid om stil te staan bij mogelijke consequenties die volgen uit de

testresultaten van hun test. Zij zouden zich moeten buigen over vraagstukken met betrekking

tot de interpretatie van hun testuitkomsten door onderwijsprofessionals en er zorg voor

moeten dragen dat dit op een valide manier gebeurt.

Als deze aanbevelingen in acht genomen worden, zal de huidige kloof tussen diagnose en


179

interventies overbrugd kunnen worden. Uiteraard vraagt dit tijd en aanpassing van alle

betrokkenen, maar zijn het kleine stappen die - op termijn - een groot verschil kunnen

maken in het professionele handelen van docenten en daarmee in de leerprocessen en

-resultaten van leerlingen.

About the author

188

About the author

Marlous Tiekstra (1985) was born in Delft. She attended primary school in Marum

(Gr.) and secondary school in Leek (Gr.), The Netherlands. In 2003 she started her

Bachelor Degree in Pedagogical Sciences at the University of Groningen and finished

in 2006. In 2004 she was board member of the students’ association of the Pedagogical

Sciences department and was chairwoman of the committee that organised a national

day for Pedagogical Sciences students in The Netherlands. She finished her Masters

studies in 2007, while spending nearly the full academic year at the University of

Geneva, Switzerland. In this year she acquired experience and conducted research in

the field of dynamic assessment and worked in l’Atelier d’Apprentissage, where she was

trained in metacognitive interventions and guided students with learning disabilities in

their learning processes. Beginning 2008, Marlous started her doctoral studies at both

universities mentioned before. The field of learning and learning processes is still of

great interest to her. Next to the researcher’s part of her PhD studies, she developed

and conducted different trainings and interventions for teachers and coaches in

primary education, and collaborated with several Dutch educational advisory agencies.

During her PhD Marlous was member and chairwoman of several PhD committees.

Furthermore, she was committee member of several international conferences, and

a scientific reviewer for the Journal of Cognitive Education and Psychology. Finally, in 2015

she completed her PhD studies. Marlous is currently working at the Dutch Ministry of

Education in The Hague.

ABOUT THE AUTHOR

189

Publications

Tiekstra, M., Hessels, M. G. P., & Minnaert, A. E. M. G. (2009). Learning capacity in adolescents with mild intellectual disabilities. Psychological Reports, 105, 804-814. doi:10.2466/PR0.105.3.804-814

Hessels, M. G. P., & Tiekstra, M. (2010). Evaluation des capacités de raisonnement et prédiction des apprentissages dans un domaine scolaire nouveau chez des élèves présentant une déficience intellectuelle légère [Evaluation of reasoning capacity and prediction of learning in a novel school domain of students with mild mental retardation]. In M. G. P. Hessels, & C. Hessels- Schlatter, (Eds.), Evaluation et intervention auprès d'élèves en difficultés [Evaluation and intervention with students with difficulties] (pp. 51-66). Bern, Switzerland: Peter Lang.

Schölvink, M., Jansen, L., Van der Aalsvoort,D., Ahlers, L., & Tiekstra, M. (2011). Scenario "Responsiveness To Intervention". In M. Schölvinck, L. Jansen, & A. Minnaert (Eds.), Passend onderwijs, anders beschikken. Budgetverdeling op basis van goed onderwijs en effectief leerkrachtgedrag [Adaptive Education, rearrangement of special education. Budgeting based on quality teaching and effective teacher behavior] (pp. 89-108). Amersfoort, The Netherlands: CPS.

Prince, A., Tiekstra, M., & Minnaert, A. (2014). When theory meets practice becomes political: Can we combine scientific credibility and societal credibility? In A. Prince, Motivation: How to tame the elephant in the classroom? (Unpublished doctoral dissertation). Groningen, The Netherlands: University of Groningen.

Tiekstra, M., Minnaert, A., & Hessels, M. G. P. (2016). A review scrutinising the consequential validity of dynamic assessment. Educational Psychology, 36, 112-137. doi:10.1080/01443410.2014.915930

Tiekstra, M., & Minnaert A. (under review). At-risk students and the role of implicit theories of intelligence in educational professionals’ actions. Manuscript submitted for publication.

Tiekstra, M., Prince, A., & Minnaert, A. (in prep.). Can didactic resistance be measured by using the Response-to- Instruction model? A pilot study.

Tiekstra, M., Minnaert, A., Hessels, M., & Verdel, R. (in prep.). The next step in bridging the gap: a pilot study to enhance teacher’s skills with at-risk children.

ABOUT THE AUTHOR

university of groningen fostering the learning potential ... · contents chapter 1 chapter 2...

Documents