teacher beliefs and teacher behaviour in portfolio assessment

ARTICLE IN PRESS

0742-051X/$ - s

doi:10.1016/j.ta

�CorrespondE-mail addr

(M.F. van der

Teaching and Teacher Education 24 (2008) 1691–1704

www.elsevier.com/locate/tate

Teacher beliefs and teacher behaviour in portfolio assessment

Marieke F. van der Schaafa,�, Karel M. Stokkinga, Nico Verloopb

aDepartment of Educational Sciences, Utrecht University, PO Box 80140, 3508 TC Utrecht, The NetherlandsbLeiden University, ICLON Graduate School of Education, The Netherlands

Received 20 July 2007; received in revised form 14 February 2008; accepted 26 February 2008

Abstract

This study concerns the relation between teachers’ beliefs towards teaching behaviour and their actual teaching

behaviour in teacher portfolio assessment. We analysed the beliefs and behaviour of 18 teachers as described in their

portfolios. In addition, each portfolio was independently assessed by two trained raters on eight content standards and the

teachers’ classroom behaviour was assessed by their own students in a questionnaire (n ¼ 317). Linear multilevel analysis

showed that part of the raters’ assessments of the teachers’ beliefs and their behaviour as described in their portfolios was

significantly related to the students’ assessments of their teachers’ behaviour. Teachers with high raters’ assessments on the

content standard about ‘choosing and arguing for teaching strategies that meet students’ knowledge, abilities and

experience’ had significantly higher student assessments than teachers who were judged low on this standard. Implications

of the results and suggestions for further research are discussed.

r 2008 Elsevier Ltd. All rights reserved.

Keywords: Teacher beliefs; Teacher behaviour; Portfolio; Assessment

1. Introduction

Many countries face a growing interest in theassessment of teachers (Cochran-Smith & Fries,2001). Quality teaching is now recognized to be aresult of complex interactions between teachers’beliefs towards teaching behaviour and their actualteaching behaviour in a certain context andteachers’ thoughtful examination of these interac-tions. It is generally assumed that only by becomingaware of their beliefs, teachers can further developtheir repertoire (Borko & Putnam, 1996; Cochran-Smith & Lytle, 1999; Shulman, 1996). Saroyan and

ee front matter r 2008 Elsevier Ltd. All rights reserved

te.2008.02.021

ing author.

ess: [email protected]

Schaaf).

Amundsen (2001) even argue that the most compe-tent teachers conscientiously try to align their beliefstowards their actual teaching behaviour with theirbehaviour in an attempt to attain specific instruc-tional goals.

In this study we focus on teachers’ beliefs towardsteaching students research skills and their behaviourwhile actually teaching their students such skills inupper secondary social science education. This topicis representative of the change in many countriestowards more constructivist views on learning andtowards developing students’ skills in an active, self-regulated and collaborative way.

Since teachers’ beliefs towards teaching behaviourappear to be crucial for the improvement of theirteaching behaviour, it is nowadays assumed thatteacher assessment should take account of teacher

.

www.elsevier.com/locate/tate

dx.doi.org/10.1016/j.tate.2008.02.021

mailto:[email protected]

mailto:[email protected]

ARTICLE IN PRESSM.F. van der Schaaf et al. / Teaching and Teacher Education 24 (2008) 1691–17041692

beliefs in relation to their behaviour (Beijaard &third author, 1996). Accordingly, teacher assess-ment is now often based on instruments focusingon an inclusive view of teaching, for exampleportfolios (Andrews & Barnes, 1990; Delandshere,1994). Depending on the content and form ofthe portfolio and on the integration of the portfoliowith a teacher’s working context, a portfolio maydo justice to the fact that teaching is a complexactivity and that teacher behaviour is inextricablybound up with teacher cognition and the teach-ing context (Bird, 1990; Lyons, 1998). Teacherscan explicate their beliefs in a portfolio by forexample self-reflections. Portfolio assessments canbe used summatively, e.g. as a tool to ascertainwhether teachers satisfy required standards, andformatively, to formulate guidelines for professionaldevelopment.

Although the existence of a connection betweenteachers’ beliefs towards teaching behaviour andtheir behaviour has been well established, the role ofteacher beliefs in teacher portfolio assessment is farfrom clear. This is due to the fact that teacher beliefsin a portfolio are based on self-reflections by theteacher and are context dependent and are thereforehard to assess accurately.

Research into the correspondence between tea-chers’ beliefs towards their behaviour and beha-viour in teacher portfolio assessment is important todeepen our understanding of this correspondence,to increase our knowledge about what can andcannot be assessed by teacher portfolios, and toimprove the validity of portfolio assessments.Therefore, in this paper we try to give answers tothe question: What is the correspondence between

teachers’ beliefs towards teaching students research

skills and their behaviour while actually teaching their

students such skills as constructed by teachers in their

portfolios and as assessed by their students and by

external raters?

We researched into this question by comparingplain portfolio data (including self descriptionsand video registrations of teachers’ lessons)with assessments by students and raters. Thestudy is part of a larger research project onthe assessment of experienced Dutch teachers’beliefs and behaviour and builds upon earlierstudies in which we developed standards(both content standards and performance stan-dards) for teaching students research skills in uppersecondary social science education (First author,2005).

2. The correspondence between teachers’ beliefs

towards teaching behaviour and their actual teaching

behaviour in teacher assessment

2.1. Teacher beliefs and teacher behaviour

Teacher beliefs are important mediators ofteacher behaviour (Calderhead, 1996). However,the relation between teacher beliefs and teacherbehaviour is far from clear, as teacher beliefs aremessy constructs with different interpretations andmeanings (Pajares, 1992). We nevertheless foundsome convergence of ideas about the nature ofteacher beliefs in the literature.

Firstly, current definitions of teacher beliefs focuson teachers’ assumptions which affect what theynotice in any set of circumstances and what theyregard as possible, the goals they will set, and theknowledge they will bring into those circumstances(Clark & Peterson, 1986; Kagan, 1992; Pajares,1992). Since teacher beliefs shape the way teachersperceive and interpret classroom interaction andinfluence their construction of intentions in responseto those interactions, we define teacher beliefs as‘‘an integrated system of personalized assumptionsabout the nature of a subject, its teaching andlearning’’ (Artzt & Armour-Thomas, 1998, p. 8). Inour study we focus on beliefs towards teachingstudents research skills.

Secondly, it is generally assumed that teacherbeliefs differ in specificity and in strength dependingon the context (as perceived), that they tend to beactivated in clusters, and that incompatible beliefsmay contend for priority (Aguirre & Speer, 2000;Ajzen, 2002; Schoenfeld, 1998). This implies thatnot all teachers’ beliefs will play a role in their actualbehaviour. Only the most salient beliefs willinfluence the execution of teaching tasks.

Thirdly, beliefs may guide teacher behavioureither deliberately or spontaneously. In a deliberateway beliefs are retrieved or constructed with a lot ofeffort in a certain context and they are assumed toguide goal setting and behaviour. In a spontaneousway beliefs are activated by routine. Teacher goalscan be conceived as ‘‘cognitive constructs thatdescribe at various levels of detail what the teacherswant to accomplish’’ (Aguirre & Speer, 2000, p.332). Goals can be conceived of as long-range goalsfor student learning (Clark & Peterson, 1986) and asshort-term mental structures that arise in interac-tion with events in the classroom (Saxe, 1991). Forthe purpose of this study we designate the former as

ARTICLE IN PRESSM.F. van der Schaaf et al. / Teaching and Teacher Education 24 (2008) 1691–1704 1693

goals and the latter as intentions to engage incertain behaviour. Fazio (1990) argues that underconditions of high motivation and sufficient cogni-tive ability, people can put effort in constructingtheir beliefs related to certain goals and intentions.Without those two conditions beliefs related to agoal or intention are only routinely activated.Routine activation of such beliefs only takes placewhen the goal or intention raises a strongly tiedattitude toward (or evaluation of) that goal orintention.

2.2. Constructed teacher beliefs and behaviour

As the nature of teacher beliefs hardly enablesaccurate assessment, it is not possible to showdirectly the correspondence between teachers’ be-liefs and their actual behaviour. The clusters ofbeliefs that guide teacher behaviour are context-related and exist predominantly as tacit knowledgethat cannot be easily articulated. Implicit beliefs canonly be studied if they are first made explicit and weassume that teacher beliefs as described in theirportfolios are constructed representations of theirbeliefs, that is, the beliefs that teachers claim theyhave.

This article is about beliefs towards teachingbehaviour and we separate teacher beliefs beforeteaching and during the interactive phase ofteaching (resulting in intentions to certain beha-viour) as explicated in the portfolios in teachers’self-reflections. To interpret these teacher beliefs weused the Theory of Planned Behaviour (Ajzen, 2002;Ajzen & Fishbein, 2005). This theory has proved itsusefulness in research into the relation betweenteacher cognitions and behaviour in teaching(social) sciences and in educational change (e.g.Crawley, 1990; Haney, Czerniak, & Lumpe, 1996).

Behavioural beliefs

Subjective norms Inten

Control beliefs

Fig. 1. Relations between beliefs, intentions, a

The theory is based on three clusters of cognitionsthat cause variance in people’s goals and intentionsto engage in certain behaviour (Fishbein et al., 2001)(see Fig. 1). These clusters are (1) the expectedeffects of particular behaviour (behavioural beliefs);(2) the expected support by significant others, e.g.colleagues (subjective norms); (3) the expectedfactors that may ease or complicate the performanceof behaviour, e.g. perceived facilitation for teachingstudents certain skills (control beliefs). On thewhole, the evaluation of the clusters is assumed toproduce overall positive or negative goals andintentions to perform particular behaviour. Forexample, teachers who think that they have theskills and the support needed to teach studentsresearch skills may develop a strong sense of self-efficacy, whereas teachers who assume that theylack some of the resources and support needed arelikely to have much weaker control beliefs and as aresult may have weaker intentions in teachingstudents these skills.

2.3. Assessed teacher beliefs and behaviour

Raters use schemata to judge, foresee andcomprehend the beliefs that teachers construct.These schemata are comparable to personal con-structs (Kelly, 1955), content categories used toorganize and simplify information. Interpersonalfilters in these schemata induce raters to lookselectively at the information about the teachersand to interpret the information according to theirown constructs. As a result, raters’ representationsabout teachers’ beliefs are not necessarily consistentwith the ones that teachers hold themselves.

On the other hand, raters’ mental representationsare shaped by their prior knowledge and experience,e.g. knowledge of the working context of the

tions Behaviour

nd behaviour (Ajzen & Fishbein, 2005).


teachers assessed (when they are internal raters,working in the same educational organization as theteachers), own teaching experience and experiencein scoring portfolios. Rater judgment can beinfluenced by training procedures (Lievens, 2001;Woehr & Huffcutt, 1994). Depending on the formand content of the training, discrepancies betweenraters’ representations and teachers’ constructedbeliefs may decrease.

Irrespective of the amount of discrepancybetween raters’ representations and teachers’ con-structions, the raters’ representations are worth-while in themselves as they enable them to explainthe meaning of teachers’ behaviour. When arater says that a teacher has a certain belief, thismeans that the teacher is behaving in a mannerconsistent with having such a belief (according tothe rater). Therefore, we distinguish between(teacher) constructed and (rater) assessed teacherbeliefs.

2.4. Content standards

In the Dutch upper secondary education examrequirements doing research (as a skill to bedeveloped by students) is seen as consisting of aseries of activities or steps, namely: (1) identify andformulate a problem using subject-specific concepts;(2) formulate the research question(s); (3) make andmonitor the research plan; (4) gather and selectdata; (5) assess the value and utility of the data; (6)analyse the data; (7) draw conclusions; (8) evaluatethe research; (9) develop and substantiate a personalpoint of view; and (10) report and present theresearch (Second author, first author, Jaspers, &Erkens, 2004).

In teaching students research skills, teachers willhave to decide how much they structure thestudents’ learning environment through assignmentsand guidance and how much they leave to thestudents themselves. Students in Dutch uppersecondary education have little experience in doingresearch and in thinking scientifically. A verystructured learning environment in which thestudents have no space for making choices is notvery obvious and the same goes for a very openlearning environment. Students learn best by‘guided’ reinvention (not too structured, not tooopen), because both extremes lead to superficiallearning (Roth & Roychoudhury, 1993).

To teach students research skills, teachers have tofulfil a number of professional tasks: pre-active

tasks (tasks before teaching, e.g. goal setting),interactive tasks (tasks during teaching, e.g.instructing, coaching), and post-active tasks(tasks after teaching, e.g. assessing and reflecting)(Reynolds, 1992). In earlier studies into these tasks,a literature study and a Delphi study with 21 experts(First author, 2005), we successfully developedeight content standards and these standardswere used in this study. Most of these standardsinclude several indicators (sub-standards), describ-ing what teachers should know and be able to do inteaching students’ research skills according to thethree teaching phases (pre-active, interactive, post-active).

Pre-active

1.
Selecting goals for students (GOAL): (a) ateacher’s long-term goals for developing stu-dents’ research skills; (b) the approach used bythe teacher to reach these goals.
2.
Choosing an appropriate assignment (ASSIGN):(a) the goals are related to subtasks of doingresearch; (b) the content is related to doingresearch in the subject; (c) the form is authenticand the assignment is approachable in variousways; (d) the assignment fits the students’ level.Interactive
3.
Preparing and managing students to work ontheir research assignments (MANAGE): (a) thepreparation and the communication; (b) manage-ment of facilities (e.g. time, rooms, sources,media) that students need for working on theassignment.
4.
Choosing and arguing for teaching strategies(THINK) that (a) meet students’ knowledge,abilities and experience and fit: (b) teachers’goals; (c) the assignment; and (d) the assessment.
5.
Teaching students research skills (TEACH): (a)correct use of the teaching strategies chosen. Theinstruction and guidance fit: (b) the concerningsubtasks of doing research; (c) the operationalstage in the research assignment; and (d)students’ level.
6.
Creating a positive pedagogical climate (CLI-MATE): (a) the provision of a safe, respectful,and stimulating learning environment for stu-dents.Post-active
7.
Adequately assessing students’ research skills(ASSESS): (a) a teacher’s argumentation forand the clarity and comprehensibility of theassessment approach used (content standards,


scoring, and norms); (b) the applicability of theapproach to the assignment; (c) the way theteacher handles the assessment and (d) commu-nicates the results to the students.

8.
Reflecting on the programme and on actions inteaching students research skills (REFLECT):the extent to which the teacher is aware of strongand weak points in (a) his or her teaching and (b)the educational program; his or her suggestionsfor improvement of (c) his or her teaching and(d) the educational program.
3. Method

3.1. Participants

3.1.1. Sample of teachers

Within a random sample of 115 upper secondaryschools in the Netherlands (20% of the population),we approached the heads of the Economics,Geography, and History departments with informa-tion about the study and a request for participation.Twenty-one teachers from 21 schools were willing toparticipate. This low response can be explained bythe demanding character of the study in terms of therequired motivation, time, and experience in teach-ing students research skills. Three teachers partici-pated in a pilot study. Eighteen teachersparticipated in the main study on which we reportin this paper (9 History, 6 Economics, 3 Geogra-phy). The average age of the 18 teachers was 44years and they had an average of 15 years teachingexperience.

3.1.2. Sample of students

Each participating teacher randomly selected aclassroom of which the students rated their teacherin a questionnaire. In total 317 students wereinvolved. The number of students varied perteacher, depending on how many students hadchosen the teacher’s subject for examination.

3.1.3. Sample of raters

Since having a teaching background themselveshelps raters assess teachers’ constructed beliefs andbehaviour (Pula & Huot, 1993), we selected raterswith teaching experience. We chose external raters(not working in the same school) to avoid biasing ofperceptions or judgments due to episodic memories.The raters also participated in earlier studies inwhich content standards, performance standards,

and a portfolio assessment procedure for teachingstudents research skills were developed. All raterswere teachers in social science subjects. Three ofthem were also teacher trainers and one of them wasalso a school principal. All raters received financialcompensation. None of the raters had assessedteacher portfolios before. All raters supported thestandards and the assessment procedure used in thisstudy.

3.2. Teacher portfolio evidence

The assessment procedure has been developedand empirically tested in an earlier study (Firstauthor, 2005). In their portfolios the teacherscollected material about teaching students researchskills in the pre-, inter-, and post-active phases ofthe teaching process. The portfolios contained sevenelements (see Fig. 2), including a number ofproductions (documents especially prepared forthe portfolio), one reproduction (a registration ofa teacher’s regular work captured on behalf of theportfolio), and several artefacts (regular products ofteacher’s work). The content of the elementsfocused on parts of the Theory of PlannedBehaviour: background factors (e.g. years of teach-ing experience, subject, class, and courses followedin teaching students research skills), teachers’ beliefsin teaching students research skills (e.g. their goalsfor engaging students in research), subjective norms(e.g. arrangements with colleagues), control beliefs(e.g. perceived facilities for student research, per-ceived difficulties students have in doing research),and behaviour (e.g. the way teachers instruct andcoach their students). The eight content standardscould be assessed using one or more portfolioelements (see Fig. 3).

3.3. Rater training

The raters were thoroughly trained in assessingteacher portfolios. They studied a manual describ-ing in detail the objectives, planning, and proce-dures of the assessment, the content standards, theanchor points, and the portfolio materials. Theraters first individually studied an example portfo-lio. They then participated in a training session (4 hplenary) in which they used judgement forms. Afterthe training, the raters individually rated threeportfolios in a pilot (one per subject), intended asa try-out of the rating procedure and to give theraters feedback in order to improve their judgments.

ARTICLE IN PRESS

Element Form Content

1. Self description

2. Series of assignments and clarification

3. Research assignment and clarification

production

artefact and production

artefact and production

Teacher’s background, context, behavioural beliefs

Teaching behaviour, control beliefs, subjective norms

Teaching behaviour, teaching intentions, control beliefs

4a. Interview before lesson

5. Videos

4b. Interview after lesson

production

reproduction

production

Control beliefs, subjective norms

Teaching behaviour

Control beliefs

6. Assessment

7. Reflections

artefact

production

Teaching behaviour

Control beliefs, subjective norms

Fig. 2. Elements, forms, and contents of the teacher portfolio.

M.F. van der Schaaf et al. / Teaching and Teacher Education 24 (2008) 1691–17041696

After the pilot, the raters received feedback abouttheir scoring and suggestions for improving theirjudgments. Suggestions included the accurate use ofthe standards, the assessment procedure, and pointsof special interest concerning the interpretation.

3.4. Data collection

3.4.1. Constructed teacher beliefs and behaviour

Since the 18 teachers in our study did not haveany experience in preparing a portfolio we struc-tured the content, while teachers were free in theform of the material and also free to add othermaterial (Shapley & Bush, 1999). The teachersreceived instructions including a description of theaims of the study, the content and performancestandards, and the assessment procedure. Theteachers recorded two periods of instruction andcoaching of a maximum of 30min each. Teacherswere provided help in developing their portfoliowhen needed, for instance by typing spoken texts(Freidus, 1998; Seldin, 1997). Each of the 18portfolios was constructed within a time span of afew months. All teachers confirmed that theirportfolios represented their beliefs and behaviour,so we suppose their portfolios are sufficiently

authentic. Afterwards, the teachers received writtenfeedback concerning their weaknesses and strengthsin view of the content standards and performancestandards and the comments made by the raters.The teachers filled out a questionnaire with ques-tions on a five-point scale (1 ¼ I strongly disagree;5 ¼ I strongly agree) to evaluate the relevance of thefeedback given based on raters’ assessments.

3.4.2. Assessed teacher beliefs and behaviour

Based on the content standards we developed aquestionnaire for students containing 15 statementson how clearly and supportively their teacher isinstructing and coaching research skills (see Appen-dix A). The students answered on a four-point scalefrom 1 ‘not at all like my teacher’ to 4 ‘very muchlike my teacher’. We successfully tested the ques-tionnaire in a pilot study with 45 students of threeteachers (Cronbach’s alpha .84).

The 18 portfolios were rated, each one indepen-dently by two raters. We organized the rater pairsaccording to their major expertise. In a 9-monthperiod, we assigned the 18 portfolios in the orderthey became available. The raters used judgmentforms to rate the portfolios. Firstly, raters illu-strated each content standard with significant

ARTICLE IN PRESS

Content standards

Portfolio elements

GOAL ASSIGN MAN. THINK TEACH CLIM. ASSESS REFL

1. Self-description X X

2. Series assignments X

3. Assignment X X X X X

4. Interviews X X X X X

5. Videos X X X X

6. Assessment X

7. Reflection X

Fig. 3. Portfolio elements and content standards.

M.F. van der Schaaf et al. / Teaching and Teacher Education 24 (2008) 1691–1704 1697

portfolio material. Secondly, they described theirinterpretations of the material. Finally, ratersassigned a score on the portfolio for each contentstandard, using a three-point scale with anchorpoints (analytic score) (for an example, see Appen-dix B). They added an overall (holistic) judgment ona three-point scale, considering the total of theevidence. The rating process produced two types ofscores: (unweighted) mean analytic scores (themeans of the scores for the eight content standards)and holistic scores (the overall judgments of theteachers’ portfolios).

3.5. Data analysis

3.5.1. Constructed teacher beliefs and behaviour

Data concerning background factors, teachers’goals in teaching students research skills, theirsubjective norms, and their control beliefs wereanalysed descriptively (frequencies, means).

The two videotapes per teacher were fullytranscribed and the teachers’ interactive behaviourwas coded. The first author and an assistantresearcher analysed the videos in terms of thecontent (the ten research steps), the structurednessof the learning environment, the setting (classroom,

multimedia centre), and the students’ grade. Wesplit every video in segments per research step. Persegment we counted the subskills taught andcomputed frequencies of the interaction betweenteacher and students (teacher central, interaction,students central).

As to the degree of structuredness of the learningenvironment here we focus only on the teacher’sbehaviour and not on the research assignment orother tools that will also structure students’ learningenvironment. Ratings of the learning environmentwere done on a scale from 1 (very structured)to 3 (very low structured), based on a detaileddescription of teacher behaviour related to thestructure of a learning environment (Vermunt &third author, 1999). High scores were given whenteachers encouraged students and gave themopportunities to work autonomously (e.g. by offer-ing students choices and refraining from givingunnecessary help). Low scores were given whenthe teacher controlled all aspects of researchactivities.

3.5.2. Assessed teacher beliefs and behaviour

We analysed the scalability of the contentstandards in the questionnaire by computing


Cronbach’s alpha. We calculated frequencies andchecked for differences between teachers using one-way ANOVA on their students’ ratings of theirclassroom behaviour.

We used the percentage exact agreement inholistic scoring and the Cronbach’s alpha coefficient(jury alpha) in analytic scoring (on the contentstandards) per portfolio within the rater pairs toestimate the inter-rater reliability.

3.5.3. The correspondence between teacher beliefs

and behaviour as constructed and as assessed

The data from the teacher questionnaire toevaluate the relevance of the feedback given, basedon raters’ assessments, were analysed descriptively.Since we had nested data (students within teachers),to analyse the correspondence between teacherbeliefs and behaviour as constructed by teachers intheir portfolios and as assessed by their students andexternal raters we used linear multilevel analysis.We used the following (sets of) predictors: (1) theteachers’ beliefs concerning the goals of teachingstudents research skills; (2) the teachers’ behaviour(research steps, teacher centrality, structuredness ofthe learning environment); and (3) the raters’ meananalytic scores. The criterion variable was thestudents’ assessment of their teachers’ classroombehaviour.

To further explore the structure of the scoresgiven we searched for differences between theteachers. To see whether teachers could be groupedaccording to their scores, we used exploratoryQ-principal component analysis on raters’ assessmentsof the teachers whose portfolios showed sufficientinter-rater reliabilities (using .60 as a minimum for asignificant loading) and we used independentsamples T-tests to analyse the groups of teachersfound for differences in their students’ assessmentsof their classroom behaviour and for differences inthe raters’ assessments on the separate contentstandards (per teacher per content standard takingthe mean of the scores of both raters).

4. Results

4.1. Constructed teacher beliefs and behaviour

4.1.1. Teacher beliefs

Teachers particularly pursued the followinggoals: teach students the research subskills(n ¼ 16) and let students practice independentlearning (n ¼ 13). A third of the teachers (also)

aimed at developing students’ subject knowledge(n ¼ 5). As to control beliefs, according to allteachers their schools provided sufficient facilitiesfor student research. Sixteen teachers used researchassignments from schoolbooks and 13 (also) devel-oped assignments by themselves, half of them incollaboration with colleagues (n ¼ 8). Only seven ofthem recognized an approach for teaching studentsresearch skills in the schoolbooks. Only three ofthem could explain this approach. According to 16of the 18 teachers students had some difficulties withresearch assignments. The main problems con-cerned the feasibility of the assignments in view ofthe available time (n ¼ 9) and the data gathering(n ¼ 6). Nine teachers gave suggestions to solvethese problems and offering more coaching andfeedback was mentioned most (n ¼ 5). Two teachersdeclared to prefer no intervention at all, because‘‘doing research concerns student’s own learningprocess’’. The teachers differed in the degree ofstructuredness of the research assignments forstudents. Most of the teachers scaffolded theassignments by giving students a step-by-stepapproach (n ¼ 6) or to provide the theme of theassignment (n ¼ 5). In contrast, eight teachers didnot offer the students a theme at all. Half of theteachers intended that the degree of structurednessof the assignments fitted students’ learning pro-cesses in doing research. All but one teacher wassatisfied with the way they teach students researchskills (only one teacher felt insecure about herapproach). Half of the teachers had followed a 1-day or 2-day training in teaching students researchskills.

About teachers’ subjective norms, only half of theteachers made clear arrangements with their collea-gues about the research assignments, and this wasmainly done with regard to the content and the timeschedule. Some teachers experienced difficulties inmaking arrangements, e.g. ‘‘Last year we tried tomake arrangements about students’ assessment, butwe did not succeed’’. We asked teachers how toimprove the collaboration with colleagues. Eightteachers made suggestions. Two teachers said thatthere is nothing to improve and the others did notanswer this question.

4.1.2. Teacher behaviour

In the interviews before the video registrationthe teachers were asked how they intended tocoach their students. Most of them aimed atgiving feedback and providing examples, because


‘‘Students learn most by self regulated thinking’’(n ¼ 7), and ‘‘Students need scaffolds to produceresults’’ (n ¼ 8).

The two video registrations per teacher (n ¼ 36)showed instructional activities (mostly explainingthe assignment) (n ¼ 11) and teachers coachingstudents in class (n ¼ 11), in small groups(n ¼ 11), or individually (n ¼ 3). Most lessons tookplace in regular classrooms, five lessons in amultimedia centre.

The teaching and coaching activities mainlyconcerned the following research steps: identifyand formulate a problem using subject-specificconcepts; formulate a research question; gatherand select data (all in two-thirds of the videos);report on and present the research (in half of thevideos). In a third of the lessons attention was givento data analysis, and in only two registrationsteachers paid attention to drawing conclusions.Attention to the subskill ‘evaluating the research’was not shown in the registrations. This result maybe due to the selection of lessons (most teacherschose lessons and coaching sessions in the first halfof the research process of their students). Thelearning environments the teachers created differedsignificantly in their degree of structuredness (range1.30–2.93, SD .70) (F ¼ 3.11; df 35; p ¼ .019). Onaverage the learning environment was very struc-tured, due to teachers’ centrality as shown in theirtalking during 76% of the total video registeredtime. However, the teachers varied substantially inthis percentage (range 43–95%; SD .13) (F ¼ 7.79;df 35; po.001).

In the interviews after the video registrationsteachers mostly reported difficulties of students inidentifying and formulating a problem using sub-ject-specific concepts and in formulating the re-search questions (n ¼ 7), in gathering and selectingdata (n ¼ 4), in presenting the research (n ¼ 3), andin subject-specific knowledge (n ¼ 3). Most of theteachers dealt with these problems by giving theirstudents extra coaching (n ¼ 12). Seventeen teacherssaid they had reached their initial goals. Conse-quently, they were quite satisfied with their lessons.Ten teachers based this conclusion on: ‘‘Thecomments of students and the questions they pose’’,or on: ‘‘The confirming attitude shown by thestudents’’. What teachers meant by ‘students’attitude’ remained quite vague, for instance: ‘‘Stu-dents did not show any resistance, so they compre-hended the lesson’’. Five other teachers based theirconclusion on students’ learning outcomes and two

teachers relied on their own intuition. However, noteacher explicitly checked whether the students hadunderstood the lesson.

4.2. Assessed teacher beliefs and behaviour

4.2.1. Students’ assessments

The items in the student questionnaire formed areliable scale (Cronbach’s alpha .81). One-wayANOVA showed significant differences betweenthe classes (teachers) (F ¼ 10.19; df 17; po.001).On average the teachers scored best on the items‘‘My teacher is enthusiastic’’ (mean 3.43, SD .77)and ‘‘My teacher is willing to explain something fora second time’’ (3.38; .79). The teachers got thelowest scores on the items ‘‘My teacher verifieswhether we understand the assignment (2.74; .96)and ‘‘My teacher verifies whether we are actuallyworking on the assignment’’ (2.88; .95). Thequestionnaire results confirmed the results of theinterviews after the video registrations showing thatthe teachers hardly monitor students’ researchactivities in an explicit way.

4.2.2. Raters’ assessments

The jury-alphas for the separate analytic scoreswere satisfactory for 12 rater pairs, ranging from .39to .76. The jury-alphas were low or even negativefor six pairs, namely �.80 (one pair) and rangingfrom �.11 to .22 for five pairs. The contentstandards formed a reliable scale (Cronbach’s alpha.76). The rater pairs gave exactly the same holisticscores on the five-point scale in 35% of thejudgments, a difference of half a point showed upin 12% of the judgments, a difference of one pointin 47% of the judgments, and a difference of 1.5points in 6% of the judgments (one rater pair).

4.3. The correspondence between teacher beliefs and

behaviour as constructed and as assessed

All teachers in the pilot (n ¼ 3) and the mainstudy (n ¼ 18) had returned the questionnaireconcerning the relevance of the feedback given,based on the raters’ assessments. The items in thisquestionnaire formed three reliable scales. The firstscale contained questions about the relevance of thewritten feedback (four items, n ¼ 19), e.g. ‘Irecognize my teaching in the feedback given’(Cronbach’s alpha .84; mean 3.68; SD .19). Thesecond scale was about the transparency of theassessment (three items, n ¼ 20), e.g. ‘It is clear on


which standards the assessment of my portfolio isbased’ (Cronbach’s alpha .84; mean 4.0; SD .78).The third scale concerned the way in which theteachers got motivated by the feedback (three items,n ¼ 21), e.g. ‘The feedback gives information toimprove my teaching’ (Cronbach’s alpha .81; mean3.8; SD .98). The results show that the teachers werequite satisfied by the feedback given and evaluatedthe raters’ assessments as being relevant.

The multilevel analyses showed that the teachers’beliefs and behaviour as constructed in theirportfolios could not significantly predict the raters’mean analytic scores, but the students’ assessmentsof their teachers’ classroom behaviour could(r ¼ .21, po.05).

A Q-principal component analysis on the externalraters’ assessments of 12 teachers whose portfolioshad shown sufficient inter-rater reliabilities to takethe average of the ratings of the two raters percontent standard resulted in two groups of teachers(see Table 1). The components accounted for 34%and 22% of the variance.

Independent samples T-tests showed that theteachers in the first group (teachers 4, 5, 6, 7, 11,12) scored on average significantly higher on oneof the content standards, about choosing andarguing for teaching strategies that meet students’knowledge, abilities, and experience, than theteachers in the second group (t ¼ 2.78; df 9;p ¼ .02) (teachers 2, 8, 9, 10). That is, accordingto the raters, the teachers in the first group arebetter in explicating the rationales behind theirbehaviour (mean 2.13 on the three-point scale; SD.22) than the teachers in the second group (mean

Table 1

Results of an exploratory Q-principal component analysis on

external raters’ assessments of teacher portfolios (N ¼ 12)

Teacher Component 1 Component 2

1 �.02 �.23

2 .01 .62

3 �.45 .28

4 .61 .51

5 .93 .04

6 .75 �.02

7 .81 �.13

8 �.07 .64

9 �.16 .92

10 .11 .65

11 .93 .04

12 .71 .46

Bold values refer to significant component loadings.

1.55; SD .14). The two groups of teachers alsodiffered in their students’ assessments of theirclassroom behaviour. An independent samplesT-test showed that the students’ assessments in thefirst group (n ¼ 102; mean 3.21; SD .49) were onaverage significantly higher than those in the secondgroup (n ¼ 109; mean 2.93; SD .53) (t ¼ 4.05; df208; po.001). The two groups did not significantlydiffer on their overall portfolio ratings (meananalytic score and holistic score).

5. Conclusions

The purpose of our study was to research into therelation between teachers’ beliefs towards teachingbehaviour and their actual teaching behaviour inteacher portfolio assessment, focussing on theinteractive phase of teaching. Despite the wide-spread idea that teacher assessment should takeaccount of teacher beliefs in relation to theirbehaviour, the role of teacher beliefs in teacherportfolio assessments is far from clear. This is due tothe context-bound and tacit nature of teacher beliefsand the fact that raters’ interpretations of teacherbeliefs do not necessarily correspond with the beliefsthe teachers hold themselves.

We distinguished between teachers’ beliefs andbehaviour as constructed (by teachers in theirportfolios) and as assessed (by students observingtheir teachers’ behaviour and by raters assessing theteachers’ portfolios). A valid portfolio assessmentassumes a correspondence between beliefs andbehaviour as constructed and as assessed and sucha correspondence is needed to make a comparisonbetween beliefs and behaviour as assessed mean-ingful.

We investigated teachers’ constructed beliefs andbehaviour by analysing 18 original portfolios(including video registrations) about teaching stu-dents research skills. The video registrations showedthat the teaching of research skills mainly concernedthe research steps ‘identify and formulate a problemusing subject-specific concepts’, ‘formulate a re-search question’, and ‘gather and select data’. Thisis consistent with the research steps that the teachersthink the students find difficult to perform. Teachersdiffered in the approaches they use to teach studentsresearch skills. Most of the teachers intended tocoach the students. However, in practice theyhighly structured the learning environments. Onaverage they talked 76% of the time, and theybarely monitored students’ progress. The teachers


particularly aimed to teach their students researchsubskills and to let them practice independentlearning. They believed that they have sufficientfacilities to teach student research skills and mostteachers were satisfied with the way they teach theirstudents. Though teachers mainly use researchassignments from schoolbooks, only a third recog-nizes an approach for developing students’ researchskills in the book. Most perceived problems arefeasibility in time and making arrangements withcolleagues.

To investigate teachers’ assessed beliefs andbehaviour, we used both their own students’ andexternal raters’ assessments. Students’ assessmentsof their teachers’ classroom behaviour showed thatthey were most positive about their teacher’senthusiasm and least positive about the degree towhich their teacher verifies whether they compre-hend the assignment. According to the externalraters assessing teachers’ beliefs and behaviour asdescribed by the teachers in their portfolios, theteachers scored best on the content standard‘Choosing an appropriate assignment’ and worston the content standard ‘Reflecting on the programand on actions in teaching students research skills’.

The evidence about the correspondence betweenteachers’ beliefs and behaviour as constructed andas assessed is mixed. The teachers evaluated theraters’ assessments, based on their portfolios, asrelevant and their feedback as satisfying. However,the teachers’ beliefs and behaviour as constructed intheir portfolios could not significantly predict theraters’ mean analytic scores (the beliefs andbehaviour as assessed).

Also, the evidence about the correspondencebetween teachers’ beliefs and behaviour as assessedis mixed. We could not clearly demonstrate such acorrespondence. This is not an unknown phenom-enon and it is often explained by the nature of theconstructs assessed (teacher beliefs are mainly tacitand teacher behaviour is mainly routine). Further,as our sample of teachers was small and also theirbeliefs and behaviour could be represented in only asmall number of portfolio elements, we should notbe surprised by this result. Nevertheless, the externalraters’ assessments of the teachers’ beliefs andbehaviour (as constructed in their portfolios) onthe content standards were shown to be significantlyrelated to the students’ assessments of theirteachers’ classroom behaviour. Also, one of thetwo groups of teachers that could be distinguishedusing Q-principal component analysis was, accord-

ing to the external raters, better than the othergroup in explicating the rationales behind theirbehaviour and the classroom behaviour ofthese teachers was more highly evaluated by theirown students. These results could be interpretedas indicating at least some correspondencebetween teachers’ beliefs (or cognitions) and theirbehaviour.

The participating teachers in the study were allexperienced Dutch social science teachers in uppersecondary education who voluntarily participated inthe study. As a result, the sample will not berepresentative of the population of all social scienceteachers, let alone teachers in other subjects. Due tothe voluntary participation it is possible and evenplausible that the sample consisted especially ofteachers who had some experience in teachingresearch skills. The results cannot be generalizedto other types of contexts (e.g. situations in whichother types of teacher assessment instruments areused) or other categories of teacher beliefs andbehaviour.

The results of this study give way to furtherquestions addressing the role of teacher cognitionsin developing a portfolio. Since we assume thatteachers’ constructed beliefs and behaviour arepartly activated by the assessment instrument andthe context they react on, research is needed into thenature of teacher’s reactions and the conditions thatinfluence their responses. Such research is likely tobe complex, as teachers differ on personalitycharacteristics, experiences, competences, andworking context. Nevertheless, we think thatquasi-experimental studies with explicit variationin the structuredness of the portfolios applied indifferent settings combined with verbal protocols ofteachers, registered while they construct a portfolio,can reveal more insight in relevant teachers’cognitions.

Acknowledgement

This study was funded by the Dutch Organisat-ion for Scientific Research (NWO-PROO, 411-21-204).

Appendix A. Psychometric report of the scale in the

student questionnaire (n ¼ 317)

The students answered on a four-point scale: 1,‘not at all like my teacher’; 2, ‘to a little degree like


my teacher’; 3, ‘to a reasonable degree like myteacher’; 4, ‘very much like my teacher’.

Items
Mean SD Rir
1. The aim of theresearch assignment isclear

3.32
.77 .46
2. We have sufficientfacilities to do theassignment

3.27
.76 .38
3. The assessmentcontent standards areclear

3.26
.85 .32
4. The assignment isdifficult (recoded)

2.88
.9 .25
5. My teacher givespractical instructions

3.11
.78 .52
6. The teacher regularlyverifies whether weunderstand theassignment

2.74
.96 .53
7. My teacher gives clearinstructions

3.24
.78 .55
8. My teacher isenthusiastic

3.43
.77 .52
9. My teacher is willingto explain something asecond time

3.38
.79 .50
10. My teacher has faithin us

3.17
.85 .56
11. My teacher has highexpectations of us

2.93
.79 .28
12. My teacher isinterested in us

3.20
.78 .64
13. My teacher showsinterest in our ideas

3.16
.83 .64
14. My teacher takesnotice of contactsbetween students

2.91
1.91 .27
15. My teacher verifieswhether we are actuallyworking on theassignment

2.88
.95 .42
Cronbach’s alpha ¼ .81, overall mean ¼ 3.13, SD ¼ .89,

skewness ¼ �.26, kurtosis ¼ .97.

Appendix B. Rater manual, an example

Content standard 5 (TEACH): Teaching students

research skills

Score 3, the teacher satisfies the content standard

completely: The teacher uses multiple teachingstrategies that support the development of studentunderstanding in conducting investigations, andchallenges students to accept and share responsibilityfor their own learning (see content standard 4). He orshe uses the strategies in an adequate manner. Forexample, if students collaborate, he or she recognizesand responds to students’ diversity and encouragesall students to participate fully in fulfilling theassignment. He or she encourages each student toshare his or her ideas with the other members. Thestrategies he or she uses suit the phases of theinvestigations the students are working at. Forexample, in the phase of formulating a researchquestion, he or she coaches students in brainstorm-ing; in the phase of gathering information, he or shegives the students keywords for searching theinternet; in the phase of data processing, he or shedirects the students to work in groups. The teachermonitors students’ understanding and the amount ofdirection given in the teaching/learning process. Inorder to gather data, he or she can observe students,listen to students, and read their logs. He or she usesthe data to improve his or her teaching practice.

Score 2, the teacher satisfies the content standard

to a reasonable degree: The teacher uses, to a certainextent, teaching strategies that support the develop-ment of student understanding in conducting investi-gations, or challenges students to accept and shareresponsibility for their own learning. He or she usesthe strategies in a reasonable manner. The strategieshe or she uses may suit the phases of the investigationsthe students are working at. The teacher sometimesmonitors students’ understanding and the amount ofdirection given in the teaching/learning process.

Score 1, the teacher does not satisfy the content

standard: The teacher does not use teachingstrategies that support the development of studentunderstanding in conducting investigations, anddoes not challenge students to accept and shareresponsibility for their own learning. The strategieshe or she uses do not suit the phases of theinvestigations the students are working at. Theteacher does not monitor students’ understandingand the amount of direction given in the teaching/learning process.

Portfolio elements

A research assignment that is central to theportfolio, including the learning goals of the


assignment and the teacher’s reasons for the assign-ment content and form.

The results of two interviews concerning teachers’practical knowledge and intentions regarding in-structing and coaching students’ research skills.

Two video recordings of lessons in which theteacher instructs and coaches students doing re-search.

Assessment instruction

1.
Illustrate the content standard with significantportfolio evidence in the portfolio elementsmentioned above.
2.
Describe your interpretation of each kind ofevidence.
3.
Summarize your interpretation in a score on athree-point scale using the anchor points abovegiving evidence from the video most weight.
References

Aguirre, J., & Speer, N. M. (2000). Examining the relationship

between beliefs and goals in teacher practice. Journal of

Mathematical Behavior, 18, 327–356.

Ajzen, I. (2002). Perceived behavioral control, self-efficacy, locus

of control, and the theory of planned behavior. Journal of

Applied Social Psychology, 32, 665–683.

Ajzen, I., & Fishbein, M. (2005). The influence of attitudes on

behaviour. In J. Kuhl, & J. Beckman (Eds.), The handbook of

attitudes (pp. 173–221). Mahwah, NJ: Erlbaum.

Andrews, T. E., & Barnes, S. (1990). Assessment of teaching. In

W. R. Houston (Ed.), Handbook of research on teacher

education (pp. 569–598). New York: Macmillan.

Artzt, A. F., & Armour-Thomas, E. (1998). Mathematics

teaching as problem solving: A framework for studying

teacher metacognition underlying instructional practice in

mathematics. Instructional Science, 26, 5–25.

Beijaard, D., & Verloop, N. (1996). Assessing teachers’ practical

knowledge. Studies in Educational Evaluation, 22, 275–286.

Bird, T. (1990). The schoolteacher’s portfolio: An essay on

possibilities. In J. Millman, & L. Darling-Hammond (Eds.),

The new handbook of teacher evaluation: Assessing elementary

and secondary school teachers (pp. 241–256). Newbury Park,

CA: Corwin Press.

Borko, H., & Putnam, R. T. (1996). Learning to teach. In D. C.

Berliner, & R. C. Calfee (Eds.), Handbook of educational

psychology (pp. 673–708). New York: Macmillan.

Calderhead, J. (1996). Teachers: Beliefs and knowledge. In D. C.

Berliner, & R. C. Calfee (Eds.), Handbook of educational

psychology (pp. 709–725). New York: Macmillan.

Clark, C. M., & Peterson, P. L. (1986). Teachers’ thought

processes. In M. C. Wittrock (Ed.), Handbook of research on

teaching (3rd ed., pp. 255–296). New York: Macmillan.

Cochran-Smith, M., & Fries, M. K. (2001). Sticks, stones, and

ideology: The discourse of reform in teacher education.

Educational Researcher, 30, 3–15.

Cochran-Smith, M., & Lytle, S. L. (1999). Relationships of

knowledge and practice: Teacher learning in communities. In

A. Iran-Nejad, & C. D. Pearson (Eds.). Review of Research in

Education, 24, 249–305.

Crawley, F. E. (1990). Intentions of science teachers to use

investigative teaching methods: A test of the Theory of

Planned Behavior. Journal of Research in Science Teaching,

27, 685–697.

Delandshere, G. (1994). The assessment of teachers in the United

States. Assessment in Education, 1, 95–113.

Fazio, R. H. (1990). Multiple processes by which attitudes guide

behavior: The MODE model as an integrative framework. In

M. Zanna (Ed.), Advances in experimental social psychology,

Vol. 23 (pp. 75–109). San Diego, CA: Academic Press.

Fishbein, M., Triandis, H. D., Kanfer, F. H., Becker, M.,

Middlestadt, S. E., & Eichler, A. (2001). Factors influencing

behavior and behavior change. In A. Baum, T. A. Revenson,

& J. E. Singer (Eds.), Handbook of health psychology (pp.

3–17). Mahwah, NJ: Lawrence Erlbaum Associates.

Freidus, H. (1998). Mentoring portfolio development. In N.

Lyons (Ed.), With portfolio in hand. Validating the new teacher

professionalism (pp. 51–68). New York: Teachers College

Press.

Haney, J. J., Czerniak, C. M., & Lumpe, A. T. (1996). Teacher

beliefs and intentions regarding the implementation of science

education reform strands. Journal of Research in Science

Teaching, 33, 971–993.

Kagan, D. M. (1992). Professional growth among preservice and

beginning teachers. Review of Educational Research, 62,

129–169.

Kelly, G. A. (1955). The psychology of personal constructs. New

York: Norton.

Lievens, F. (2001). Assessor training strategies and their effects

on accuracy, interrater reliability, and discriminant validity.

Journal of Applied Psychology, 86, 255–264.

Lyons, N. (Ed.). (1998). With portfolio in hand. Validating the new

teacher professionalism. New York: Teachers College Press.

Pajares, M. R. (1992). Teachers’ beliefs and educational research:

Clearing up a messy construct. Review of Educational

Research, 62, 307–332.

Pula, J. J., & Huot, B. A. (1993). A model of background

influences on holistic raters. In M. M. Williamson, & B. A.

Huot (Eds.), Validating holistic scoring for writing assessment.

Theoretical and empirical foundations (pp. 237–265). Cresshill,

NJ: Hampton Press.

Reynolds, A. (1992). Getting to the core of the apple: A

theoretical view of the knowledge base of teaching. Journal

of Personnel Evaluation in Education, 6, 41–55.

Roth, W. M., & Roychoudhury, A. (1993). The development of

science process skills in authentic contexts. Journal of

Research in Science Teaching, 30(2), 127–152.

Saroyan, A., & Amundsen, C. (2001). Evaluating university

teaching: Time to take stock. Assessment and Evaluation in

Higher Education, 26, 337–349.

Saxe, G. B. (1991). Culture and cognitive development: Studies in

mathematical understandings. Hillsdale, NJ: Lawrence Erl-

baum.

Schoenfeld, A. H. (1998). Toward a theory of teaching-in-

context. Issues in Education, 4(1), 1–94.


Seldin, P. (1997). The teaching portfolio: A practical guide to improved

performance and promotion/tenure decisions. Bolton, MA: Anker.

Shapley, K. S., & Bush, M. J. (1999). Developing a valid and reliable

portfolio assessment in the primary grades: Building on practical

experience. Applied Measurement in Education, 12, 111–132.

Shulman, L. S. (1996). Paradigms and research programs in the

study of teaching: A contemporary perspective. In M. C.

Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp.

3–36). New York: Macmillan.

Stokking, K., Van der Schaaf, M., Jaspers, J., & Erkens, G.

(2004). Teachers’ assessment of students’ research skills.

British Educational Research Journal, 30(1), 93–116.

Van der Schaaf, M., (2005). Construct validation of teacher portfolio

assessment. Procedures for improving teacher competence assess-

ment illustrated by teaching students research skills. Doctoral

dissertation, Utrecht University, Utrecht, The Netherlands.

Vermunt, J. D., & Verloop, N. (1999). Congruence and friction

between learning and teaching. Learning and Instruction, 9,

257–280.

Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for

performance appraisal: A quantitative review. Journal of

Occupational and Organizational Psychology, 67, 189–205.

Marieke van der Schaaf (PhD) is an assistant professor in

Educational Sciences at the Department of Educational Sciences,

Utrecht University, The Netherlands. Her main areas of research

include teacher evaluation and teaching and assessment of

students’ general and academic skills.

Karel Stokking is a professor of education at the Department of

Educational Sciences, Utrecht University, The Netherlands. His

areas of interest include the development and assessment of

general and academic skills in secondary and higher education.

Nico Verloop is professor in education and director of ICLON

Graduate School of Education, Leiden University, The Nether-

lands. He is immediate past president of the Dutch Educational

Research Association. His major research interests are teachers’

practical knowledge, learning and professional development of

teachers and teacher evaluation.

teacher beliefs and teacher behaviour in portfolio assessment

Documents