This article was downloaded by: [Universitat Politècnica de València]On: 21 November 2014, At: 02:53Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Assessment & Evaluation in HigherEducationPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/caeh20

Validity and reliability of portfolioassessment in pre‐service teachereducationWil Meeus a , Peter Van Petegem a & Nadine Engels ba Universiteit Antwerpen , Antwerp, Belgiumb Vrije Universiteit Brussel , Brussels, BelgiumPublished online: 06 Jun 2009.

To cite this article: Wil Meeus , Peter Van Petegem & Nadine Engels (2009) Validity and reliabilityof portfolio assessment in pre‐service teacher education, Assessment & Evaluation in HigherEducation, 34:4, 401-413, DOI: 10.1080/02602930802062659

To link to this article: http://dx.doi.org/10.1080/02602930802062659

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Assessment & Evaluation in Higher EducationVol. 34, No. 4, August 2009, 401–413

ISSN 0260-2938 print/ISSN 1469-297X online© 2009 Taylor & FrancisDOI: 10.1080/02602930802062659http://www.informaworld.com

Validity and reliability of portfolio assessment in pre-serviceteacher education

Wil Meeusa*, Peter Van Petegema and Nadine Engelsb

aUniversiteit Antwerpen, Antwerp, Belgium; bVrije Universiteit Brussel, Brussels, BelgiumTaylor and Francis LtdCAEH_A_306433.sgm(Received 2 October 2006; final version received 2 March 2008)10.1080/02602930802062659Assessment and Evaluation in Higher Education0260-2938 (print)/1469-297X (online)Review Article2008Taylor & Francis0000000002008WilMeeusWil.Meeus@ua.ac.be

This article concentrates on the validity and reliability of portfolio assessment as usedin pre-service teacher education. It is not possible to make general pronouncementsabout the validity of portfolio assessment in pre-service teacher education as thereare multiple portfolio applications. The validity depends on the purpose, namely thedivers competencies which the course organisers wish to assess with it. Therefore,three categories of competencies and consequently three types of portfolios weredistinguished in order to determine the validity of portfolio assessment. For theassessment of teaching and partnership competencies, it is argued that the validity islow due to the roundabout nature of the assessment. On the contrary, the validity ofportfolio assessment for learning competencies can be high. The execution of a self-regulated learning process can be accurately assessed using portfolios. The reliabilityof portfolio assessment is problematic, since it is incapable of fulfilling the classicpsychometric requirement of reliability. Nevertheless, provided that the necessarymeasures are taken, the reliability of portfolio assessment can still be brought to anacceptable level. Five measures are proposed.

Keywords: portfolio assessment; teacher education; validity; reliability; self-regulatedlearning; learning competencies

Introduction

An exhaustive literature study of the use of portfolios in education (Meeus, Van Petegem,and Van Looy 2006) highlighted the existence of a considerable number of reports byteachers sharing their practical experience with this tool. But relatively few scholarlystudies are available yet, due to the rather recent popularity of the tool. It seems that theliterature is, in effect, split down the middle with regard to the validity of portfolio assess-ment: roughly half the authors express serious reservations about the validity of portfolioassessment, while the other half is positive about its validity. Why is the literature sodivided on this issue and who is right?

Oddly enough, both groups are right. The problem lies in a confusion regarding theconcept of what portfolios actually are. In reality there is not one portfolio, but rather thereare a series of different portfolio applications, each of which has its own format. Theconcept ‘portfolio’ has a degree of generality analogous to the concepts of ‘file’ or ‘assign-ment’. Statements on the validity of assignments cannot be done in any meaningful waywithout knowing what kind of assignment we are talking about. The same applies in the

*Corresponding author. Email: wil.meeus@ua.ac.be

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

402 W. Meeus et al.

case of portfolios. Before we can make any meaningful judgement with regard as to itsvalidity, we first have to specify the portfolio objectives at issue.

In pre-service teacher education, a variety of different portfolio applications is used.Methods of assessment need to be tailored to the objectives which are being pursued (VanPetegem and Vanhoof 2002). Therefore, we will focus on three categories of competencieswhich are the different objectives of portfolio assessment. We will look at the validity ofthe assessment of the respective three types of portfolios. By the validity of portfolioassessment, we mean the extent to which the teacher educator actually assesses what is theobjective of the assessment by means of portfolios. Thereafter, we go on to examine thereliability of portfolio assessment in pre-service teacher education. By the reliability ofportfolio assessment, we mean the extent to which extraneous influences are excludedfrom the assessment.

One competency is not the same as another

In teacher education, portfolios are used for the assessment of competencies needed to carryout the professional tasks of the teacher. We argue that different categories of competencieslead to substantial difference in the validity of portfolio assessment with regard to thesecompetencies. Wolf, Lichtenstein, and Stevenson (1997) distinguish between evaluationportfolios and professional development portfolios. Evaluation portfolios are used to assessteaching performance, while professional development portfolios are focussed on the learn-ing process and are therefore used to assess learning competencies. Following this line ofreasoning, portfolios can serve two different educational purposes in teacher education: thatof learning to teach and that of learning to learn.

All kinds of portfolio entries are useful for both categories. Examples are lesson plans,descriptions, observation notes, pictures, audio- and videotapes, etc. But these entriesdiffer in function (Bird 1990). For professional development portfolios, it is desirable toselect teacher’s worst work, so that it may be improved. Evaluation portfolios insteadshould be expected to contain a sample of the teacher’s best work, unless we suppose thata teacher can be persuaded to indict him or herself.

Teaching performance directly refers to supervision, education, expertise transfer andclassroom management. The competencies required here relate to teaching situations withpupils. We simply call these teaching competencies. But teaching performance is notlimited to classroom tasks. Teachers also have to be competent to deal with parents andcarers, colleagues and school management or others involved in school life. The compe-tencies required for performance of the professional tasks outside the classroom, inessence, relate to partnership. We refer to these as partnership competencies. The learningcompetencies of the teacher correspond to the teacher as lifelong learner, innovator andresearcher. Examples of learning competencies are: skills to work independently, theability to plan, the capacity for reflection, being able to modify one’s behaviour, etc.

The distinction between teaching competencies and partnership competencies is ofvalue because there is a significant degree of difference between how portfolios are usedfor these two sorts of competencies. This can be understood by placing it in the context ofthe existing array of assessment tools. For the assessment of teaching competencies,teacher education can fall back on a range of methods and tools. For instance, direct obser-vation of the student’s performance during his or her classroom work at placement iscommonly known and used by teacher educators. Portfolios can fulfil a supplementary rolehere. For partnership competencies, direct observation is almost impossible as actuallyattending conversations with parents or sitting in on staff meetings in order to assess the

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

Assessment & Evaluation in Higher Education 403

student’s performance usually is not an option. In this case, portfolios could play morethan a supplementary role. In fact, they could be the cornerstone of the assessment.

In conclusion, we distinguished three different categories of competencies which canbe assessed by the use of portfolios (see Table 1).

The validity of portfolio assessment for teaching competencies

Teaching competencies manifest themselves during teaching activities, usually in theclassroom. A valid portfolio assessment for teaching competencies presupposes portfoliosthat are suitable to decide whether or not students are able to teach.

The methods most commonly used to assess teaching competencies are: observation(e.g. by means of classroom visits); personal interviews (e.g. with the tutor or the studentsthemselves); and the evaluation of documents (e.g. lesson plans). Teacher educators havebeen using a combination of these methods to assess students’ teaching competenciesfor years, and it works perfectly well. So there seems no need for a new tool. However, acriticism which students make of the existing methodology is that the teacher educator’sevaluation is sometimes overly based on a mere handful of snapshots and does not givesufficient weight to the progress which the student is making. Supervisors usually see theirstudents once per teaching practicum. Portfolios might have something to offer here.

Portfolio assessment is an indirect method of assessment. It gives the teacher educatoran impression of the student’s teaching competencies, but this impression is not obtaineddirectly as in the case of observation. The impression is ‘mediated’. Portfolios could allowstudents to present their work in the classroom, but this raises the question of the truthful-ness of the representation as students will focus on their teaching success (Leggett andBunker 2006). This poses two significant problems.

First, there is the possible dishonesty of the student’s reflections on his or her perfor-mance. It is assumed that portfolios have to include reflection: the students’ own evaluationof their teaching competency. Students should be capable of conducting a critical appraisalof their own performance and should be prepared to recognise their weaknesses as well astheir strengths. But the student is placed in a critical position if he or she is asked to revealhis or her weaknesses when there is a lot at stake, in this case a teaching certificate. Thismeans in reality that student teachers can be penalised for revealing their weaknesses. In

Table 1. Three categories of competencies related to professional tasks of the teacher.

Teaching competenciesThe teacher as: • supervisor of learning and development processes• educator• content expert• classroom managerPartnership competenciesThe teacher as partner vis-à-vis: • parents and caretakers• colleagues and school management• third partiesLearning competenciesThe teacher as: • lifelong learner• innovator• researcher

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

404 W. Meeus et al.

this situation, student’s self-protection may lead to invalid portfolio assessment (Smith andTillema 1998). Harland (2005) and L. Fernsten and J. Fernsten (2005) emphasise thenecessity of a safe and confidential environment as a condition for honest reflections.

Obligatory reflections from students, posed in a vulnerable situation, lead to ‘contrivedcollegiality’ (Orland-Barak 2005). One might equally say that there is a chance that whatis really being assessed here is just the students’ ability to write what the teacher educatorwants to read (Korthagen 2004). For this reason, we advocate that students should not berequired to include reflection in portfolios used to assess teaching competencies. Studentshave to be able to limit themselves to reporting: giving a factual account of what they havedone.

A second important problem regarding the validity of portfolio assessment with respectto teaching competencies is the interference which is created by the fact of it being aconsciously constructed self-portrait of the student’s competencies (Wubbels, VanTartwijk, and Brekelmans 1996). The mark that the teacher educator awards thus comes todepend not only on the teaching competency of the student, but also on how successful thestudent has been in representing his or her teaching competency. After all, a lot of skillsare required in portfolio construction. The student’s writing skills influence how convinc-ing the reporting ultimately is. His or her skill in using ICT and other media impact thequality of the photographic material or video recordings used (Kommer and Meeus 2001).Students who are skilled in these areas are able to unfairly dress up their achievements,while students with a more modest command of these competencies run the risk ofreceiving an assessment which is below their real teaching competency.

There are ways of reducing the disadvantages that this interference causes. We suggestthree possibilities. First, course tutors could provide specific training to ensure that allstudents acquire the necessary competencies to put together portfolios which adequatelyreflect their teaching competencies (Tanner et al. 2000). Second, interim supervisionsessions can be organised. Teacher educators can then tone down overly self-promotingcompilations or offer additional help, if required. In this connection, Wolf, Whinery, andHagerty (1995) and Davis and Honan (1998) argue for supervision meetings with smallgroups of students. Finally, the quality of how the picture is put together can be used as aseparate assessment criterion so as to make the assessor more aware of the interference.Nevertheless, the indirect nature of portfolio assessment remains a pitfall.

So far, we have assumed a situation in which the teacher educator bases his or her eval-uation of students’ teaching competencies exclusively on portfolios. It is, of course, quitea different matter when portfolios are used in addition to, for example, observations andpersonal interviews. In this way portfolios can be beneficial to the validity of the assessment.The teacher educator then has additional information at his or her disposal, including a betterappreciation of the student’s overall progress. Nevertheless, this does raise the issue of theefficiency of portfolios for this purpose. After all, portfolios are very labour intensive, bothfor the students and for the assessors. The amount of materials that needs to be collectedto prove the acquisition of teaching competencies is probably the cornerstone of the wholeidea (Snyder, Lippincott, and Bower 1998).

In conclusion, the validity of portfolio assessment seems to be limited for teachingcompetencies due to the indirect nature of the assessment. For high-stake assessments, itis problematic to require students to provide reflections on their performance. Thesereflections may not be genuine if students run the risk of being penalised for being honest.In these cases, only factual reporting should be required. However, portfolios can make apositive contribution to the validity of the assessment of teaching competencies when usedin addition to other assessment tools such as observation and personal interviews. Yet, if

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

Assessment & Evaluation in Higher Education 405

this option is being considered, a cost–benefit analysis needs to be made before proceed-ing. The required investment in terms of time for the preparation and the assessment ofportfolios should be worthwhile in relation to the additional information they are likely toprovide.

The validity of portfolio assessment for partnership competencies

Partnership competencies manifest themselves during all activities which are peripheral toactual classroom teaching. These activities cannot, strictly speaking, be regarded as teach-ing but form an integral part of the teacher’s professional tasks in and outside the school.These activities are varied but related to maintaining appropriate contacts with educationalpartners other than pupils. If we want to examine the validity of portfolio assessment whenapplied to partnership competencies, we have to ask whether portfolios are suitable toassess activities of this kind.

The same disadvantages apply to portfolio assessment for partnership competencies asapply to its use in relation to the assessment of teaching competencies. The assessment isof an indirect nature and student reflection may well not be genuine. However, when usedto assess partnership competencies, portfolios can provide valuable extra informationgiven that the possibilities of direct observation of these competencies are limited. Let ustake the example of a conversation with parents or a staff meeting. Teacher educators canobtain information about the performance of the students during a conversation withparents or a staff meeting by talking to colleagues and reading over reports and otherdocumentation, but actually attending conversations with parents or sitting in on staffmeetings in order to assess the student’s performance seems neither desirable nor practical.Yet portfolios can fill this gap.

In conclusion, the validity of portfolio assessment also seems to be limited for partner-ship competencies by reason of the same disadvantages which are inherent in portfolioassessment of teaching competencies. However, portfolios can fill a larger gap with regardto the existing tools, given that observation of partnership activities is usually bothundesirable and impractical.

The validity of portfolio assessment for learning competencies

Since the importance of lifelong learning has penetrated into teacher education courses,many courses now pay more explicit attention to learning competencies (Kelchtermans2001). Yet there is still some way to go towards the recognition of the importance oflearning competencies.

In order to clarify the distinction between teaching competencies and learning compe-tencies we have taken inspiration from Elliott (2003), who uses Vygotsky’s concept of thezone of proximal development (1978). Let us imagine two hypothetical students who, at aparticular moment in time, have the same teaching competencies (see Figure 1), but a differ-ent learning capacity, i.e. the extent to which they possess the necessary competencies tolearn from and for their teaching practice. The size of the zone of proximal developmentis a measure of the learning capacity of the students. Now imagine that both students areconfronted with a problematic teaching situation and both have access to the same super-vision. Student A will be able to gain more from the learning situation than student B. StudentA has a greater learning competency than student B.Figure 1. Difference in learning competencies given equal teaching competencies.Assessing learning competencies means assessing students’ capacity to execute a self-regulated learning process. Learning competencies like analysing personal teaching

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

406 W. Meeus et al.

strengths and weaknesses, planning and reflective thinking can be accurately assessed byportfolio assessment. Moreover, there are few tools to compete with portfolios for theassessment of learning competencies as teacher education does not usually has the meansto closely observe students during a very long period. Meeus, Van Looy, and Van Petegem(2005) describe the method to be followed: the student chooses a personal learning compo-nent in consultation with the course tutors; draws up a personal learning plan; puts thisplan in practice using the cycle of action and reflective thinking; and provides the neces-sary reporting. To discuss the validity of this type of portfolio assessment, it is importantto explore what is meant by reporting, reflection and the personal learning plan in thecontext of portfolios.

Reporting teaching activities

In portfolio assessment of learning competencies, the student produces a report on his orher school activities. However, this reporting has a totally different function than it doesin the case of portfolio assessment of teaching or partnership competencies. For portfolioassessment of teaching and partnership competencies, students have to use materials whichallow them to show themselves at their best. This probative material is crucial to theassessment of the quality of their teaching and partnership competencies. To illustrate howthis works in practice, we have included a quotation from a teacher educator duringsupervision relating to portfolio assessment for teaching competencies:

In the video clip I can hardly see the pupils, but I can hear a lot of noise in the background,which makes me wonder whether they are paying attention. The tutor’s report also says thatyou have difficulty keeping the class under control and this is not a badly behaved group. SoI can’t give you a good mark for classroom control.

Portfolios as assessment tools for learning competencies take a completely differentapproach. A student’s learning capacity is assessed on the basis of their development, noton the basis of reporting. The students have to describe their teaching activities and the

���������

�����

����������

���������

�����

����������

���� �� �������

������������ �� �

������ ����

���� �� �������

������������ ����

������ ����

����������������

Figure 1. Difference in learning competencies given equal teaching competencies.

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

Assessment & Evaluation in Higher Education 407

associated context so that the teacher educator can form an impression of their develop-ment. The material is essentially informative in nature so as to enable the assessor to eval-uate the student’s learning process. In this case, we would do better to talk in terms ofinformative rather than probative material. Students who are not as good as some of theirclassmates at presenting themselves in their best light will not be at a disadvantage.Furthermore, students have nothing to gain by exaggerating the positive elements in theirreporting. They can represent their teaching performance honestly, including all theirerrors and shortcomings without being afraid of jeopardising their chances of a good grade.After all, they are being marked on their learning not their teaching, thus eliminating themost important disadvantages of the indirect nature of portfolio assessment. To illustratehow this works in practice, we again include a quotation from a teacher educator duringsupervision, this time relating to portfolio assessment for learning competencies:

I see on the first video clip that your class control leaves something to be desired. In yourreflection, however, you say that you can see what the problems are. You have tried to makeyour lessons more interesting and to be more consistent in your interaction with the pupils.The last video clip shows that you haven’t quite managed this yet, but you have clearly madesome improvement. This shows that you are able to learn from your teaching practice. I amtherefore giving you a good mark.

Even if the informative material (in the case of portfolio assessment of learning compe-tencies) is less crucial than the probative material (in the case of portfolio assessment ofteaching competencies), it still has to meet a number of requirements in order to be useful.In the first place, it has to be relevant. It has to say something about the teaching compe-tencies that the student wants to work on. If the student wants to work on improving his orher interpersonal contact with the pupils, there is not much point in showing a video clipof him or her giving unidirectional instruction. Secondly, the reporting has to includeaudiovisual aids, as sound and images enhance the written material. Thirdly, portfoliosneed to contain a variety of materials. The greater the variety of materials included, themore informative portfolios will be.

Using portfolio assessment for learning competencies could create the erroneousimpression that the students are no longer being assessed on their teaching competency.This is still necessary, of course, but not by means of portfolios. The placement supervisorassesses the teaching competencies of the students based on teaching practicum visits,personal interviews and lesson plans. It is also important that the students have differentteaching and learning assessors, as combining the two types of portfolio assessment wouldjeopardise the validity. The students would end up in a situation in which they might bepenalised in the assessment of his or her teaching practicum for presenting honest learningprocesses in their portfolios.

Reflection on the teaching activities

Reflecting is a learning competency which can be assessed via portfolios. Reflectivethinking is not a spontaneous everyday activity. The only reflective thinking which we dospontaneously is when something goes wrong or when we are worried about failing.Reflective thinking requires an investment in terms of time and effort that we are notalways prepared to make. It is a process whereby the unconscious selection of spontaneousthoughts is channelled in the direction of a conscious objective while weeding outirrelevant thoughts (Gelter 2003). It is a conscious and active scholarly practice of goal-orientated and structured thinking. It is therefore a process which has to be systematically

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

408 W. Meeus et al.

learned. Reflective thinking is a powerful tool for successfully coming to grips, as ateacher, with the complexity of teaching situations.

Reflection can be assessed on the basis of form and content. Korthagen (1983) has devel-oped a specific reflection cycle for teacher education courses, which is a normative modelfor the form that the reflective thinking should take. The cycle consists of five phases, ofwhich three are directly related to reflective thinking: looking back on past performance;awareness of essential aspects; and the creation of alternatives (see Table 2). Reflectionhas to include these phases. Students must devote sufficient time to analyse their teachingactivities. Moreover, they must come up with adequate alternative action strategies in orderto find solutions to the problems they encounter.

Kelchtermans (2003) states that the scope of the reflection has to be sufficiently broadand deep. In terms of breadth it must cover four domains:

● Reflection on the technical domain, meaning that students should think about theirinstructional approach.

● Reflection on the emotional domain, which is mainly concerned with the student’sself-esteem as a teacher.

● Reflection on the political domain, which refers to issues of control, influence andpower in the classroom and in the school.

● Reflection on the moral domain, which is fundamentally concerned with what is inthe pupil’s best educational interests.

Reflective thinking also has to be sufficiently deep in terms of its exploration of theissues involved. Students need to be capable of going beyond the level of mere efficientfunctioning as teachers and to penetrate as far as the level of their underlying assumptions,values and standards. The student must examine his or her personal interpretative frame-work critically. This personal interpretative framework includes their professional self-image (how they see themselves as teachers) and their subjective theory of teaching (theirvision of what teaching involves). Table 2 summarises the requirements for qualitativereflection in terms of form and content, which is how to arrive at an effective frameworkfor the assessment of reflection as a learning competency.

To illustrate the categories of reflection two different written reflections of student teach-ers are included. An analysis is added to provide insight in the differences in reflective quality.

Last week I had some trouble keeping the pupils under control. The lesson didn’t really startbad. In fact the pupils weren’t noisy during the first 10 minutes. But as the lesson wore on,the level of noise became more irritating. I gave some remarks but the effect didn’t last long.At the end of the lesson I was yelling. I was relieved when the bell rang. I guess I let it cometoo far, and yelling at the class doesn’t work. I spoke with my mentor and she gave me someleads. Next time I will try to respond sooner and to adopt a more individual approach.

Table 2. Requirements for qualitative reflection in teacher education.

Qualitative reflection

Content Form

Breadth: Four dimensions Depth: Two levels

Three phases of the reflective thinking cycle

• Technical• Emotional• Political• Moral

• Efficient and effective teaching• Personal interpretative framework:

• Professional self-image• Subjective theory of teaching

• Looking back on past performance• Awareness of essential aspects• Creating alternative methods

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

Assessment & Evaluation in Higher Education 409

Concerning the form of reflection we can recognise the three phases. The student looksback on her past performance and becomes aware of essential aspects like waiting too longto intervene and addressing the class as a whole. Alternatives were formulated during aconversation with the mentor. The reflection sticks to the level of efficient functioningwithin the technical domain (respond sooner, individual approach). A supervisor shouldaddress deep-level reflection by directing towards her personal interpretative framework.The moral, political and emotional domain could also be addressed. Compare this with thenext one:

Last week I had some trouble keeping the pupils under control. It didn’t really start bad, butas the lesson wore on, the level of noise prevented me from finishing what I prepared. Itcrossed my mind to give a disciplinary punishment but I hesitated. My intention was never topunish as a teacher. I want the pupils to like me and that will never happen when I punishthem. I’m also a little bit scared that the pupils will oppose to the punishment. How can I winthis?

Concerning the form of reflection we can recognise only one phase: the student looks backat her past performance. But she doesn’t look for causes and she doesn’t formulate alter-natives, except maybe a disciplinary punishment. But since we lack insight in the cause,we haven’t any guarantee that it is what needs to be done. She mentions aspects within thepolitical domain (winning) and the emotional domain (to be scared). We have evidence ofdeep-level reflection as her professional self-image (want to be liked, never to punish) isunder discussion. A supervisor should address better awareness and analyses beforejumping to alternatives. The technical and moral domain could also be addressed.

Personal learning plan

Planning is also a learning competency which can be assessed via portfolios. Drawing upa personal learning plan is therefore a key element of portfolio assessment for learningcompetencies. Students can use their individual personal learning plan as a means ofmonitoring and fine-tuning their progress (Campbell et al. 2000). The personal learningplan contains a chronological list of all the activities which the students wish to carry outin order to learn. The personal learning plan also needs to take account of the reporting ofthe teaching activities. Collecting the necessary materials together and using media tocreate material requires a great deal of planning and organisation (Pleasants, Johnson, andTrent 1998).

The personal learning plan must comply with a number of requirements. First, it needsto be realistic. Learning is generally a slow process in which a large number of small stepshave to be taken, whereas most students want to gallop ahead. Second, the planning has tobe as comprehensive as possible. Important steps in their learning process must not beoverlooked. Third, it should be used in a flexible way. Students should be able to deal withunexpected developments (see Table 3).

Final conclusions on portfolio assessment for learning competencies

In conclusion, portfolio assessment seems to be valid for the assessment of learningcompetencies, especially because there are few practical alternatives at hand. Students’capacity to execute a self-regulated learning process can be accurately assessed by portfo-lio assessment. In this case, reporting does not serve as a means of assessing teachingcompetencies, but rather as a way of getting a more accurate impression of the student’s

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

410 W. Meeus et al.

learning process. Learning competencies like planning and reflective thinking are keyelements of the learning process.

The reliability of portfolios

By the reliability of assessment, we mean the extent to which extraneous influences areexcluded from the assessment. A reliable assessment should give accurate measurements.Calibrated and standardised tools, such as multiple choice exams, can be relativelyfree from measurement errors because they allow the assessors little or no margin forinterpretation. The desirability of a high reliability might, therefore, be an argument forstandardising portfolios, but this is contrary to the nature of the tool. The aim of portfoliosis, after all, to allow authentic learning in various contexts. Therefore, students need to begiven the necessary freedom to decide for themselves which teaching activities they wantto represent in their portfolios. In a nutshell: more standardisation means impoverishedportfolios. It is probably impossible to obtain a 100% reliability of portfolio assessment.However, not everything that is of value in education can be measured with complete reli-ability (Luken 2004). Yet there are certainly ways of improving the reliability of portfolioassessment, all of which are aimed at removing undesired influences as far as possible. Weset out five such possibilities.

First, reliability is increased if a single assessment protocol is established for all asses-sors (Heller, Sheingold, and Myford 1998). A protocol of this kind needs to provideanswers to questions such as: Are the activities to be assessed separately or only in theirtotality? Is assessment to be continuous or only after final submission? Are interim assess-ment sessions to be provided as well as final assessments? The assessors have to make cleararrangements about this. Dierick, Van de Watering, and Muijtjens (2002) call this protocolmoderation, in other words, structural consultation, in this case before the assessment.

Second, the reliability of portfolios is enhanced by the use of a common interpretativeframework (LeMahieu, Gitomer, and Eresh 1995). A brief checklist with the overallassessment criteria is quite simple to use. Assessment scales (e.g. from inadequate to verygood) or assessment rubrics (with a description of the quality levels) can be used.

We also make a distinction between analytical and holistic marking (Mabry 1999).With analytical marking the assessor awards marks for separate elements which he or

Table 3. Part of a personal learning plan: student teacher in Biology trying to motivate her pupils.

Date Planned activities Done Context data

20/02 Immunology: practical assignments Yes 5th class, 18 pupils (8 girls, 10 boys).Difficult to motivate

13/03 Immunology: quiz + presentation on HIV and AIDS

Yes Idem

16/03 Biotechnology: practicum ‘on paper’ – simulation of a DNA fatherhood test

Yes 6th class, 15 pupils (5 girls, 10 boys). Girl with psychological problems

and boy with minor form of autism30/03 Biotechnology, genetically manipulated

organisms: class discussionReplaced by

other lessonIdem

18/05 Evolution: video + discussion Yes Idem29/05 Ecology: nitrogen and phosphorus

cycle – microscopy, observation of living material, combine exercise and word searcher

Yes 5th class, 18 pupils (8 girls, 10 boys). Difficult to motivate

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

Assessment & Evaluation in Higher Education 411

she then adds up or averages out to arrive at a single final mark. In the case of holisticmarking, the assessor awards a single (global) final mark and separate elements are notgiven separate marks. Different assessment criteria are still used, but only qualitativemeasures are applied. Research shows that analytical portfolio marking is less reliablethan holistic portfolio marking (Baume and Yorke 2002). Holistic marking is, therefore,a preferable way to be used when assessing portfolios.

Inevitably, there is always a certain assessment margin. In the case of portfolios, theso-called inter-assessor reliability is by definition problematic (Carpenter and Ray 1995;Verloop and Wubbels 2000). Therefore, the expertise of the assessors needs to be fosteredby training and support. At all events, portfolio assessment is always better done by anumber of assessors. If there is a wide discrepancy in the assessments, consultation isindicated, which is also a form of moderation, but after the individual assessment.

The reliability of portfolios can thus be improved by:

● using a common assessment protocol (prior moderation);● using a common checklist of assessment criteria;● holistic marking;● adequate training of assessors; and● use of various assessors (retrospective moderation).

Conclusion

It is not possible to make general pronouncements about the validity and the reliability ofportfolio assessment in teacher education. The validity of portfolio assessment depends onthe competencies which the teacher educators wish to assess with it.

For the assessment of teaching or partnership competencies, the validity is rather lowdue to the indirect nature of the assessment. It is to the student’s advantage to give a betterimpression of his or her performance than is really the case. Portfolios can be used for theassessment of teaching competencies as a complement to other tools. Portfolios offeradvantages for partnership competencies given that direct observation is usually neitherpossible nor desirable.

The validity of portfolio assessment of the learning competencies is high, however.The student’s capacity to execute a self-regulated learning process can be accuratelyassessed using portfolio assessment. In this case, the reporting of the teaching activities isa less crucial element in comparison with portfolio assessment for teaching competencies.The students only have to give an impression of the learning process by collectinginformative representations.

By definition, portfolios are incapable of fulfilling the classic psychometric requirementof reliability. Portfolios and standardisation are essentially incompatible. Nevertheless,provided that the necessary effort is made, the reliability of portfolios can still be broughtto an acceptable level. The reliability of portfolios for the assessment of competencies inthe teacher education thus depends on what measures are taken to ensure this. We haveproposed five measures.

Notes on contributorsWil Meeus is lecturer in the Institute of Education and Information Sciences at the UniversiteitAntwerpen. He teaches the modules Educational Sciences, Reflective and Research-orientedPractice, and Portfolio. His doctoral thesis focussed on the use of portfolios in teacher education.

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

412 W. Meeus et al.

Peter Van Petegem is professor in the Institute of Education and Information Sciences at theUniversiteit Antwerpen and heads the research group EduBRon (www.edubron.be) and theExpertisecentrum Hoger Onderwijs (www.ua.ac.be/echo).

Nadine Engels is senior lecturer and head of the Department of Educational Sciences at the VrijeUniversiteit Brussel. Among other modules she teaches Reflective and Research-oriented Practice,and Learning Processes.

ReferencesBaume, D., and M. Yorke. 2002. The reliability of assessment by portfolio on a course to develop

and accredit teachers in higher education. Studies in Higher Education 27, no. 1: 7–25.Bird, T. 1990. The schoolteacher’s portfolio: An essay on possibilities. In Handbook of teacher

evaluation: Elementary and secondary school teachers, ed. J. Millman, and L. Darling-Hammond, 241–55. Newsbury Park, CA: Sage.

Campbell, D.M., B.J. Melenyzer, D.H. Nettles, and R.M. Wyman Jr. 2000. Portfolio andperformance assessment in teacher education. Needham Heights, MA: Allyn & Bacon.

Carpenter, C.D., and M.S. Ray. 1995. Portfolio assessment: Opportunities and challenges.Intervention in School and Clinic 31, no. 1: 34–41.

Davis, C.L., and E. Honan. 1998. Reflections on the use of teams to support the portfolio process.In With portfolio in hand: Validating the new teacher professionalism, ed. N. Lyons, 90–102.New York: Teachers College Press/Columbia University.

Dierick, S., G. Van de Watering, and A. Muijtjens. 2002. De actuele kwaliteit van assessment:ontwikkelingen in de edumetrie [The current quality of assessment development in theedumetrics]. In Assessment in onderwijs [Assessment in education], ed. F. Dochy, L. Heylen,and H. Van de Mosselaerde fa, 91–122. Utrecht: Lemma.

Elliott, J. 2003. Dynamic assessment in educational settings: Realising potential. EducationalReview 55, no. 1: 15–32.

Fernsten, L., and J. Fernsten. 2005. Portfolio assessment and reflection: Enhancing learningthrough effective practice. Reflective Practice 6, no. 2: 303–9.

Gelter, H. 2003. Why is reflective thinking uncommon? Reflective Practice 4, no. 3: 337–44.Harland, T. 2005. Developing a portfolio to promote authentic enquiry in teacher education.

Teaching in Higher Education 10, no. 3: 327–37.Heller, J.I., K. Sheingold, and C.M. Myford. 1998. Reasoning about evidence in portfolios: Cognitive

foundations for valid and reliable assessment. Educational Assessment 5, no. 1: 5–40.Kelchtermans, G. 2001. Reflectief ervaringsleren voor leerkrachten: Een werkboek voor opleiders,

nascholers en stagebegeleiders [Reflective experience-based learning for teachers: A workbookfor teacher educators, on-going education providers and placement supervisors]. Deurne:Wolters Plantyn.

Kelchtermans, G. 2003. De kloof voorbij: Naar een betere integratie van theorie en praktijk in delerarenopleiding [Bridging the gap: Towards a better integration of theory and practice inteacher education]. Brussels: Vlaamse Onderwijsraad.

Kommer, S., and W. Meeus. 2001. Computergebruik in de school, een kwestie van mediadidactiekof mediapedagogiek? [Computer use in schools, a question of media didactics or media peda-gogics?] Schokla 43, no. 1: 2–6.

Korthagen, F.A.J. 1983. Leren reflecteren als basis van de lerarenopleiding [Learning to thinkreflectively as the basis of teacher education]. The Hague: Foundation for Research on theEducation.

Korthagen, F. 2004. Zin en onzin van competentiegericht opleiden [Sense and nonsense in compe-tency-oriented education]. VELON Tijdschrift voor lerarenopleiders 25, no. 1: 13–23.

Leggett, M., and A. Bunker. 2006. Teaching portfolios and university culture. Journal of Furtherand Higher Education 30, no. 3: 269–82.

LeMahieu, P.G., D.H. Gitomer, and J.T. Eresh. 1995. Portfolios in large-scale assessment: Difficultbut not impossible. Educational Measurement: Issues and Practice 14, no. 3: 11–16 and 25–28.

Luken, T.P. 2004. Zijn competenties meetbaar? [Are competencies measurable?] Tijdschrift voorHoger Onderwijs 22, no. 1: 38–53.

Mabry, L. 1999. Portfolios plus: A critical guide to alternative assessment. Thousand Oaks, CA:Corwin Press.

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014

Assessment & Evaluation in Higher Education 413

Meeus, W., L. Van Looy, and P. Van Petegem. 2005. Het portfolio als professioneel gerichteindwerk: Een onderzoek in de lerarenopleidingen kleuter- en lager onderwijs [The portfolioas profession-orientated dissertation: A study in the teacher education for pre- and primaryschool education]. Brussels: VUB Press.

Meeus, W., P. Van Petegem, and L. Van Looy. 2006. Portfolio in higher education: Time for aclarificatory framework. International Journal of Teaching and Learning in Higher Education17, no. 2: 127–35.

Orland-Barak, L. 2005. Portfolios as evidence of reflective practice: What remains ‘untold’.Educational Research 47, no. 1: 25–44.

Pleasants, H.M., C.B. Johnson, and S.C. Trent. 1998. Reflecting, reconceptualizing, and revising:The evolution of a portfolio assignment in a multicultural teacher education course. Remedialand Special Education 19, no. 1: 46–58.

Smith, K., and H. Tillema. 1998. Evaluating portfolio use as a learning tool for professionals.Scandinavian Journal of Educational Research 42, no. 2: 193–206.

Snyder, J., A. Lippincott, and D. Bower. 1998. Portfolios in teacher education: Technical ortransformational. In With portfolio in hand: Validating the new teacher professionalism, ed.N. Lyons, 123–42. New York: Teachers College Press/Columbia University.

Tanner, R., D. Longayroux, D. Beijaard, and N. Verloop. 2000. Piloting portfolios: Usingportfolios in pre-service teacher education. ELT Journal 54, no. 1: 20–30.

Van Petegem, P., and J. Vanhoof. 2002. Evaluatie op de testbank: Een handboek voor hetontwikkelen van alternatieve evaluatievormen [Evaluation on the test bank: A handbook for thedevelopment of alternative methods of evaluation]. Mechelen: Wolters Plantyn.

Verloop, N., and T. Wubbels. 2000. Some major developments in teacher education in theNetherlands and their relationship with international trends. In Trends in Dutch teacher educa-tion, ed. G.M. Willems, J.H.J. Stakenborg, and W. Veugelers, 19–32. Leuven-Apeldoorn:Garant.

Vygotsky, L. 1978. Mind in society: The development of higher psychological processes.Cambridge, MA: Harvard University Press.

Wolf, K., G. Lichtenstein, and C. Stevenson. 1997. Portfolios in teacher evaluation. In Evaluatingteaching: A guide to thinking and best practice, ed. J.H. Stronge, 193–214. Thousand Oaks,CA: Corwin Press.

Wolf, K., B. Whinery, and P. Hagerty. 1995. Teaching portfolios and portfolio conversations forteacher educators and teachers. Action in Teacher Education 17, no. 1: 30–9.

Wubbels, T., J. Van Tartwijk, and M. Brekelmans. 1996. The compilation of portfolios which meetthe needs of assessment objectives. In Proceedings of the 2nd International Conference onTeacher Education: Stability, Evolution and Revolution, 891–904. Netanya, Israel: MofetInstitute.

Dow

nloa

ded

by [

Uni

vers

itat P

olitè

cnic

a de

Val

ènci

a] a

t 02:

53 2

1 N

ovem

ber

2014