1. The Problem of Policy Implementation: the case of performance appraisal

2. Teacher Learning Through Portfolios

Group 8 Members

Lam Ka Po, Angela

Lam Wing Chung, Terence

Lok Wai Yi, Begonia

Ng Siu Kai

The Problem of Policy Implementation

The case of performance appraisal

Aim of the research

• To identify the obstacles which lead to slow

and sporadic policy implementation

• To investigate new perspectives of

implementing policy as solving a problem

• To identify how policy implementation

could be improved

Obstacles of current Appraisal system

• Interest group approach

• Initiatives come from governments and

their associated ministries and departments

• Affected by the political origins of these

initiatives

• Conflict between the government and the

teachers associations

Policy as Solving a Problem

• Offer a conceptualization of performance

appraisal as a complex problem

• Specify the requirements for a quality

solution at both national and local levels

• Prevent satisfying the requirements of

particular interest groups rather than

teachers themselves

Definition of Problem

All the conditions or constraints on the solutio

n (variously weighted) plus the demand that

the solution (an object satisfying the constra

ints) be found

- by Thomas Nickles

Constrains on the solution

• Values– the policy should recognize the professionalism

of teachers

• Practical requirements– it must not cost more than schools can afford

• Goals– it must improve the quality of teaching and

learning

• or any combinations

Integration of constraints

• Particular constraints is likely to be in conflict in complex problems

• Need to integrates the constraints taken as a set by reaching a compromise between different interest groups

• Require shifts in the definition of particular constraints to obtain an adequate solution

Phases in Policy Making

• Intended Policy– What various groups wants

• Actual Policy– The document, legislation and/or report

• Policy-in-use– Regional and school-level reaction

- by Bowe et al.

Scenario Highlights

Principal of Riverview High School

John Burton’s dilemma

Appraisal system• Aim :

to improve teaching and learning

• Safeguard :negative evaluations of individual staff performance would not be reported to the senior management team.

Education Review Officereported:• appraisal system failed

to give the management information on the performance of staff.

Good points of the old system

• Staff were comfortable

• Staff were honest with their chosen appraiser

• Allow staff to identify professional development

goals that would impact positively on children’s

learning

• Staff has the authority to report and to report

what

John Burton’s Dilemma

How to make the appraisal system more structured and accountable

while simultaneously supporting my staff to challenge themselves professionally and to be open about identifying their professional development needs.

A Revised Appraisal System• Guidelines:

– Staff could choose their own appraiser– limited to themes determined by the SMT– observe the work of another staff member in an

area in which they wanted to improve

• Outcomes:– 40% of staff returned the questionnaire on this

revised system– Some staff were critical and some made very

positive comments.– 60% of staff ignored the appraisal guidelines

• Do I have the authority to check up

on those who have not responded to

the questionnaire?• What I can do about those who

had not participated?

Staff’s desire

• non-threatening scheme that emphasized professional development

Management’s concern

• enhance teaching and learning

• introduce a more accountable system that mettheir reportingrequirements

Theory-in-useConstraints

Practices Consequences

Enhancesteaching &learning

Equality of authority

Non-judgmental&non-threatening

SMT suggests what should be appraised& requires classroom visits

Entire processremains confidential

Self-appraisal emphasized

Peer supportencouraged &individuals select own appraiser

Some staff positive,others critical

Participation rates & impact of appraisal unknown

Principal unable to fulfil reporting requirements

The Intervention Process

By

the School of Education, University of Auckland

Invited by

the School Management Team of

Riverview High School

Aims of the Intervention

• to assist Riverview High School to develop an effective performance appraisal system

Methodology : Problem Based

• Examined the way Riverview solved the problem by discovering

– the constraints they set on the solution

– the strategies they used to integrate those

constraints.

Data Collection Methods

• 3 interviews with the senior management staff

• 6 interviews with other staff

• 2 feedback meetings with the senior

management team

• Observations and audio transcripts of both staff

and departmental meetings.

• Review relevant documents, e.g. appraisal

policy

How to implement

• The assistance comprised of 8 workshops

– focusing on the adequacy of current practice

– formulation of a new set of constraints

– developing appraisal practices consistent with

those constraints

– interpersonal skills training

• enabled the SMT to debate the constraints that

led to an integration, rather than opposition

The First Phase

• To identify the values, beliefs and practical

issues that underlay current practice in

order to determine what needed to be

revised and what could be further

developed.

The second Phase

• Formulate a new set of constraints which enabled

the SMT to make explicit their requirements.

• Five nominated constraints were

– made a difference to teaching and learning

– linked to professional development

– respected and supported staff

– held all levels of staff accountable for performance

– practical and efficient

The Third Phase

• Meetings of staff– answer questions

– staff learnt that the problems are difficult

because they had multiple requirements with

considerable tensions between them.– Continued to debate and resolve questions

The Last Phase

• To en-skill the senior management team to articulate the tensions between their proposed constraints and debate their resolution with their staff.

Two full staff meetings

• Both meetings devoted to appraisal

• Showed a shift in the staff’s thinking and the quality of the debate

11 weeks later

First meeting

• The Principal made no mention of the list of constraints as some separate points but an interacting whole

• Staff discussed selected constraints that were of concern to them

• The trained SMT explained that making a difference in the classroom was the key factor but the allocation of staff development was a link only

Second Meeting (11 weeks later)

• Staff began to acknowledge– the tensions between the constraints and to

search for ways to integrate them– they were concerned with the relationship

between appraisal and other school practices– how aspects of the proposal met two or more of

the proposed constraints

• Both staff and SMT deepened their understanding of the integrated nature of the appraised problem

Evaluation of Riverview practice

• Enabled the senior management team to :

– Determine what was desirable

– Determine what was problematical

– Identify the values and beliefs that sustained it

• Enable new policy to be formulated in

addressing current constraints

• Determine what should be retained and

revised

Implications for National Policy

Formulation of a new constraint set at

national level would encourage policy

makers to identify:

• The purpose of appraisal

• The values guiding its introduction

• The practical problems

• What practices are ruled in or ruled out

Skills required for improvement

• Demands new interpersonal skills

• Satisfy a complex constraint set under considerable tensions

• Requires skills to confront and debate the merit of competing perspectives

• Requires co-operations

• Requires the commitment and skills to develop a shared understanding

How policy implementation could be improved ?

• Integrating the constraint set as a whole• Testing proposed constraints by developing

exemplars of practice• Revising either the constraints, the practice

or both• Consulting with practitioners and interest

groups• Formulating new approach – Portfolio ?

Teacher Learning Through Portfolios

Background behind Using Portfolio for Professional Development

• Charles Handy (1996)

– Incremental change becomes discontinuous cha

nge in the Information Age

– Under Incremental change: learning by followi

ng preceder’s footsteps

– Under discontinuous change: learning becomes

the voyage of exploration, of questing and expe

rimenting

Teaching Portfolio

• Can be seen as a voyage of exploration, of questin

g and experimenting

• Definition by Wolf & Dietz (1998):

“A teaching portfolio is a structured collection of tea

cher and student work created across diverse conte

xts of time, framed by reflection and enriched thro

ugh collaboration, that has its ultimate aim the adv

ancement of teacher and student learning.”

Why portfolios are a good idea for PD (I)

• The learning portfolio documents

– not only the teaching experience

– but also, the reflection and enquiry which

produces professional learning

Why portfolios are a good idea for PD (II)

• Groundwater-Smith (1999): “… is underpinned by the idea of the scholarship of practice”

• Gibbons et al (1994): practitioners are “active agents in the definition and solution of problems as well as the evaluation of performance.”

• Grant & Huebner (1998): “… to have significant impact on the development of teacher knowledge and professional practice.”

• Frid & Reid (1999): “…(being) the formation of professional subject – a ‘teaching self’”

Purposes of Portfolios

• Learning portfolio: promote teacher reflection and

ownership over the learning process.

• Assessment portfolio: presents administrators with

information about a teacher’s effectiveness.

• Employment portfolio: provides prospective

employers with information about a teacher’s

suitability for a position.

Process of Portfolios

• It is an extended period of time rather than a specific event.

• There are different processes• To be successful, we need:

a good deal of structured support for the individuals concerned, such as:

– Teacher release time,

– Resources for participating in professional development activities,

– Timetable flexibility for participating meetings,

– Teachers’ willingness to collaborate on the purpose and the process of the portfolio

Content of Portfolios

• Different purposes and processes will require different kind of content

• More than a collection of artifacts for teaching or professional activities:– Retallick & Groundwater-Smith (1996): “…(to be) a compilati

on of evidence which demonstrates the acquisition, development and exercise of knowledge and skills in relation to your work practice.

– Wolf (1996): “… (to be) a set of accomplishments attained over an extended period

– Dietz (1993): “… to be dynamic and changing as the learner experiences discoveries that lead to new directions and activities”

Assessment of Portfolios

• Depending on the purposes,

• For self-reflective professional learning

– Informal assessment (such as self assessment,

Peer evaluation process)

• For other purposes (such as for promotion,

employment):

– Formal assessment (also known as high-stakes

assessment): a higher level of monitoring

Two problem areas for assessment

• Comparability

• Subjectivity

Two problem areas for assessment (I)

• Comparability:

– Difficulty arises from the wide range of

approaches adopted by different persons as

each portfolio is on individual basis.

– Suggestion: stipulating certain core items to be

included

– Problem again: what are these items? Who

makes such decision?

Two problem areas for assessment (II)

• Subjectivity:

– As the evaluation is qualitative rather

quantitative, the question becomes how to make

the evaluation reliable and valid.

– Suggestions:

• A panel of assessors to integrate different views

and to avoid bias,

• A common understanding of the set of

assessment criteria used.

University accreditation: Formal assessment

• More universities are changing their policie

s on credit to allow students to submit a port

folio of their on-job learning for credit purp

oses

• This indicates an acceptance of the portfolio

approach

Summary of advantages by portfolios

• (by Edgerton, Hutchings & Quinlan, 1992)• capture the intellectual substance and ‘situatedness’

of teaching• encourage teachers to take important new roles in the

documentation, observation and review of teaching• …. to use these new roles as powerful tools for impr

ovement; and,• … are forging a new culture of professionalism abou

t teaching

Areas of tensions

• Multi-purpose nature: for self-reflection, for

assessment (comparability and subjectivity)

etc.

• Resources allocated: time allocated to the pr

ocess, teacher’s timetable flexibility etc

• Teacher’s willingness: may largely hinges o

n the above areas.

Local context

• The Hong Kong Institute of Education

conducted the study “Levels of Information

Technology (IT) competency, core course

elements and assessment tools for teacher

training in IT in Education” in 1999

One of the research questions in the study

What are the appropriate methods/tools to

assess the IT competency of teachers?

Recommendation

• It is recommended the portfolio be adopted

as a major form of assessment for teachers’

IT competency in education.

What are the issues raised in relation to this kind of school-based assessment?

Issues were raised in relation to school-based assessment

• Is there sufficient accountability in

school-based assessment?

• Are the principals in good positions to

assess the teachers in terms of using IT in

education?

Is there sufficient accountability in school-based assessment?

• The study suggested that principals are

ultimately accountable for whatever

happens in their schools. The attainment of

their colleagues in using IT as a teaching

tool should not be any different.

Are the principals in good positions to assess the teachers in terms of

using IT in education?

• The study suggested that– principals should be the first ones to attain good

proficiency in applying IT in learning and teaching.

– principals could delegate this job to someone else – a senior staff in the school or an external consultant

Content of portfolio - BIT

• a scheme of work consisting of a slide

show, Web materials to be downloaded

from the Internet as reference link, and the

use of at least one readily available

educational courseware

• rationale for using IT in the lessons: apply

what and how to apply

Content of portfolio – BIT (cont’d)

• a reflective statement of how they have

used IT in their duties as teachers

• self-assessment form completed by the

teachers

Assessment

Portfolio checklist

• to be used by the teachers for self-assessme

nt purposes - before and after the course

• for the school principals to assess the IT co

mpetency in education of their teachers

Other Forms of Assessment ?

• “Assessment by portfolio prepared by individual teachers is only one of possible forms of assessment. Instead of using assessment by portfolio, schools are free to use other forms such as lesson observations with assessment reports completed by school head, or group portfolio compiled by teachers each demonstrating his/her application of the required skills and knowledge in the teaching and learning processes. ”(extracted from the letter of 29.3.2000 to schools)

IT teacher training reference materials (ITERC)

http://www.ited.ed.gov.hk/English/ITinschool/Teacher_training/Teacher_Training.htm

Reflection/ Discussion

• What’s the situation in your school?

– Mode of IT training/ assessment

• Accountability?

• Principals to assess the teachers in terms of using IT

in education?

– Appraisal System