GEMS: GGEMS: Growth & rowth & EEffectivenessffectiveness MMeasures for easures for

SSchoolschools

State of IsraelState of IsraelMinistry of Education Culture & SportMinistry of Education Culture & Sport

Office of the Director GeneralOffice of the Director GeneralDivision of Evaluation & MeasurementDivision of Evaluation & Measurement

Formative Assessment for Formative Assessment for SchoolsSchools

2

Gems is an aid to management for school principal and teachers

Gems’ major purpose is to aid school staff in making data-based decisions, in order to use resources wisely and to improve school functioning

Gems is a school oriented assessment elucidating the strengths and weaknesses of the school and thus enabling it to allocate resources where most needed

Gems perceives the school as an holistic system

3

Achievements

Learning environment

Teacher-student relations

Staffdevelopment

Curriculum

Teacher-Parents

relations

Educational plansTeacher-Teacher

relations

Organization

4

Gems indicators were based on requests by 200 school principals and 200 supervisors when asked what reliable information they need in order to improve school functioning

Gems includes the following indicators: Pedagogical Environment

Action Plan School priorities and staff consensus Teaching resources & teaching methods Weak students and the help they get

Academic Achievements in 4 basic subjects (Language, Mathematics, Science, English)

School Climate and Work Environment

5

GEMS is administered each year in 50% of elementary and junior high schools, so that each school participates in GEMS once every two years

GEMS sources of information: Tests for 5th and 8th grades in Language,

Math, Science and English Questionnaires for all students in 5th through

9th grades dealing with school climate, pedagogical environment etc.

Interviews with all principals and teachers dealing with work environment, teaching methods, staff development etc.

Gems is administered at the beginning of the school year so that schools may get the report during that same year, and shorten the possibly stressful period of anticipation of the assessment

6

GEMS is school oriented. Its main goal is not to

provide a national “state of the union message”,

but rather to enable schools themselves to plan

rationally and to follow up, over time, the fruits

of their efforts

However, the data also serve the staff of the

Ministry of Education and its units. Thus, over 20

different aggregate reports are prepared each

year to help policy makers in different branches

of the Ministry to make data-based decisions

7

The rationale of GEMS is that a necessary condition for improving the educational system is to improve the interaction between teacher and students on the class level

Such an improvement is depends on schools basing their plans and practices on continuous, accurate, reliable, longitudinal data

In order to plan and make data-based decisions schools have to take the following steps: to set specific goals to know were they are to take actions directed to the goals

GEMS is a tool facilitating these steps

8

Setting Specific GoalsSetting Specific Goals

9

Goals have to be specific in order to be measurable and to avoid contradiction between goals

For example if the goal is “enhancing academic achievements” assessment must specify at least 4 indicators: Average grade percent of failures percent of distinctions difference in grades between students from

deprived SES and those from high SES As shown below these are not

necessarily identical:

10

“enhancing academic achievements” -- 1st measurement

School ASchool BSchool C

Average606060

S.D999

% failure33%33%33%

% excellent0%0%0%

“enhancing academic achievements” -- 2nd measurement

School ASchool BSchool C

Average676767

S.D23114

% failure56%11%0%

% excellent33%0%0%

Goal set for schools: raise average by 10%Goal set for schools: raise average by 10%

11

How schools operated to reach the goal?How schools operated to reach the goal?Does assessment distinguish between themDoes assessment distinguish between them??

Year 13 schools

School A Year later

School B Year later

School CYear later

Student 1701008075

Student 270958070

Student 370958070

Student 460606565

Student 560506565

Student 660506065

Student 750506065

Student 850506065

Student 950505060

12

““Without data you are just another Without data you are just another person with an opinion”person with an opinion”

Knowing where we areKnowing where we are

AA few examples of data for a few examples of data for a specific school specific school

13

School priorities: School priorities: consensus?consensus? School Example

Principal:

Achievements Teachers:

language skills school climate violence prevention Math Arts school climate English Respect for teachers Keeping school clean Independent study Parental participation “Closing the Gap” language skills Don’t know

41%

8%

51%

74%

7%19%

0%

20%

40%

60%

80%

100%

noconsensusamong staff

staffconsensus-

not withprincipal

consensusamong staff& principal

elementary schools junior high schools

National data

14

Weak studentsWeak students

31%25%

16%

0%

20%

40%

60%

80%

% failedGEMStests

% WS byteachers

report

% WS byschoolofficialreport

school national mean

Teachers: reasons forTeachers: reasons for learning difficultieslearning difficultiesweak studentsweak students

41%

17%14%

0%

20%

40%

60%

80%

low learningability

deprivedSES

poorpractice by

school

school national mean

15

Students’ AchievementsStudents’ Achievements

57 6350

5867

75 7261

0

20

40

60

80

100

Hebrew Math Science English

Me

an

Sc

ore

2002

2004

16

In order to evaluate the changes in students'

achievements, the following analysis was undertaken:

comparing the change in a given school (over the two

years) to the change found in a Comparison Group,

consisting of schools with same background

characteristics

and same starting point in 2002.

The school may thus draw conclusions such as:

Size of change

in CG

Size of change

in school

Grades

2004

Grades

2002

9+3+7471Hebrew

8+9+7465Math

8+5-6267Science

6+13+7663English

Achievements improved more in the school than in

comparable schools

The change in the school is similar to the change in the

comparison group

Achievements in the school show less improvement than

that in other, similar, schools

Assessing Change in 2004Assessing Change in 2004

17

Percent of students that didn't meet Percent of students that didn't meet standards standards

23%18%19%21%

43%

61%

30%40%

0%

20%

40%

60%

80%

100%

Hebrew Math Science English

Mea

n S

core

20% to 25% of students have private tutor paid by parents

2002

2004

18

Achievements by SES -- 2004Achievements by SES -- 2004

6661

76 79

5950

6658

0

20

40

60

80

100

120

Hebrew Math Science English

Me

an

Sc

ore

High SES Low SES

The gap in achievements between the two groups is ½ to 2/3 standard deviation

19

Teaching methods and Homework -- Teaching methods and Homework -- 20042004

10% of the teachers coordinate amount of homework

Students are required to prepare 25 minutes homework for each hour in school

Homework is mainly drill of material taught in class

10% of the teachers require learning new material or looking for sources

60% of teaching time is frontal teaching

20

Student perception of teachers’ attitudesStudent perception of teachers’ attitudes

41% 44%31% 34% 30% 34% 35% 41%

60% 63%

0%

20%

40%

60%

80%

100%

Teachersbelieve instudentssuccess

Student-teacher relationsare good

Teachershelp &

support

There areteachers

who insultstudents

Studentslike

school

2002

2004

21

Teachers’ perception of the work Teachers’ perception of the work environmentenvironment

82%80% 81%78%

50%37%

95%81%

38%

18%

0%

20%

40%

60%

80%

100%

The principalhas pedago-

gical standing

Theschool iswell run

Teachersare partnersin decision

making

Teamsenhance

professionaldevelopment

Teachersfeel

burnout

2002

2004

22

Taking actions to Taking actions to

reach goals: Action Planreach goals: Action Plan

We are free to choose our actions, . . . but we are not We are free to choose our actions, . . . but we are not

free to choose the consequences of these actionsfree to choose the consequences of these actions ..Stephen R. Covey (1996)Stephen R. Covey (1996)

23

To define To define specific indicatorsspecific indicators and to and to diagnose school relative to goalsdiagnose school relative to goals

indicatorsindicators derivedderived

from schoolfrom schoolgoalsgoals

indicatorsindicatorsderivedderived

from from previouspreviousdatadata

?? ??What to What to diagnosdiagnos

ee??

24

To analyze To analyze GEMSGEMS data on two data on two levelslevels

SCHOOLSCHOOLCLASSCLASS

?? ??GEMSGEMS

25

To ask questionsTo ask questions

Every school supervisor, principal, teacher should have a dialogue with available data (from GEMS and from school internal evaluations) and to ask himself: Did I know… Did I ask myself… Did I check…

26

To plan according to dataTo plan according to dataAn example: what should be done in classes with different An example: what should be done in classes with different

compositioncomposition

0%

10%

20%

30%

40%

50%

Do notmeet

standards

meetstandards

proficient

0%

10%

20%

30%

40%

50%

Do notmeet

standards

meetstandards

Proficient

Class AClass B

27

Prepare school action plan Prepare school action plan

and define (and define (specificallyspecifically))

where am I and where I want where am I and where I want

to be…to be…

28

evaluati

evaluati

ngng

PlanninPlannin

gg

Work

ing

Work

ing

29

Feedback to assessment from 1,036 principals: Feedback to assessment from 1,036 principals: 20042004

89%

78% 76%72%

65%

50%

0%

20%

40%

60%

80%

100%

The reportwas clear

Helped insetting

priorities

Was usefulas

managerialtool

Coveredimportantindicators

Helped inpooling effort& resources

I learned newthings aboutmy school