study of comparison of performance of year ...pills : pacific island literacy levels (niveaux de...

STUDY OF COMPARISON OF PERFORMANCE

OF YEAR 6 AND YEAR 10 STUDENTS

IN ANGLOPHONE AND FRANCOPHONE SCHOOLS

IN VANUATU

BY

ROY CARR-HILL1

ALETTA GRISAY2

(Study financed by UK Department for International Development and the French Government,

carried out in collaboration with the National Examinations Office)

1. Institute of Education, London, UK December 1998

2. University of Liege, Belgium

Vanuatu/1999/translate 15/01/99 15:51 1

ABRÉVIATIONS USED IN THIS REPORT:

BETC : Basic Education Training Centre (CFBT Centre de Formation pour l’Éducation de

Base).

DFID : Department for International Development (Département pour le développement

international).

IALS : International Adult Literacy Study (Etude internationale sur la littératie des adultes).

IEA : International Association for the Evaluation of Educational Achievement

(Association internationale pour l’Evaluation du rendement scolaire).

IRT : Item Response Theory (Théorie de la réponse aux items).

MCQ : Multiple Choice Questions (QCM Questions à Choix Multiple).

NEO : National Examinations Office (BNE : Bureau National des Examens).

PILLS : Pacific Island Literacy Levels (Niveaux de compétence en lecture des îles du

Pacifique).


Prologue

This is the final report of the Study comparing the performance of students in Years 6 and

10 in Anglophone and Francophone schools in Vanuatu. The original text of the report was

presented earlier this year to a workshop held in the USP campus on Monday 8th March

1999. As a result of the discussion at that workshop and subsequently, we have made a

small number of changes to the report; and included a number of appendices: the first

includes the text of the speech made by the Minister at the opening of the workshop; the

second an edited version of the questions that were put to us and of our replies (including

where appropriate additional material); and the third includes reactions from two sources: a

follow-up meeting held at the Curriculum Development Centre; and a statement from the

French Cultural Attache.


BACKGROUND

1. A distinctive characteristic of the Vanuatu educational system, reflecting the

country’s colonial history is that there are two teaching mediums (in English and in

French). The duality is not simply linguistic: the traditions, the “school culture”, teaching

methods, differ in the two systems: moreover, the apparent disparities in performance

between students in anglophone and francophone students, revealed by the results of the

national examinations at Year 10, has been a source of concern for the government, the

teaching profession and the parents.

2. Are these disparities simply a consequence of the different levels of difficulty of the

tests taken by the two groups of students, or of differential levels of severity used when

correcting by the two sets of markers? Or are they a reflection of a real difference in the

performance of the two populations? If there are real differences in performance, can they

be attributed to the characteristics of the population over which the educational systems has

little or no control - for example, more deprived social background in one of the groups?

Or again, should one point to school based factors - such as, less effective pedagogy in the

schools attended by one group of students?

3. The current study was carried out in order to provide (some) answers to these

various questions - which are important in order to evaluate the equity of the education

system - at the request of the Ministry of Education in Vanuatu, with funds provided by

DFID and the French Government.

4. The study, carried out in 1997 and 1998, was directed by two bilingual independent

consultants in close collaboration with the Vanuatu National Examinations Office who

have throughout provided extremely effective support. The study had several components:

• examination of available documents (both current and historical) about the Vanuatu

school system;

• interviews with the principal actors of the educational system;

• visits to numerous primary and secondary schools, both in Efate and other islands,

observing lessons and interviewing the teaching staff.


• on the basis of information collected in these ways, establishing a series of working

hypothesis which might explain the observed differences

• systematic collection and analysis of objective data in order to test these hypotheses.

This report briefly describes these working hypotheses, the data that was collected to test

the hypotheses, and the principal conclusions of the study. More detail will be provided in

the follow-up technical report.

Working Hypotheses

5. On average, the results of pupils in francophone schools on the standardised Pacific

Islands Literacy Levels tests taken at the end of primary cycle are not as good as those of

pupils in anglophone schools. The same differences are observed with the national exams

taken at the end of the primary cycle (at the end of Year 6) and during secondary cycle (at

the end of Year 10). The differences could have several explanations.

Hypothesis One: Bias Linked to the Exams Themselves

6. It could be that the tests given to the two groups of pupils are not of the same

difficulty either because of:

H.1.1. their content;

H.1.2 poor translation of some of the tests;

H.1.3 differential familiarity of the students with format of tests items

H.1.4 cultural biases;

H.1.5 differential severity of the marking schemes and procedures used by

examiners.

Hypothesis Two: Differences Linked to the Social Origin of Students or their Initial

Aptitudes.


7. It could be that the populations going to the francophone schools are different from

those in anglophone schools because the students are:

H.2.1 from a more deprived social origin;

H.2.2 from a family environment less conducive to learning whether because

of linguistic characteristics or because of cultural resources;

H.2.3 (for secondary students) subject to less selection at the transition from

Year 6 to Year 10, which would have enabled access to secondary school for

a group of students who were slightly weaker than the average students

promoted in the anglophone group.

Hypothesis Three: Differences Linked to the Quality of Teaching Delivered

8. It could be that the handicap of francophone schools in comparison to the

anglophone schools is due to:

• poorer material resources;

• less effective teaching practices;

• less opportunity to learn the material which is tested in the examinations;

• less qualified or less well prepared teachers.

Data Collection Procedures

9. In order to examine these hypotheses, information was required on the following:

(a) the examination tests taken by students and their psychometric qualities;

(b) the real performance of students, measured by independent and objective tests which are

of rigorously comparable difficulty;

(c) the characteristics of students in the two groups, and of the learning environment they

have experienced both in the family and at school;


(d) the relation between the content of the tests they have taken and the material they have

learnt (opportunity to learn);

(e) the characteristics of the teaching staff;

(f) (for students in Year 10) the results they obtained when they took their Year 6

examination.

1. Population

10. Given the relatively small school population in the country, it was judged preferable

to carry out an exhaustive study (involving all the schools in the country and all the

students in Years 6 and 10 in those schools) rather than a sample survey. Data was

therefore collected for two groups of students finishing in 1997, respectively, Year 6 (about

4,800 pupils) and Year 10 (about 1,150 students) and involved all schools in the country

which included either a Year 6 or Year 10 class.

2. Instruments

A Examinations

11. Thanks to the well ordered archives of the national Examinations Office, and the co-

operation of the resident DFID personnel, the consultants were able to obtain a sample of

the tests taken by pupils/students at the end of Year 6 and Year 10 between 1991 and 1996,

as well as the PILLs tests taken during the primary cycle (in both Years 4 and 6). The

examiners in charge of marking the national examinations for the 1997 cohort (which was

the target year of the study), agreed to carry out the marking on an item-by-item basis for

the examination for language and for mathematics in both Year 6 and Year 10 and for

sciences in Year 10, in order to permit a more detailed statistical analysis of the results of

these tests and to relate them to independent measures of student achievement. Finally, for

the group of students who took the Year 10 examination in 1997, it proved possible to

extract from the archives, their score on the examination taken by the same student four


years earlier, at the end of Year 6, in order to study the possibility that there was an effect

due to the relative level of selectivity imposed by these tests.

B. Tests of Achievement

12. It was important, in the context of this study, that the instruments used to measure

the achievement of the anglophone and francophone populations were strictly comparable

and appropriate for the age and educational/school environment of students in Vanuatu.

We obtained authorisation from the International Association for the Evaluation of

Educational Achievement (IEA), to use some of the tests developed recently as part of the

large scale international studies on reading (IEA/Reading Literacy Study 1991) and for

mathematics and sciences (IEA/Third International Maths and Science Study, 1995).

13. The comparability of the English and French version of the original tests

instruments had been carefully verified in order that they could be used in those

international studies. A national (Vanuatu) committee of teachers and examination

moderators including both anglophone and francophone personnel was asked to select,

from among the IEA test batteries available, a sub-sample of items which they considered

most appropriate given the culture and curricula in Vanuatu. A small number of minor

corrections have also been made to the items retained for use in order to adapt them to the

Vanuatu situation.

14. The tests developed as a result of this process include:

• for Year 6 - a reading test (54 items);

• for Year 10 - a reading test (41 items), a mathematics test (31 items), and a science test

(31 items).

15. We also obtained authorisation to use some of the tests from the International Adult

Literacy Study (IALS, STATCAN/OECD, 1995) in order to evaluate the reading levels (30

items) and capacity to manipulate numerical data (13 items) of the trainee student teachers

at the Basic Education Training Centre (BETC) in Port Vila. Like the IEA test instruments,

the IALS instruments already existed in English and French versions, the equivalence of


the versions having been strictly verified when they were used in the previous international

study. The data available concern 108 future teachers (42 future anglophone primary

school teachers, 36 future francophone primary school teachers, and 30 future francophone

secondary school teachers).

C. Assessment of Opportunity to Learn

16. Copies of the versions of the IEA tests in mathematics and science which were to be

used in Year 10 were given to the corresponding subject teachers in the schools where the

test was already administered. For each of the test items, these subject teachers were asked

to indicate, on a check list, whether the material evaluated by the test items had been

covered or not by their students.

D. Questionnaires

17. In order to collect the information on the pupils and students on their learning

environment and on the material and pedagogical resources at their school, the ‘Student’,

‘Teacher’ and ‘School’ questionnaires were developed in collaboration with the team at the

National Examinations Office. These questionnaires were administered to all students in

Year 6 and Year 10 who took the tests, to their teachers (in primary school, to the class

teacher; in secondary school to the teachers of English in the anglophone schools, teachers

of French in the francophone schools and to the teachers of mathematics and science in

both schools), as well as to the directors and principals of the primary and secondary

schools.

3. Administration and Marking

18. The various tests and questionnaires were piloted in May 1997, in order to check the

feasibility of the exercise. Based on this pilot, there were a few revisions and the definitive


version of the year 10 instruments were administered in July-August 1997 by the

Examination Office Team. The administration of the tests and questionnaires for Year 6

took place at the same time as the national examinations (end October 1997) and profited

from the infrastructure and staff who were organising and administering the national

examinations. Finally the IALS tests for teacher trainees were also administered in October

1997 by one of the consultants in collaboration with the personnel of the National

Examination Office. In all three cases, the test administrators received a guidebook in

order to standardise the procedures for taking the tests. Similarly, standard guidelines for

the tests were developed for coding and correcting the questionnaire and tests; the large

amount of work involved was managed and organised by a local assistant trained by the

consultants and appointed by DFID. The data were entered into computer files at the

University of York.

4. Methods of Analysis

19. For the analysis of the results of the IEA based tests and the various national

examinations, the analysis was based on Item Response Theory (IRT). This technique

allows one to scale both the difficulty of the test items and the proficiency of the groups of

students who took tests including the same items, even in the particular situation where

only some of the items had been administered to both groups. In the current study, the IEA

test items served as the ‘anchor’ group of items which was common to the two populations

which enabled us to estimate - at least for some of the tests (see below) - the relative

difficulty, for the whole population, of the items which were only taken by either the

anglophone or francophone group. In other words, the techniques makes it possible to

obtain an estimate of the difficulty which the anglophone student population would have

had if they had taken the francophone examination, and the difficulty the francophone

student population would have had if they had taken the anglophone examination.

20. The data collected through the questionnaire responses were analysed using

standard univariate and bivariate techniques, but certain groups of variables were also

manipulated so as to devise a series of composite indicators which would both be more

robust and more easily interpretable in the multivariate analysis.


21. The more detailed analysis of the various factors associated with differences in

student achievements, separating out effects associated with differences between students,

between the schools they attended and between the two teaching medium groups, was

carried out using multi-level statistical models (multivariate and multilevel regression).

These models are much closer to the hierarchical structure of the data being analysed than

the classic ordinary least squares regression analysis and have the advantage of providing a

more accurate estimate of the partitioning between the different variance components, as

well as more accurate estimation of the regression coefficients.

RESULTS

1. Differences in Achievement as Measured by Tests and Exams

A. Differences in the performace of the two groups of students on the

IEA tests

22. Both in Year 6 and in Year 10 the IEA tests show that students in the francophone

system do not achieve as well as students in the anglophone system

23. In Year 6, the anglophone students obtain significantly higher scores than the

francophone pupils in the IEA reading test. The difference is +0.420 of a standard

deviation (a statistical measure of the variation in the scores) which means that an

anglophone student getting the average score for her or his group has a result which is

better than about three quarters of the students in the francophone group.

24. In the three IEA tests given to students in Year 10, the anglophone students obtain

significantly better scores than the francophone students (Table 2). The differences are

particularly large in Reading and in Sciences, where the anglophone students on average

score more than one standard deviation higher than the francophone students in Reading


and three quarters of a standard deviation higher in Sciences. The francophone ‘handicap’

is smaller in Mathematics (about a quarter of a standard deviation)1.

Table 1: Year 6 - scores in the IEA reading test broken down by teaching medium

Teaching medium

Raw score(number of items answered

correctly out of 36)1

IRT score

Anglophone Francophone Altogether

N of students Mean Standard deviation Reliability N of students Mean Standard deviation Reliability N of students Mean Standard deviation Reliability

3194 24.6

6.8 0.85

1511 22.3 5.9

0.79

4705 23.85 6.60 0.84

3194 1.56

1.35 0.85

1511 1.03 0.96 0.79

4705 1.39 1.26 0.84

Note: A few of the IEA items were excluded from this analysis (as in the following analyses) because they did not provide any substantial discrimination in the two populations, or because they behave differently according to the teaching medium, which would have implied an eventual bias in favour of either the anglophone or the francophone students (i.e. those items are significantly easier or more difficult for one of the two groups of students, even for those students who perform equally well on the remaining items in the test).

1 The difference of a standard deviation in the reading score means that the average score of the anglophone group is better than about 85% of the scores of the francophone group; for the science and mathematics results, the average score in the anglophone group is better than about 75% and 60% respectively of the scores of the francophone group..

012-VanuatuReport-ENG 17/03/2003 7:57 10

25. For the different IEA tests, the IRT analysis shows that the hierarchy of difficulty of

items and their fit indices are very close in the two populations, which means that the

hypothesis that the content or the format of the items have any overall effect on the two

groups, cannot be sustained (Hypothesis One).

Table 2 Year 10: scores on IEA tests by teaching medium

Teaching Medium Reading Mathematics Science

Raw score

IRT score

Raw score

IRT score

Raw score

IRT score

Anglophone

Number of students

Mean

Standard Deviation

666

29.5

4.2

666

1.96

1.05

667

14.8

4.3

667

0.17

0.81

667

15.6

4.0

667

0.29

0.75

Francophone

Number of students

Mean

Standard Deviation

403

23.3

5.3

403

0.73

0.88

407

12.7

4.1

407

-0.22

0.80

407

11.4

3.5

407

-0.46

0.64

Together

Number of students

Mean

Standard Deviation

1069

27.2

5.5

1069

1.46

1.10

1074

14.0

4.3

1074

0.02

0.83

1074

14.0

4.4

1074

0.01

0.80

Reliability of the test 0.83 0.71 0.70

Difference ENG/FRA S (0.0001) S (0.0001) S (0.0002)

26. We have, however, tested the more specific hypothesis, sometimes proposed, that the

Francophone students might be disadvantaged because they are unfamiliar with multiple

choice questions (MCQ) and that they would be more at ease with open questions

(Hypothesis 1.3).

27. In general, with those who are not used to multiple choice questions, there is a

substantial level of ‘multiple’ answers (the student does not follow the instruction to only tick

the most appropriate of the four or five answers suggested). He (or she) ticks all those which

are, at least in part, appropriate answers. In this study, there were very few such ‘multiple’

responses, suggesting that the francophone students were not severely handicapped in this


way - even though it is true that the level of ‘multiple’ answers is slightly higher among the

francophone students as compared to their anglophone peers.

28. If one calculates, for each of the three tests, separately the partial scores only for the

multiple choice questions and only for the open questions, the differences observed with the

global scores is repeated for each of the partial scores. The differences are however,

significantly greater for the partial scores for multiple choice questions than for the partial

scores for open questions, which does tend to confirm that the multiple choice questions

impose a small handicap on the francophone students. Note however that this factor is in

no way sufficient on its own to account for the size of the francophone deficit. Even if based

only on open questions, the scores of francophone students are significantly inferior to those

of anglophone students, particularly in science and in reading.

B. Differences in Content between the Anglophone and Francophone National

Examinations.

29. The examination at the end of Year 6, which determines transition to the secondary

cycle, and the examination at the end of Year 10, which marks the end of schooling for the

majority of secondary school students, are the two major tests for the school career of the

students. Verifying that the selection, which is a consequence of those examinations, is

equitable (or, more precisely, assessing whether or not students of the same capacity are

likely to obtain similar scores, whichever their teaching medium) was one of the important

objectives of the study. From the point of view of the psychometric quality of the tests

making up the examinations, the conclusion has to be negative. The English and French

versions of the tests that we have examined are too far apart for us to consider that, overall,

the examinations are operating so as to rank the two groups of students in the same way.

30. The language tests both in Year 6 and Year 10 have not yet been unified. The

content of the language tests, which reflect these curricula, consequently differ very

significantly. It is true that real efforts have been made to ensure that the English and French

versions of the tests give the same weight for the various partial scores (for example, the two

versions of the tests give the same number of points for writing and for reading: the number


of items and the time allowed is about the same in the two versions). However, these

similarities are more apparent than real, as we can see from the language test in Year 10 in

1997 (Table 3).

Table 3: Content of Language Examination in Year 10

English Examination French Examination

Three tests of writing: a choice of four subjects (minimum 250 words); a precis/resume of one of the books read during the year; a letter. The three answer sheets given to the student each have 60 lines

One writing test: a choice of three subjects (250 words). The one answer sheet given to the student has 28 lines

A cloze test (ten items: principally articles, prepositions, etc. )

A test of grammar, 30 items (conjugation, grammatical agreement, phrase structure)

A comprehension test: text about 900 words; 15 multiple choice questions; 5 open ended questions. Two of these 20 questions refer to vocabulary.

A comprehension test: text of about 900 words; 10 multiple choice questions, 5 open ended. Six of these 15 questions are on vocabulary; one on grammar

31. As one can see, the two tests are not equivalent in terms of the type of competence

that they are principally testing: the English version emphasises functional skills (written

expression and comprehension): the French version gives a more important role to formal

competencies (grammar, vocabulary).

32. The same is true, to a lesser extent, for the mathematics examination in Year 6:

although the two versions of the test do include nearly ten items with the same content,

the wording of the items given to the student is often substantially different.


33. In contrast, the mathematical and science exams of Year 10 are entirely parallel

in terms of the content of the items - the curriculum having been unified for several years

now. The test items are generally of excellent quality and their translation from one language

to another cannot be faulted. In mathematics, a few differences can be observed in respect of

the formulation of the questions and the prompts used to obtain replies (the wording of the

items to the student tends to be formulated slightly more precisely and rigorously in the

French version).

34. Both for the English and French versions of the tests (and for whichever subject), the

correction procedures for open-ended questions do not seem to be sufficiently

standardised. One of the most problematic aspects is the coding of partially correct

responses. Several of the francophone markers appear to operate a very analytic correction

procedure, giving half or quarter points to responses for which the official marking scheme

only allows for awarding whole number scores (one, two or three points). This procedure

also occur, but less frequently, with the English markers. It has a number of disadvantages:

• firstly, there are no strict guidelines for the procedure (which means that the frequency

with which these decimal scores are used, vary significantly from one marker to another).

• secondly, and for the same reason, there is no guarantee that the same incomplete or

partially correct response given by two different students will receive the same mark.

• finally, it makes the aggregation of the scores more complex (which multiplies the risk of

error when making a manual calculation of total scores). For certain tests we have been

able to recalculate by computer the total of the points obtained by the student and to

compare this to the global score calculated by the marker. The number of discordant cases

is far from negligible: it is rarely less than 7-8% and in some cases was as high as 15-

20%, which seems too high for examinations of this importance.2

C. Statistical Analysis of the National Exam Results

2 Note however that these error rates are over estimated, because they include also the errors due to data entry. These latter errors are however obviously not relevant to the examination mark actually received by the student because that was calculated manually.


35. The scaling of the various tests making up the examinations using IRT was made

more difficult because of the inconsistencies in the content and marking procedures which we

have just described. Whilst the IRT models themselves are robust, they can only be used

reliably when the raw data meet several basic requirements: on the one hand, the two tests

that one wants to calibrate on the same scale, should be measuring the same latent

competence or trait; on the other hand, the same score awarded to the response of two

students for the same item should, in principle, signify exactly the same thing. We have seen

that the raw data do not strictly conform to either one of these two requirements, which

necessarily limits (sometimes severely) our ability to make a comparison.

36. For the language examinations both in year 6 and Year 10, it did not prove possible

to scale the totality of the questions included in the English and French versions of the tests:

the clearly multidimensional character of the competencies tested by each of these

examinations made it too difficult to identify a common latent trait. We have therefore had to

drop from the analysis all items concerning written expression as well as the majority of the

cloze items (English examinations), grammar (French examinations), and of vocabulary (IEA

primary tests), only retaining those items measuring reading comprehension. The

comparison presented below therefore only concerns that part of the national language

examination which measures the student competencies in the two teaching mediums in

reading, and not on the other skills measured by the national exams.

37. For all the items with a decimal code (which means in practice for all of the open

questions in the tests in language, mathematics and in science) we had to recode by

computer the replies in order to make them more coherent and manageable in the analytic

framework. This means that the comparisons presented below are not based exactly on the

way in which the students scripts were corrected by their markers, but rather on the basis of

what can be considered to be a computerised rationalisation of the marking procedures

which were actually used in the field.

38. For the mathematics exams in year 6, we did not have available the item-by-item

results of the PILLs tests, which would have provided us with a common anchor point for the

two populations. The few items which were common to the English and French version of

the examination proved to be insufficiently stable to provide this anchor point. It was not


therefore possible to generate a common scale for the English and French versions of the

mathematics examinations for Year 6.

39. Taking account of these restrictions, the analysis presented in Table 4 below are much

too partial to provide a firm response to the question: are the national examinations of

similar difficulty or do they include a bias in favour of one or other of the two linguistic

groups? They do however permit us to draw two important conclusions:

• the data on examination tests which are based on unified curriculums and where the

content is equivalent (the mathematics and science examinations of Year 10) confirm

the significant superiority of the anglophone students over the francophone students.

• both in Year 6 and in Year 10, the reading comprehension section of the language

tests (the only section which could be modelled into a common scale for the two

populations) similarly confirms that the anglophone group has better results.

40. Each of the scores in Table 4 has been calculated by combining into a single scale all

of the items available for the particular subject for that Year group (items from the IEA test,

from the National Anglophone test and the National Francophone test) as if we were

considering one ‘unique’ test taken by the whole of population (where the anglophone

students had simply omitted to reply to the items on the francophone examination, and the

francophone students had omitted to reply to the items on the anglophone examination). The

metric of the scale is given by the ‘anchor’ items from the IEA tests which were common to

the two populations. Both for the IEA test and for the two examinations, the only items

included in calculating the students score where those with satisfactory psychometric

properties (i.e: measuring the same competence for the whole population, and presenting no

significant bias in favour of one or other of the two teaching medium groups).


Table 4 Examination scores callibrated on the basis of IEA anchor items

Year 6 IRT Reading

Score1 English

Number of students

Mean

Standard Deviation

3040

0.73

1.63 French

Number of students

Mean

Standard Deviation

1555

0.14

1.16 Altogether

Number of students

Mean

Standard Deviation

4595

0.56

1.53 Fidélité 0.92

Year 10 IRT Reading Score2

IRT Maths Score3

IRT Science Score4

Anglophile Number of students

Mean

Standard Deviation

683

1.87

0.82

699

0.16

1.19

700

0.38

0.78 Francophone Number of students

Mean

Standard Deviation

429

0.71

0.77

438

-0.43

0.95

437

-0.61

-0.63 Altogether

Number of students

Mean

Standard Deviation

1112

1.44

0.98

1137

-0.07

1.14

1137

0.00

0.87

Reliability of the test 0.88 0.86 0.84

Difference ENG/FRA S (0.0001) S (0.0001) S (0.0002)

Notes 1. Reading Score Year 6 based on 27 items IEA; 26 items ENG; 37 items FRA 2. Reading Score Year 10 based on 38 items IEA; 33 items ENG; 29 items FRA 3. Mathematics Score Year 10 based on 11 items IEA; 31 common ENG/FRE items 4. Science Score Year 10 based on 13 items IEA; 37 common ENG/FRE items.


41. The examination scores calibrated in this way can therefore be considered ‘equitable’

(or ‘non-biased’), in the sense that a francophone student and an anglophone student with the

same level of competence would have obtained the same score: that is, they would have the

same statistical probability of giving the right (or wrong) answer to items of equivalent

difficulty included in the tests which each of them had taken.

42. The differences estimated by this method are of the same order of magnitude as those

suggested by the IEA test on its own: a little less than half a standard deviation in language

in Year 6; more than a standard deviation of difference in language and in sciences in Year

10. Mathematics in Year 10, where the difference is about half a standard deviation, is the

area where the handicap of the francophone group of students at the end of secondary cycle,

whilst still very significant, is the smallest.

Conclusions about the Examinations

43. It appears therefore that the handicap of students in francophone schools is not a

simple artefact linked to the content of examinations, to the way in which the questions were

posed, or to the marking schemes and procedures used. These various factors mean that it is

not easy to compare directly the results of the examinations taken by the two groups (this is

particularly true in respect of the language tests, but also, to a lesser extent, for the other

tests). However, the performance of the students in the francophone groups remains

significantly weaker when they are assessed using rigorously parallel instruments or when

more detailed statistical treatments are applied in order to calibrate the examination scores

on the same scale. The size of the observed differences is pedagogically worrying. A

‘handicap’ of the size of a standard deviation (which were observed for both reading and for

sciences in Year 10) reflects a considerably slower rate of learning, probably equivalent to at

least a school year, and possibly more.

44. In respect of the functioning of the national exams themselves, the analysis has

demonstrated the good psychometric properties and comparability between the two versions

of the mathematics and science tests in Year 10 - although this is a little compromised by the

fact that the marking schemes for the open ended questions are not sufficiently standardised.


The language tests suffer from the same marking problems (even more severely in the sense

that there are many more open ended questions), but also have differences of content and of

design which makes their equivalence entirely problematic

2. Differences in the characteristics of the populations

2.1 There are only small differences between the anglophone and francophone students

in terms of socio-economic background, although the latter tend to come from a more

rural environment.

45. On the whole, the two populations appear to be very similar (as we would expect given

the historical pattern of growth of the Vanuatu school system). The differences that are

observed are relatively small (see Table 5). Thus, whilst the numbers of boys and girls are

similar in Year 6, there is a slightly higher ratio of boys to girls in the anglophone secondary

system. Whilst the proportion of Year 6 students boarding at school is the same in both

systems (at 15%), nearly three quarters of Year 10 students in the anglophone system board

at school compared to under half of those in the francophone system.

46. About 30% of Year 6 students had access to water from a tap inside the house, although

the percentage was higher (at 43%) among secondary school students which is not

unsurprising as they are an elite population. However substantial numbers were also likely

to fetch water from a river or spring and this was more likely among students in the

francophone system, suggesting they are from a rural environment. Similarly, the proportion

eating a traditional breakfast (banana, yam or manioc) were all greater among students in the

francophone system (although the differences were very small among Year 6 students).


Table 5: Comparisons between Anglophone and Francophone populations in respect of selected characteristics and rank correlation of characteristics with Year 10 Fair IRT scores Distribution of the

characteristic in Rank correlation of the variable with the IRT scores in …

Year 6 Year 10 Year 6 Year 10 Anglo Franco Anglo Franco Lang. Lang. Maths Scien. Boys =0 Girls = 1 50.5 49.5 46.7 50.6 -.08 +.07 -.08 -.13Water from river or spring? 42.8 47.1 23.8 38.9 -.21 -.22 -.09 -.12Cook on gas? 35.8 35.6 58.4 53.0 -.02 +.27 +.14 +.14Eat ‘traditional’ foods? 53.5 41.6 16.1 31.9 -.23 -.19 -.16 -.16 Books at home? 80.1 50.5 82.5 74.1 +.31 +.31 +.19 +.21Parents to primary school? 74.2 58.2 86.6 80.7 +.22 +.14 +.11 +.04Active Reading? na na 81.2 43.9 na +.23 +.15 +.16Uses Teaching Medium 32.4 24.5 72.3 61.3 +.19 +.13 +.08 +.12 Textbooks? (2+) 31.6 35.5 52.1 25.9 +.37 +.32 +.38Separate Exercise Books 58.2 49.6 75.5 76.6 +.01 -.04 -.03 -.13Use ringbinder? na na 11.6 86.4 na -.50 -.29 -.55Own calculator and dictionary? 58.2 45.0 59.1 26.9 +.16 +.33 +.24 +.28Own geometrical insts?(2+) 50.0 71.9 38.8 78.2 +.06 -.32 -.15 -.26 Bored in language lessons? na na 71.7 49.7 na +.16 +.04 +.12(Most) Teacher(s) corrects work 29.7 55.6 46,6 21.4 -.01 +.19 +.03 +.22(Most) Teacher(s) has favourites 40.7 43.3 51,5 66.9 -.07 +.11 +.13 +.13Likes subjects? 93.5 85.7 37.4 19.3 +.01 +.18 +.22 +.24Student pays for materials etc.? 12.8 27.9 29.1 73.6 -.06 -.26 -.11 -.29

47. Slightly more of the Year 10 anglophone students have a father who ‘works for the

government’ or ‘does some other paid job’.

2.2 The francophone students appear to be significantly handicapped in terms of their

linguistic background

48. When asked what language they usually spoke, whilst nearly half Year 6 students report

speaking their own language with their friends outside school, and over three quarters with

their families, nearly one in three say that they speak Bislama as the lingua franca with their

friends outside class (and one in six with their families). Although the pattern of languages


spoken, of course, varies across the archipelago (see Annexe), in every region there is a

minority who are using Bislama as a lingua franca with their friends and are not using their

‘own’ language at home. Among Year 10 students the tendency is even more marked with

three quarters of students saying that they speak Bislama as the lingua franca with their

friends outside school, although nearly the same proportion also say that they speak their own

language with their family. Although there are few differences in respect of the usual

language spoken, when asked whether or not they often spoke Bislama in different situations,

it was clear that Bislama is much more a lingua franca with Year 10 anglophone students and

their families than for francophone students and their families.

2.3. The available data on the Year 6 examinations passed in 1993 by the students

enrolled in Year 10 in 1997 confirm that francophone students were subject to less

selection at the end of Year 6 (Hypothesis 2.3)

49. Using the student’s scores on the 1993 Year 6 examinations, we have compared

the selectivity of francophone and anglophone educational systems with regard to the access

of students to the secondary schools (see Table 6). Although the scores are on totally

different tests, in terms of the proportions who are chosen, it is clear that the Anglophone

system is more ‘selective’ (about 15% of the anglophone students, compared to over 35% of

the francophone, were accepted in secondary schools).

Table 6. Scores of current year 10 students when they took the ‘Selection’ Exam in year 6 (in 1993) compared to the scores of all Year 6 students taking the examination in 1993 Anglophone Francophone

All N = 3236

Selected N = 495

All N = 1540

Selected N = 382

Language M Range 0-76 SD

Mathematics M Range 0-49 SD General Studies M Range 0-47 SD

37.3 17.3 14.7 10.0 27.1 10.5

53.8 15.8 26.1 8.4 36.5 5.3

44.0 17.6 21.6 10.5 20.2 9.2

56.9 10.6 30.5 7.9 27.8 6.8


50. In terms of their marks in the Year 6 examination relative to the populations from which

they were drawn, there is a difference in that the mean scores of those selected tend to be

relatively higher than the overall ‘parent’ means among the Anglophone than among the

francophone but it is not strong.

51. Indeed, compared to what is usually observed in similar studies of scores of the same

pupils separated by a few years, the correlations between the Year 6 marks and Year 10 IRT

scores are low, especially for francophone students (see Table 7). This observation, for

which there is no obvious explanation, should be investigated further, especially as it leads

one to query the predictive value of the assessment at the end of Year 6 and thence it’s

usefulness as a means of selection of candidates for secondary school. In the francophone

group in particular, the weakness of the correlations, suggests that there is very little

continuity of the teaching-learning processes between primary and secondary, or that the

assessment instruments do not correspond to what has been taught and learnt.

Table 7 Correlation between scores of Year 10 students in the ‘selection exam in Year 6 (in 1993) and their score in the Year 10 exam Year 6 Exam Score Year 10 IRT Scores Anglophone Francophone Lang Maths Science Lang Maths Science Language Maths General Knowledge

.53

.37

.43

.46

.63

.37

.36

.39

.38

.31

.24

.21

.06

.31

.12

.08

.17

.09

Conclusions about the characteristics of the two groups of students

52. There are only small differences between the anglophone and francophone students in

terms of socio-economic background although, among Year 10 students, those in the

francophone system tend to come from a more rural environment. Year 6 and Year 10

students in the francophone system tended to come from more deprived cultural home

environments although they tended to be helped more often with their homework. The

development of Bislama as a lingua franca amongst adolescents is perhaps an obvious

reflection of what they hear being spoken among adults in the wider community, but almost


certainly benefits students in the anglophone system, given that their teaching medium is

closer to Bislama. Finally, it is clear that the transition from primary to secondary in the

anglophone system is more ‘selective’; although in the absence of detailed comparison of the

examinations used in 1993, it is difficult to say whether or not this means that the students

entering the secondary system in the francophone system are less able. This uncertainty is

compounded by the low correlations observed between the scores of the Year 10 students in

1997 and the marks they obtained when they took the Year 6 examination in 1993.

3. Differences in the school environment

3.1. Less material resources (especially textbooks) are available in the francophone

schools (Hypothesis H3.1)

53. Among Year 6 primary students, in fact slightly more francophone than anglophone

students had their own language and mathematics textbooks. However, among Year 10

students, in each subject, significantly more students had their own textbook in the

anglophone than in the francophone system (53%, 59% and 58% compared to 48%, 43% and

27% respectively) with the greatest gaps in science; and, in contrast, many more students in

the francophone system reported that there was only one or a few copies per class (9%, 14%

and 12% for language, mathematics and science), this was the case for many more students in

the francophone system (28%, 43% and 54% respectively). There is clearly a substantial

disadvantage here even assuming the textbooks are equally appropriate.

54. More than half of anglophone students in Year 6 compared to slightly less than

half of francophone students reported separate exercise books for different subjects,

compared to three quarters of both groups in secondary. However among Year 10 students,

whilst very few of those in the anglophone system reported using a ring binder, this was true

of most in the francophone system. Among both Year 6 and Year 10 populations, students in


the francophone system were more likely to have geometrical instruments (compass,

protractor…) whilst those in the anglophone system were more likely to have a calculator

and dictionary.

55. At primary level, nearly a third of those in the francophone system said they paid for

materials compared to one in eight of those in the anglophone system. Similarly, at

secondary level, whilst three quarters of those in the francophone system said they paid for all

their school materials, this was true for less than a third of those in the anglophone system.

56. At primary, more teachers in the anglophone system (53/124) reported shortfalls in basic

teaching materials compared to the francophone system (17/57). However, there are fewer

cases of severe shortage in that more teachers in the anglophone system said that there was

at least one textbook between two students. In contrast, at secondary level, more teachers in

the francophone system (10/36) reported shortfalls in basic teaching materials compared to

six out of 70 in the anglophone system, and more said that they only had a few books in the

whole class. More teachers in the francophone schools gave out lecture notes in the form of

photocopies or stencils and many more asked students to copy lessons from the blackboard.

3.2. Most of the differences evidenced in the instructional practices tend to confirm

Hypothesis H3.2 (better pedagogy in anglophone schools).

57. On the basis of our observations when visiting schools in Vanuatu, we expected to find

more differences in the school environment than what we were able to substantiate through

the students and teachers questionnaires. For example, there were fewer differences than we

had anticipated in the students’ reported experience of lessons in the previous week.

In Class

58. Among Year 6 students there was no consistent patterns, although teachers in the

francophone system were reported to be more likely to correct the pupils exercise books. In

contrast, among Year 10 students, considerably more students in the anglophone system


reported that their teachers corrected their homework (46% compared to 21% in the

francophone system) and checked that they had copied down their homework (57% compared

to 23% in the francophone system).

59. An astonishing 86% of Year 6 pupils said that their teacher was sometimes difficult to

understand and 61% that their teacher often complained about them but this was true in both

groups. However, more Year 10 students in the francophone system reported that they did

not understand their teachers all the time - but at the same time that they were never bored in

class.

60. Among Year 6 students, teachers in the anglophone system were reported to be more

likely to help when the pupil did not understand and to give lessons that everyone liked.

Similarly, among Year 10 students, teachers in the anglophone system were reported to be

more likely to be concerned with their difficulties, wanting to make them succeed and giving

interesting lessons. However, in contrast to Year 6 students, more Year 10 students in the

anglophone students thought their teachers had favourites in class.

61. Among Year 6 students, those in the anglophone system were tested most frequently,

whilst Year 10 students in the francophone system said they were tested more often and

answered questions more often. However, teachers in the anglophone system were more

likely to use last year’s tests as revision.

62. Finally, whilst among Year 6 students, there was little difference in the extent to which

subjects were liked, considerably more Year 10 anglophone students claimed that they like a

lot both Maths and Science lessons than was the case in francophone schools.

Homework

63. Similar proportions of Year 6 teachers in both groups said that, for homework, their

students used a textbook, or an exercise book or worksheets. In contrast, among Year 10

teachers, the homework given by anglophone teachers more often (51%) involved students

using their textbooks compared to those in francophone schools (25%). Two thirds of

anglophone teachers gave homework that involved students using their notebooks compared


to half in francophone, and whilst anglophone teachers nearly always used previously

prepared homework sheets, this was true of only a third in francophone schools.

64. In terms of the content of the homework, Year 6 teachers in the francophone system were

more likely to give lessons to memorise, although the proportions were about the same

among Year 10 teachers; perhaps the major difference was that in both Year 6 and Year 10,

more of the teachers in anglophone schools gave their students a passage to read compared to

those in francophone schools.

3.3 The francophone students received the same amount of mathematics, but less

science instruction than the anglophone (Hypothesis 3.3 on Opportunity to Learn)

65. The Maths and Science Opportunity to Learn check lists were completed by 55

teachers ( 34 Anglophone, 21 Francophone; 26 Science and 20 Maths teachers). No

significant differences were observed in Mathematics, where in both groups the average

percent of teachers reporting that the IEA item content had been taught to their Year 10

students was 86%. In Science, however, there is some evidence that the francophone

students were offered less opportunities to learn the topics measured by the IEA sciences test

(although all items used were part of the official curriculum). On average, the science items

had been taught by 73% of the francophone teachers compared to 82% of the anglophone

teachers.

3.4 The proportion of untrained teachers is higher in the francophone schools

(Hypothesis H3.4).

66. Whilst nearly all the Year 6 teachers in the anglophone system had 2 or more years

training, this was true for less than two thirds in the francophone system. Similarly, among

Year 10 teachers, about twice as many in the francophone system had no training (31%)

compared to those in anglophone schools (17%), although roughly the same proportion also

had been on In Service Training (INSET). Clearly, these patterns are different from those

portrayed in, for example, the Master Plan, Annexes 20-25. However, it is important to


understand that our data refer to those teachers who were currently teaching in the schools

and who replied to the questionnaire: some of them may not have had contracts with the

Ministry of Education.

67. Whilst a higher proportion of the francophone Year 6 teachers were female (43%

compared to 29%), there was a significant male bias among Year 10 teachers in francophone

schools (23M, 5F), compared to anglophone schools (47M 23F).

68. Finally, it is noteworthy that more francophone teachers thought they “spoke French very

well” compared to how the anglophone teachers thought they spoke English.

3.5 The results of the IALS test show that the anglophone trainee teachers preparing

for the primary teaching outperform the francophone trainee teachers in both the

primary and secondary training sections.

69. For both the IALS reading and quantitative scales, it is the group of primary anglophone

primary teachers who score highest, followed by the francophone students taking the

secondary course and then the francophone students taking the primary course. These

significant differences in the level of competencies in reading and arithmetic relate, of course,

to future teachers and not to those currently teaching. Nevertheless, they seem to confirm

Hypothesis H3.4 (that francophone teachers are less well prepared).

70. The ‘background’ questionnaire completed by the same future teachers suggests a

different attitude to books: the francophone trainee teachers claim to use the BETC library

much less, spend less time reading, and less often have books in their rooms than the

anglophone trainee teachers. Finally, in contrast to the teachers currently teaching, the

francophone trainees appear to be less confident of the mastery of the teaching medium (both

spoken and written) than the anglophone trainees.

Conclusion


71. In Year 6, slightly more students in the francophone system had their own language

and mathematics textbooks. However, in Year 10, significantly fewer students in the

francophone system had appropriate material resources both in terms of textbooks and

exercise books, and these reports were confirmed by the teachers. More students in the

francophone system more often had to pay for their own materials. Although the differences

in instructional practices were smaller than expected from our initial observations, the

teachers in the anglophone system tended to adopt the more interactive style of teaching

which is known to be more effective. Further, not only is the level of qualification of existing

teachers in the francophone system lower among both Year 6 and Year 10 teacher

populations, the scores of francophone teacher trainees, on an internationally comparable test

of linguistic competence, were lower.

Factors Associated with Success

72. As can be seen, the examination of the hypotheses in the light of the evidence has

enabled us to eliminate some and retain others. Thus, whilst there are some differences in the

content and organisation of the examinations, these are not sufficient to account for the

substantial differences in scores. The background characteristics of the two groups of

students are similar and therefore unlikely to be a major factor in accounting for the

differences between the groups. There are substantial differences in cultural environment (in

particular in terms of books in the home) in favour of anglophone students; and, for

anglophone Year 10 students, their linguistic environment is favourable. Whilst the students

selected after the Year 6 exam are clearly from the top of their respective groups, relatively

more francophone students are chosen. There are differences in instructional practices but

they are relatively small and inconsistent between primary and secondary; although

anglophone students have a clear textbook advantage in Year 10.

73. The following results, based on more comprehensive analyses in the more extensive

technical report, aim to assess the relative power of the various factors. They are based on

the adjusted ‘fair’ scores for Year 6 students in the language examination, and on the adjusted

‘fair’ scores for year 10 students in language, mathematics and science. These adjusted


scores have been derived from items which proved to be equally difficult (or fair) as between

students of the same competence in the anglophone and francophone groups.

74. The bivariate associations show that nearly all the factors identified as potentially

important make a difference to the scores of individual pupils within both anglophone and

francophone groups. Some examples of the correlations between the various explanatory

factors and these IRT scores are given in Table 5 above. The most important variables (in

terms of the size of the correlations) are:

• background characteristics and environment (age, gender, whether or not eat traditional

foods, access to water, gas and electricity, books at home, and parental education and

help with homework).

• selectivity (for Year 10 students only), their score on the Year 6 exams in 1993 is

correlated with their Year 10 score although not as much as one might expect from

international experience.

• material resources (whether or not pupils has corresponding textbooks, separate exercise

books, pays fees for materials).

• instructional practices (whether or not pupil is bored in lessons, reads stories for

homework, perceives the teacher as fair).

75. Clearly however, these different factors interact and overlap: if there are more books

in the home, the students linguistic environment will be different which also means that they

can take more advantage of teaching: certain pedagogies will be favoured by greater/lesser

reliance on textbooks, and so on. It is therefore important to assess the relative weight of

each of these factors; and we do this by estimating equations relating the scores to the

candidate variables that we have discussed above.

76. The interpretations below are based on seven different models:

• the first (null model A) aims to divide the total variance for each of the dependent

variables (the IRT scores) into two components: the variance between the schools and the

variance between students in the schools.


• the second (model B) and the third (model C) allows one to estimate the fraction of the

total variance between schools which can be ‘explained’, respectively by the teaching

medium and by the mark obtained in 1993 when the student in Year 10 in 1997 took the

Year 6 examination.

• the fourth model (B + C) allows us to demonstrate the extent to which the two preceding

factors overlap each other (and therefore to estimate the extent to which the disparities

between secondary schools can be explained by the differential difficulty of the

examinations taken in Year 6 by anglophone and francophone groups)

• the fifth model D provides an estimate of the total variance in each of the IRT scores

which we can account for by taking into account the background, cultural and school

environment of the students which were described in paras 45-71 above. These variables

explain, at the same time, part of the differences observed between schools (for example,

the variables describing pedagogical practices), and part of the variance observed between

students in the same school (for example the variables describing the cultural environment

of the students).

• the sixth model (B+D) helps to understand the extent to which the environmental variables

described immediately above are associated with the differences identified between

teaching mediums. The comparison between the results of models B, D and B+D allow us

to draw the conclusion that the variance between schools, which is apparently explained

by the teaching medium, is associated - at a statistically significant level - to the handicaps

suffered by the francophone group of students in terms of their socio-economic status and

cultural background and in terms of the quality of the education provided by the schools to

which these students go.

• finally the seventh model (B+C+D) allows us - for Year 10 - to make a better estimate of

the bias introduced by the differential selectivity between anglophone and francophone

groups, and therefore to estimate with greater precision the effect of the various other

factors. In addition, it gives us an idea of the proportion of the total variance which we

have not been able to account for with all the information we have collected. This ‘non-

accounted-for’ variance is due to (a) factors other than those studied here which affect the


students results; (b) the factors that we have been able to include in the model not being

measured sufficiently precisely by the information we have collected; (c) the scores which

are the dependent variables are, just as much as the independent variables, affected by

measurement error.

77. The first observation is that, apparently, a very large proportion of the variance

between individual students can be ascribed to variation between schools: a little under a

third (32%) of the variance in Year 6 IRT scores in language is variation between schools;

the proportions are even higher for the analysis with the Year 10 IRT scores (a little under

60% for language, 40% for mathematics and 50% for sciences - Table 8, model A). These

proportions are much higher than those usually observed at the start of this kind of analysis

suggesting that the differences between schools are much greater than one would

typically meet in other school systems.

78. Model B in Table 8 shows that a significant part of this between school variance can

be ascribed to differences between anglophone and francophone schools: and this effect is

particularly marked for the language and science Year 10 scores, where the introduction of

the variable ‘teaching medium’ into the equation appears to explain about 35% of the

variance in the language score (out of the 58% of the between schools variance) and 30% of

the science score (out of the 49% of the between school variance).

79. In mathematics, (where we only had data that was comparable between the

anglophone and francophone group for the Year 10), the impact of the teaching medium is

much less important (about 8% of the 39% of the between school variance can be attributed

to the teaching medium). As we have seen with the results of the analysis of the IEA scores,

the handicap of the francophone students is smaller in this subject than for the two others.

The differences from one school to another however remain considerable; and they cannot be

attributed exclusively to the contrast between the anglophone system and the francophone

system. BOTH systems are characterised by large differences between schools.

80. As we have seen from the analysis of the IRT scores derived from both the IEA tests

and the National examinations, the differences linked to the teaching medium are much less

strong in Year 6 as compared to Year 10: apparently only 2.4% of the variance of the


language scores in Year 6 can be attributed to the teaching medium (Table 8, model B),

although the between school variance is still very substantial at that level (31%). That

suggests that, at least for language, the handicap of students in francophone schools is

accentuated whilst they are at secondary school.

81. This result could be, at least in part, linked to the less severe selection of students

into the francophone systems. Models C and B+C tend however to show that the ‘selectivity’

factor is complementary to the effects of the teaching medium rather than the effects

overlapping with each other (the overlap between the two factors is weak, not only for

language but also for the mathematics and the sciences). However it should be noted that the

Year 6 score entered into these analysis is standardised to the average of the Year 6 scores in

each language group.

82. It is interesting to note that the Year 6 score is a better ‘predictor’ for the Year

10 exam results in mathematics, than for either languages or for science (probably

because the mathematics tests are more reliable) (Model C).

83. Surprisingly, this transition score to the secondary system contributes more to an

‘explanation’ of the disparities between schools than to an explanation of the differences

between the students within the schools (Model C). This suggests that the students who

obtain better results in the Year 6 examination tend to congregate in certain secondary

schools rather than others. This phenomenon is most marked in languages and in sciences,

where the two variables ‘teaching medium’ and ‘Year 6 score’ by themselves, allow us

to account for more than three quarters of the differences in achievement observed

between the schools in Year 10 (Model B+C).


Table 8 : Partitioning Variance between Schools and Students with Different Models

Year 6 Language

Year 10 Language

Year 10 Maths

Year 10 Science

(A) Null Model Level 1 (School) Level 2 (Student)

0.655 (31.9%) 1.396 (68.1%)

5670 (57.9%) 4120 (42.1%)

4954 (39.1%) 7711 (60.9%)

3864 (50.6%) 3769 (49.4%)

(B) Medium Level 1 (School) Level 2 (Student) Effect on School Variance

0.603 (29.4%) 1.396 (68.1%) -2.4%

2234 (22.8%) 4122 (42.1%) -35.1%

4094 (32.3%) 7715 (60.9%) - 6.8%

1433 (18.8%) 3782 (49.5%) - 31.8%

(C) Previous Score Level 1 (School) Level 2 (Student) Effect on School Variance Effect on Student Variance

na 4787 (48.9%) 3856 (39.4%) -9.0% - 2.7%

3268 (25.8%) 6458 (51.1%) -13.3% - 9.8%

3123 (40.9%) 3593 (47.1%) -9.7 -2.3%

(B) Teaching Medium + (C) Previous Score Leve1 1(School) Level 2 (Student) Effect on School Variance Effect on Student Variance

na 1282 (13.9%) 3862 (39.4%) -44.0% -2.7%

2336 (18.5%) 6468 (51.1%) -20.6% -9.8%

833 (10.9%) 3611 (47.3%) -39.7% -2.1%

D) Background, Cultural and School Environment Leve1 1 (School) Level 2 (Student) Effect on School Variance Effect on Student Variance

0.464 (22.6%) 1.203 (58.6%) -9.3% -9.5%

3279 (33.5%) 3737 (38.2%) -24.4% -3.9%)

3654 (28.9%) 6976 (55.1%) -10.2% -5.8%

2384 (31.2%) 3397 (44.5%) -19.4% -4.9%

(B) Teaching Medium + (D) Background, Cultural and School Environment Level 1 (School) Level 2 (Student) Effects on School Var. On Student Variance

0.443 (21.6%) 1.203 (58.6%) -10.3% -9.5%

1424 (14.5%) 3739 (38.2%) -43.4% -3.9%

3539 (28.0%) 6974 (55.1%) -11.1% -5.8%

1166 (15.3%) 3409 (44.7%) -35.3% -4.7%

(B) Teaching Medium + (C) Previous Score + (D) Background, Cultural and School Environment Level 1 (School) Level 2 (Student) Effects on School Variance Effect on Student Variance

na 742 (7.6%) 3536 (36.1%) -50.3% -6.0%

2055 (16.2%) 5956 (47.0%) -22.9% -13.9%

714 (9.4%) 3296 (43.2%) -41.2% -6.2%

Notes:

(a) Estimations are presented for seven separate models: the first simply partitions the variance between the schools and the students in those schools; the second and third show how much the teaching medium and the previous score separately account for the differences between the schools; the third shows the combined effect of the previous score and the teaching medium; the fifth shows the effect of all the environmental variables (both home and school); the sixth the combined effect of these environmental variable with the teaching medium; and the final model, the effect of also including the previous score on differences between both schools and students.

(b) The percentages are calculated by reference to the total variance in the null model: for example, 3726 is 50% of 7453 not of 5508 (=1479+3829). (c) The ‘Effect on the Variance’ is the percentage amount by which the proportion of variance attributable to the school or to the student level is decreased by the introduction of the variable. 84. The same is true to an even greater extent for the collection of variables which

describe the student and her/his educational environment (both at home and at school). Many


of these variables are related to the personal characteristics of the student and her/his family

milieu: one might therefore have expected that they would account for a much greater part of

the individual between-student variance. In fact, they only ‘explain’ about 10% of the

variance between students within the schools in Year 6 and only between 4% and 6% in Year

10. In contrast they are associated with a substantial fraction of the between school variance,

particularly in Year 10 for languages and sciences (Model D). This confirms the existence of

non-negligible aggregation effects: the secondary schools in the country are significantly

different, both in terms of the social, cultural and educational background of the

students enrolled and the quality of the education environment that they provide for

their students.

85. If one compares the results of Models B, D and B+D, one can estimate the fraction

of the between school variance which can be attributed specifically to the characteristics of

the educational environment (i.e. the effect, whichever the teaching medium) and to

distinguish it from that fraction which is linked to the joint effect of the teaching medium and

the environmental characteristics. The identification of this latter effect was one of the most

important objectives of the study, because it allows to verify hypothesis 2 and 3: are the

weaker results of the students in the francophone schools due to a less favourable educational

environment to which they are exposed, either at home or at school?

86. The ‘joint’ effect is in fact negligible in Year 6 (only 1.4% of the between school

variance in the language score); in contrast, it is highly significant in Year 10 (16% of the

between school variance for both the language and science scores, and 5% for the

mathematics score). The handicaps observed among the francophone students at

secondary level in terms of socio economic and cultural background and the poorer

school environment allow us to account for about half the differences of the results in

languages and science observed between the anglophone and francophone groups (for

the language score, 16% out of the 35% between school variance associated with the teaching

medium; 16% out of 32% for the science score).

87. Whilst these analyses are very important, it is also useful to examine the pattern of

variables which retain statistical significance in the ‘fixed’ part of the model. For both Year

6 and Year 10 students, after controlling for school membership (which we have to do in


order to obtain unbiased estimates), whilst gender always and age nearly always has an

effect, the other ‘background’ characteristics appear only occasionally as statistically

significant in the analyses. When we further include either teaching medium or the previous

score or both in the analysis, this tends to substitute to a limited extent for these background

variables (and the effect of the previous score and the teaching medium are reduced), but not

by as much as one might expect. The most robust effects are as follows: with the analysis for

language in Year 10, there are effects attributable to whether or not the family draws water

from a river (negative), whether they cook on gas, whether or not there are books at home,

and active reading on the part of the child (all positive); with the analysis for mathematics in

Year 10, there are effects for whether or not the family eats a traditional breakfast (negative),

and whether or not the head of the household does paid work (positive); and for the analysis

of science in Year 10, the only effect is active reading on the part of the child (see Table 9).

88. For both Year 6 and Year 10 students moreover, even after controlling for school

membership, some of the variables reflecting material resources or instructional practices

remain statistically significant in all three analyses at Year 10; and these effects are also not

affected substantially by the presence of teaching medium and/or previous score in the

model. Thus, whether or not the student has a calculator of a dictionary is always significant

(positively); in the analyses for language, whether or not the student has the textbook has a

positive effect as does the possession of geometrical instruments (although this is a negative

effect); in the analyses for mathematics and for science, whether or not the student uses a

ringbinder has a large negative effect, whilst whether or not they like the subjects has a

positive effect; in addition for the analysis with mathematics, whether or not the student pays

for their materials appears to increase their score (see Table 9).

Table 9 Coefficients and Standard Errors of ‘Best’ Models for whole population

Language Mathematics Science Constant 174.9 (16.7) -16.9 (31.7) -45.5 (23.3) (0-1) Effect of teaching medium -103.8 (13.0) -37.0 (22.1)

-73.0 (15.6)


Score in Year 6 on that subject 19.9 (2.5) 39.4 (3.2) 8.0 (2.4) Gender Age

--10.8 (2.2)

-15.5 (5.6)-9.8 (2.9)

-20.6 (4.2) -9.6 (2.2)

Water from River Cook on Gas Books at Home Eat traditional breakfast Occupation of Head of Household Active Reading

-10.8 (5.5)12.4 (5.1)

6.1(3.1)--

7.7 (3.1)

---

-7.6 (3.1)13.8 (6.2)

-

- - - - -

6.5 (3.0) Subject Manual Use Ringbinder Own calculator and Dictionary Own Instruments Pays for Materials Like Subjects

5.4 (2.6)-

4.6 (3.5)-8.8 (2.3)

--

--17.8 (9.0)12.8 (4.6)

-16.8 (8.2)7.3 (2.1)

- -18.5 (6.8)

8.4 (3.4) -

3.5 (1.6)

RESET test (separately for anglophone) (separately for francophone)

4.03.91.9

4.73.21.3

2.9 2.2 1.9

Notes: (a) only variables that are statistically significant at the conventional 5% level have been retained (b) the RESET test is a general test for mis-specification of the model

89. The equations have been tested for completeness and it appears that there are some

systematic effects which have not been captured by the variables that we have been able to

include in the model. It is important to recall here first, that it was not possible to include the

teacher level variables into these analyses without losing too much data; and second, that we

had ourselves been surprised, on the basis of our observations, that there were fewer

quantitative differences in the questionnaire responses between the instructional variables. If

we had been able to measure the differences that we thought we saw, we would doubtless

have achieved a more complete model. Moreover, when separate models are estimated for

the anglophone and francophone populations, even though similar (although not exactly the

same) variables appear in both groups, the models for the francophone population and for the

science analysis for the anglophone population appear to be well-specified, suggesting that

there are different processes at work.


90. Overall, therefore, the picture appears to be as follows:

• younger children do better than older;

• girls tend to do better than boys;

• those from a rural or poorer background perform worse at language and mathematics (but

not necessarily at science)

• it helps if the children like subjects!

• books at home and active reading are associated with better language performance

• students with their own calculator and dictionary perform better at all three subjects

• students are handicapped with a ringbinder

These will be taken up in the recommendations.

RECOMMENDATIONS

91. On the basis of the preceding analyses and the results obtained, a series of

suggestions can be made which might help to diminish the observed disparities and/or to

improve the quality of the educational system in Vanuatu. We have distinguished, in these

recommendations, between those which are short term (ST), medium term (MT) or long term

(LT).: where short-term refers to changes requiring resources but little prior planning or

preparation, medium term to changes requiring preparation and planning as well as resources,

and long term to those reforms needing delicate negotiations between the different groups

and a substantial political commitment.

Material Resources

92. Compared to many other developing countries, the children do have basic materials

and textbooks, although the Master Plan paints perhaps too rosy a picture of the coverage

(see section II.3(a) Availability of Textbooks and other teaching/learning materials). In


particular, where the approach to teaching relies on those textbooks then even small shortfalls

can become serious (ST); and in this respect the francophone students appear to be

disadvantaged especially in science. Moreover, it seems clear that, at least in Year 10, many

teachers in the Francophone system think that that the quality of the mathematics and science

textbooks are not appropriate and need to be revised (MT). The replies to the Opportunity to

Learn questions confirm, at least for science, that the subject matter taught to students in the

francophone system is more limited and corresponds less to the official curriculum than is the

case in the anglophone system..

93. In addition to the issue of textbooks, the use of ring-binders - which are difficult to

manage - should be discouraged; and serious efforts should be made to rectify the apparent

discrepancy between the students in the francophone system, who have to pay for their

exercise books, whilst these appear to be free - or subsidised - in the anglophone system

94. More generally, the students who attend francophone schools have a more limited

learning environment. Not only do they have fewer textbooks in Year 10 (although not in

Year 6), but there are fewer books in their homes and in their school libraries (or they are less

accessible). They are less frequently asked to read, and francophone student teachers also

read less. The government could launch a national campaign to sensitise parents and teachers

to the importance of reading; donors could support this through the provision of appropriate

materials.

Instructional Practices.

95. The differences whether reported by students or by the teachers were not as large as

we had originally surmised. However, international experience and some of the results

observed in the present study, suggest that there would be advantages in moving towards a

more interactive style of learning for both systems but especially for the francophone system.

Whilst some short term progress could be made with in-service training (INSET), this also

implies revision of the initial teacher training programme and substantial retraining over the

medium term.


96. At the same time, it is also evident that there are substantial differences in the

qualifications and experience - even within their own systems - of the anglophone and

francophone teachers. Although, administrative data show that ‘few teachers are unqualified’

(Master Plan section II.3(c), Teachers and Teaching Conditions), our data suggest that,

among those currently teaching in the schools, who may include some teachers not employed

by the state, there is a substantial minority without basic teaching qualifications. Rectifying

this would imply a substantial investment over the long term; although a comprehensive

INSET programme would contribute to reducing these disparities.

Selectivity

97. The problem posed by the difference of selectivity of the examination at the end of

Year 6 is probably the one which is the most politically sensitive to resolve. Should one

maintain the current pragmatic and apparently ‘open’ approach which means that the rate of

promotion/transition at the end of the primary cycle, can be adjusted according to the

differential capacity of the anglophone and francophone secondary systems? Should one

completely integrate the curriculum and the examination in the primary cycle, in order to

guarantee that students of the same capacity have the same chance of being promoted, with

the risk of serious instability in the secondary school systems, as the numbers of enrolments

in the francophone system could drop rapidly whilst those in the anglophone system might

grow rapidly. Should one instead increase the capacity of the anglophone secondary school

system, in order to reach a better equilibrium - and at the same time reduce the potential

tension because of the relatively severe selection in the anglophone examination at the end of

Year 6? This latter has the advantage of opening up access to secondary schooling to a larger

proportion of the country's juvenile population. However, even this 'equilibrium' is unlikely

to be acceptable to the francophone popul

study of comparison of performance of year ...pills : pacific island literacy levels (niveaux de...

Documents