EFA 2000 Assessment Surveys Report

ASSESSING BASIC LEARNING COMPETENCIESAMONG YOUTH AND YOUNG ADULTS

IN DEVELOPING COUNTRIES:

Analytic Survey Frameworkand Implementation Guidelines

Final Report Based onExpert Workshop for EFA2000

April 12-14 1999, ILI-UNESCO, Philadelphia

Co-organized by

Produced and printed by ILI, Philadelphia, September 1999

Assessing BLC, September 1999, page 2

TABLE OF CONTENTS

ACKNOWLEDGEMENTS

EXECUTIVE SUMMARY

1 . INTRODUCTION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 . POLICY FRAMEWORK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1 PRIORITY POPULATIONS.......................................................................................................... 52.2 POLICY AND ASSESSMENT NEEDS.............................................................................................. 62.3 CHALLENGES FOR POLICY-DRIVEN ASSESSMENTS ......................................................................... 82.4 WHAT MIGHT POLICY MAKERS WISH TO KNOW ABOUT BLCS? ........................................................ 9

3 . DEFINING BASIC LEARNING COMPETENCIES (BLCS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.1 CHALLENGES IN DEFINING BLCS............................................................................................ 103.2 DIRECT AND INDIRECT MEASURES........................................................................................... 103.3 CONCEPTUAL OVERVIEW OF ASSESSMENT MATRIX ..................................................................... 113.4 DEFINING LITERACY (READING AND WRITING) .......................................................................... 123.5 DEFINING NUMERACY .......................................................................................................... 153.6 DEFINING LIFE SKILLS .......................................................................................................... 16

4 . SAMPLING AND SURVEY DESIGN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 7

4.1 PROPOSED LOW-COST BLC HOUSEHOLD SURVEY DESIGN ............................................................ 174.2 SOME ADVANTAGES OF THIS HOUSEHOLD SURVEY DESIGN .......................................................... 184.3 PRINCIPLES FOR EFFECTIVE HOUSEHOLD SURVEY DESIGNS ........................................................... 194.4 COST CONSIDERATIONS ........................................................................................................ 21

5 . MEASURING BLCS: AN ASSESSMENT MODEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1

5.1 LEVELS OF BLC SKILLS ........................................................................................................ 225.2 READING ASSESSMENT......................................................................................................... 225.3 WRITING ASSESSMENT.......................................................................................................... 245.4 NUMERACY ASSESSMENT....................................................................................................... 255.5 LIFE SKILLS ASSESSMENT....................................................................................................... 26

6 . IMPLEMENTATION AND CAPACITY BUILDING. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 8

6.1 DESIGN DECISIONS AND OPTIONS FOR BLC SURVEYS .................................................................. 286.2 HUMAN RESOURCES: RECRUITING AND TRAINING PERSONNEL...................................................... 296.3 TEST DEVELOPMENT, PRETESTING AND PILOT STUDIES................................................................ 306.4 ENSURING THE QUALITY OF DATA COLLECTION......................................................................... 326.5 DATA PROCESSING AND ANALYSIS .......................................................................................... 326.6 REPORTING AND DISSEMINATION ........................................................................................... 33

7 . CONCLUSIONS AND NEXT STEPS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3

TABLES

REFERENCES

ENDNOTES

ANNEXES

Assessing BLC, September 1999, page 3

ACKNOWLEDGEMENTS

This report is an outcome of a collaboration between theInternational Literacy Institute and UNESCO. It began with a firstexpert seminar at UNESCO-Paris in June 1998 entitled LiteracyAssessment in Out-of-School Youth and Adults, and from which afinal report was published in December 1998. The present report isbased on a follow-up workshop which was held in Philadelphia inApril 1999.

The overall goal of this activity is three-fold: to try toimprove our understanding of the nature of basic learningcompetencies (BLCs) within and across societies; to assistgovernments and multinational agencies in obtaining reliable anduseful information in this domain in order to improve in-schooland out-of-school learning; and to assist schools and programsthat provide instruction to better understand the effectiveness oftheir activities.

This project has been undertaken in the lead up to theEFA2000 activities in the year 2000. Original funding was providedto UNESCO by the IPIEO program funded by the World Bank to improveinternational data gathering. It also received support from withinUNESCO from the EFA, MLA, and ED/BAS programs. Within UNESCO, wewould especially like to thank Vina Chinapah, Dieter Berstecher,Aicha Diallo Bah and Abdulwahid Yousif for their support andencouragement.

The participants in the present workshop are listed in Annex1 of this report, and we are most grateful to them for theirinsights and critical comments during the workshop, as well as tovarious drafts which followed the workshop. Given the complexityof the task of trying to balance between the varied goals ofpolicy intent, statistical reliability, realistic scope of futurestudies, and limits of time for this work, we hope that our co-participants and the readers of this report will understand why wecould not cover every topic requested, or respond affirmatively toevery interesting idea. While we benefited from many comments, allremaining errors are those of the principal authors: Dan Wagner,John Sabatini, and Iddo Gal.

Assessing BLC, September 1999, page 4

EXECUTIVE SUMMARY

The World Conference on Education for All (WCEFA) in 1990 atJomtien, Thailand, included a number of educational targetsrelated to out-of-school youth and adults, including: (1) toreduce the number of adult illiterates to half of the 1990 levelby the year 2000; and (2) to improve learning achievement to anagreed percentage of an appropriate age cohort. WCEFA alsostressed the need to monitor and evaluate the performance ofindividual learners as well as the delivery mechanisms andoutcomes of literacy and other non-formal education programmes. Atthe Mid-Decade Review meeting on EFA in Amman (in June 1996), theinternational community further called for efforts at bothinternational and national levels to adopt new techniques andstrategies to collect and analyze meaningful data to monitorprogress towards the Jomtien goals. In the year 2000, UNESCO andother agencies will gather together worldwide data on education inorder to take stock of the worldwide progress toward the WCEFAgoals, a decade after Jomtien.

At a prior expert seminar held at UNESCO-Paris in June 1998,the focus was on describing and analyzing the need for betterassessment and monitoring of basic learning competencies (BLCs),from the basic skills (reading, writing, math) to basic lifeskills, as well as describing methodologies for assessments atthe national survey level and at the program level.

Following on the June 1998 seminar, the present workshop(organized jointly by the International Literacy Institute andUNESCO, and held on April 12-14 1999, in Philadelphia), focused onthe creation of a model for BLC assessment, especially for out-of-school youth and young adults. The present report provides both ananalytical framework and implementation guidelines, with five maindimensions: Policy Framework; Defining Basic Learning Competencies(BLCs); Sampling and Survey Design; Measuring BLCs: An AssessmentModel; and Implementation and Capacity Building.1

This report describes a low-cost and adaptable methodologyfor the assessment of basic learning competencies in diversecultural contexts. Next steps would involve a set of initialsurveys, including assessment instrumentation, for use either atthe national level or at a program level. This report, conductedin light of the 1990 Jomtien Conference and in anticipation of theEFA2000 evaluation activities, is a reflection of the increasedconcern about the status of basic learning competencies in manynations, as well as about the effectiveness of education program(formal and non-formal) at all levels. Through the use of a BLCassessment framework, such as described in this report,information collection for policy formulation and planning can bemuch improved in the coming years.

Assessing BLC, September 1999, page 5

1. INTRODUCTION

The World Conference on Education for All (WCEFA), held in1990 in Jomtien, Thailand, has highlighted the need to improvelearning opportunities and learni