the surprising lack of evidence about...
TRANSCRIPT
THESURPRISINGLACKOF
EVIDENCEABOUTWHATWORKSINEDUCATION‐ IMPLICATIONSFORPOLICYMAKERS
NICHOLASBURNETT12NOVEMBER2014
JICATRAINING,HIROSHIMAUNIVERSITY
OUTLINE
1. TheGlobalLearningCrisis2. Quiz:WhatImprovesLearning?3. WhyLearningMatters‐ EvidenceonLearningandEconomicDevelopment4. TypesofEvidence5. Video:WhatWorksinPrimaryEducation?6. SourcesofEvidence7. WhyEducationInnovationMatters8. HowtoAssessInnovation?9. ImplicationsforPolicyMakers
2
1.THEGLOBALLEARNINGCRISIS
3Source:GMR2013
4
1.THEGLOBALLEARNINGCRISIS
• Learning(e.g.UWEZOinEastAfrica)
TestpassratesforchildrenenrolledinStandard3
Notes:“combined”referstopassesonbothnumeracytestandatleastoneoftheliteracytestinthesurvey
TestpassratesforchildrenenrolledinStandard7
Source:UWEZO,2014
1. WeneedtofocusonWHATWORKSFORLEARNING
2. WeneedtoencourageINNOVATIONFORLEARNING
5
1.THEGLOBALLEARNINGCRISIS
• ImplicationsoftheLearningCrisis
2.QUIZ:WHATIMPROVESLEARNING?
6
1. Increasedspendingoneducation
2. Mathematicsmaterialsatprimarylevel
3. Increasedteachers’salaries
4. Schoolbasedmanagement
5. Separatetoiletsforgirls
6. Extracurricularclubsforgirls
7. Lowcostprivateschoolsforthepoor
8. Conditionalcashtransferslinkedtogrades
Whichofthefollowingimproveslearning?
NO
YES
NOTCLEAR
NOTCLEAR
NO
YES
SOMETIMES YES
7
3.EVIDENCEONLEARNINGANDECONOMICDEVELOPMENT
SourceHanushek 2013
• EconomicDevelopment
3.EVIDENCEONLEARNINGANDECONOMICDEVELOPMENT
8
• Individualreturnstoeducationarelinkedto
learning,notjusttoyearsofeducation
•Theoveralleconomybenefitsfromquality
education
• It’saboutchildrenlearning...
3.EVIDENCEONLEARNINGANDECONOMICDEVELOPMENT
9
• Returnstocognitiveskills(literacy)aregenerallystrongacrosscountries
3.EVIDENCEONLEARNINGANDECONOMICDEVELOPMENT
10
• EstimatedReturnstoCognitiveSkills:Examples
ThreestudiesconductedintheU.S.showa12%increaseinearningsforevery1SDincreaseinmathscores.(Mulligan,1999;Murnane etal,2000;Lazear,2003)
Chileandatashowsthat1SDincreaseintestscoresontheIALSisassociatedwithhigherearningsof15‐20%.(Sakellariou,2006)
3.EVIDENCEONLEARNINGANDECONOMICDEVELOPMENT
11
• EducationQualityandEconomicGrowth
SourceHanushek andWoessman,2007
3.EVIDENCEONLEARNINGANDECONOMICDEVELOPMENT
12
3.EVIDENCEONLEARNINGANDECONOMICDEVELOPMENT
13
• A‘general’theoryofwhatinfluenceslearningoutcomes
Demand Supply
StudentsParents
Community
TeachersSchools
Education authoritiesCommunity
Endowments & Behaviors
of Policy actions, specific programs
Systemic reforms
Student learning
EducationPolicy
Economic, social, political context
Endowments & Behaviors
of
Source:VegasandPetrow2008
4.TYPESOFEVIDENCE
14
• RandomizedControlTrials – thegoldstandardforimpactevaluation• Useofcounterfactual• Advantages‐ reliable• Drawbacks– costs,findingsmaynotbepolicyrelevant• Difficulty– definingresults
• RegressionAnalysis• Standardstatisticalandeconometrictechnique• Advantages– simple,cheap• Drawbacks– selectionbias,correlationmaynotbecausation• Difficulty– definingresults
• AggregatingandInferring• Crowdsourcing‐ wisdomofthesuccessful?• Meta‐analysis• IdentificationofEssentialIngredients
• NoneofthesesayanythingaboutCostsorPovertyImpact• Needbenefit‐costanalysis,benefit‐incidenceanalysisetc.aswell
4.TYPESOFEVIDENCE
15
Whatsortofdatameasurelearning?
• Gradecompletion– allstudents?
• Exams– allstudents?
• Assessments– validstatisticalsamples• Nationalassessments– doesyourcountryhave?• Internationalassessments– EGRA,PISA,TIMSS,PIRLS,
SACMEQ,PASECetc.
• School‐basedorpopulation‐based– studentsorschool‐agepopulation?
5.VIDEO
16
https://www.youtube.com/watch?v=kX1Hp7ZkjO4
ShariKrishnaratne“whatworksineducationindevelopingcountries”
17
6.SOURCESOFEVIDENCE
• PrimaryEducationKrishnaratne etal.(2013)Qualityeducationforallchildren?Whatworksineducationindevelopingcountries.WorkingPaper20.3ie.http://www.3ieimpact.org/evaluation/working‐papers/working‐paper‐20/
• SecondaryEducationBanerjeeetal(2013)ExpandingAccessandIncreasingStudentLearninginPost‐PrimaryEducationinDevelopingCountries:AReviewoftheEvidence.AbdulLatif Jameel PovertyActionLab(J‐PAL)Post‐PrimaryEducationInitiativeReviewPaper2http://www.povertyactionlab.org/publication/expanding‐access‐and‐increasing‐student‐learning‐post‐primary‐education‐developing‐count
• Girls'EducationUnterhalter etal(2014).Girls’educationandgenderequality.EducationRigorousLiteratureReview2014.EPPI‐Centerhttps://www.gov.uk/government/uploads/system/uploads/attachment_data/file/326205/Girls__Education_Literature_Review_2014_Unterhalter.pdf
• LowCostPrivateSchoolAshleyetal(2014)Theroleandimpactofprivateschoolsindevelopingcountries.EducationRigorousLiteratureReview2014.EPPI‐Center:https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/307032/Private‐schools‐2014.pdf
• InnovationsEducationInnovationFoundation.http://www.educationinnovation.org/
• EarlyChildhoodDevelopmentEngleetal (2011).ChildDevelopment2:Chapter2Strategiesforreducinginequalitiesandimprovingdevelopmentaloutcomesforyoungchildreninlow‐incomeandmiddle‐incomecountries.Lancet.378:1339‐53
Behrmanetal(2014)Chapter3:PreschoolPrograminDevelopingcountries.InP.Glewwe (Eds.).EducationPolicyinDevelopingcountries,Chicago:UniversityofChicagoPress
References(seeannex)
18
• PrimaryEducation
Source:3ie2013
6.SOURCESOFEVIDENCE
Outcome Whatworks? Whatdoesn’t work? What promising?Notpromisingbutcouldlookfurther
researchWhatwedon’tknow
Enrolment/Attendance CCTsHealthinterventions(attendance)
Materials(attendance)
Teacher Resources(attendances)BuildingsECD(enrolment)SchoolFeeding(enrolment)
SBMSchoolFeeding(attendance)
Teacher Resources(enrolment)Health(enrolment)ECD(attendance)Materials(enrolment)VouchersSchoolsFeesProvidingInformation
Progression, Repetition,Dropout
Teacher Resources(dropout)CCTsSBM(progression)
SBM(dropout) Teacher ResourcesHealthECDMaterialsSchoolFeedingVouchersSchoolFeesProvidingInformation
TestScore Materials(maths)
Materials(language) Teacher ResourcesSBM(maths)SchoolFeeding(maths)
Teacher Resources(global)Health(maths,language)CCTs(global)SchoolFeeding(language)
Health(global)CCTsECDMaterials (global)SBMSchoolFeedingVouchersSchoolFeesProvidingInformation
19
• Girls’Education(ImpactonLearning)
Source:EPPI‐Center2014
6.SOURCESOFEVIDENCE
Interventions Strong Promising More needed
ResourceandInfrastructure
Resource Interventionincashforfamilyorchildrenimpactonsuccessinthegradesachieved
Theprovisionofadditional schoolsinunderservedareaimpactsongirls’learningoutcomes
Research intotheimpactofresourcesandinfrastructuretendstofocusonattendanceatschools.Moreevidenceinneededtoinvestigatetheimpactoftheseinterventionsonlearning
Institutional andPolicy
Grouplearning isassociatedwithimprovedlearningoutcomes
Learningoutsidetheclassroomsthroughformalandinformalextracurricularactivities(eg tutoringafter‐schoolclubs)hasapositiveeffectongirls’learningoutcomes
Thereispromisingevidencetoshow thatinformalteachertrainingtodevelopattitudesofinclusiontolerancehasapositiveimpact
Theinvolvementofwomeninschoolgovernanceandcommunitymobilizationandincommunityleadershipmayhaveapositiveimpactongirls’confidence
Girl‐friendlyschool,andquality–mix,orcombinedschool‐levelreformsmayhaveapositiveimpactongirls’learning
More researchisneededtoreinforceexistingpromisingevidencethattheempowermentofwomenteachersmayhaveapositiveimpactongirls’learning
Moreresearch isneededintotheimpactofschoolchoiceandinclusivestrategiesongirls’learning
NormandInclusion Thereis promisingevidencethatcomplementarylearningspacestoschool(e.g.NGOprogrammes,girls’clubs)areeffectiveingivingopportunitiestodiscussgenderequality
Teaching aboutpersonal,socialandhealthissuelinkedwithsexeducationbothatschoolandincomplementaryprogrammes iseffectiveinprovidingknowledge
Women’sliteracyprogrammes,particularlywhentheyofferwomenandgirlsthechancetodevelopliteracyknowledge,skills,andspacesforreflectivediscussion
Moreresearchneeded intoreligion,educationandshiftinggendernormsnotetheimportanceofkeyindividuals,thesignificanceofsupportivenetworkswithinfaithcommunities,andthepotentialforreligiousinstitutionstositedforcriticalreflectionongendernorm
Alsomoreneededontheeffectoftacklinggender‐basedviolenceinschools,workingwithboysongenderequality,worktoincludemarginalizedgirls,developingcombinedprogrmmes linkingworkinschoolsongenderequalitytocommunitywork
20
• LowCostPrivateSchools
Source:EPPI‐Center2014
6.SOURCESOFEVIDENCE
StrongEvidence ModerateEvidence
Teachingisbetterinprivate schoolthaninstateschools,intermofhigherlevelsofteacherpresenceandteachingactivityaswellasteachingapproachesthataremorelikelytoleadtoimprovedlearningoutcomes
Privateschoolpupils achievebetterlearningoutcomeswhencomparedwithstateschools.However,thereisambiguityaboutthesizeofthetrueprivateschooleffect.Inadditionmanychildrenmaynotbeachievingbasiccompetencieseveninprivateschools
Thecostofeducationdeliveryislowerinprivateschoolsthanstateschoolsoftenduetolowersalariesforprivateschoolteacherscomparewiththeirgovernmentschoolcounterparts.Moreoftheevidencedoesnorigorouslyanalyzethecost‐effectivenessofprivateschools;however,thereissomelimitedevidencetoindicatehighercost‐effectivenessofprivateschoolsthanstateschoolsinspecificcontexts
Girlsarelesslikelythanboystobeenrolledinprivateschools.However,thisfindingiscontextspecificwithaminorityofstudiesfindingthatprivateschoolsreducethegendergapincertaincontexts
Perceivedbetterqualityofprivateschools(intermsofteaching,teacherattendance,schoolperformance,smallclasssize,discipline)comparedwithstateschoolsisakeyfactorinparents’choiceofprivateschools.OtherimportantfactorscitedincludeEnglish‐languageinstruction,futureoccupationpossibilitiesandpromotionratestosecondaryschool.
Theperceptionof‘privateschoolsasbetterquality’isinformedinformally,oftenthroughparents’informalsocialnetworks;suchsourcesplayasignificantbutoftenunder‐recognisedroleininformingusersintheirchoiceofschool.
Attemptsbystatestointerveneintheprivateeducationsectorareconstrainedbyalackofcapacity,legitimacyandknowledgeofthesectortoimplementeffectivepolicyframeworks.
Wherestateregulationofprivateschoolsexists,itisnotnecessarilyeffectiveormaybeselectivelyenforcedofferingopportunitiesforrentseekingandbribery.Althoughthefindingsaremostlynegative,therearesomeexamplesofpositivestateregulationsupportingtheexpansionofprivateschoolprovision;howevertherearealsoconcernsthatprivatesectorprovisionmaybepromotedbystateswithoutadequateregulationandqualitycontrols.
21
• LowCostPrivateSchools
Source:EPPI‐Center2014
6.SOURCESOFEVIDENCE
Inconclusive Limited
Theevidenceisambiguousaboutwhetherprivateschoolsgeographicallyreachthepoor.Althoughprivateschoolsarecontinuingtofocusonurbanareas,theyarealsobecomingincreasinglyprevalentinruralareas;butresearchcautionsagainstassumingthismeanstheyarereachingthepoor.
Thereisasmallbutconsistentevidencebasethatprivateschoolsaremoreexpensivethanstateschoolsintermsofbothschoolfeesandhiddencostssuchasuniformsandbooks.
Theevidenceonwhetherthepoorareabletopayprivateschoolfeesisambiguous.Mostisneutral,someisnegative,butthereisnopositiveevidence.Anumberofstudiesfindthataverysmallminorityofchildrenoflowereconomicquintilesaccessprivateschools.Financialconstraintsareakeyfactorlimitingorpreventingpoorerhouseholdsfromenrollingtheirchildreninprivateschools.Wherechildrenofpoorerhouseholdsdoattendprivateschools,researchindicatesthatwelfaresacrificesaremadeandcontinuedattendanceisdifficulttosustain.
Thereareveryfewstudiesaddressingrigorouslytheaccountabilityofprivateschoolstousers.Ofthese,thereissomeconsistentevidencethatusersparticipateinandinfluencedecisionmakinginprivateschools.Whileasmallbodyofmainlyanecdotalevidenceindicatesthatteachersandschoolsmayrespondtoparents’demandsandcomplaintsandultimatelythepotentialthreatofparentsexercising choice,noevidencewasfoundinthestudiesreviewedthatusersdoinfactchangeschoolsinresponsetoqualityconcerns,oraremorelikelytodosointhecaseofprivateschoolsthangovernmentones.
Thereislimitedevidencetosupportanyfirmconclusionaboutthefinancialstabilityofprivateschools.However,theevidenceavailableindicatesthatprivateschools,especiallyLFPs,maybevulnerabletoclosingdownaftershortperiodsoftime.Howevertherewasalsocounterevidenceofestablishedrecognizedprivate schoolsoperatingovermanyyears.Importantlyanassessmentofthesustainabilityofdifferentfinancialmodelsislackingintheliterature.
Theevidenceonsubsidiesislimitedinscope,sizeandcontext,butonedonorfundedprogramme inPakistanindicatesthatconditionalandtargetedsubsidiescanraisethequalityofinputs.
Thereislittleevidencetosupportorrefutethequestionofthesystem‐wideeffectsofprivateeducation.Theevidencebaseonwhetherprivateschoolscomplementorcompetewithgovernmentschoolprovisionisverysmall,howeversomeevidenceindicatesasupply‐sidesynergybetweengovernmentandprivateschoolsprovisionandthereisevidencethatprivateschoolsarefillinggapswheresupplyofgovernmentschoolsislow,butalsowheregovernmentschoolsareperformingpoorly.Theevidenceonwhethertheeffectofcompetitionistodriveupthequalityofpublicschoolsortodepleteitbyencouragingmoreablestudentstoexitthestatesectorissparseandcontested.
7.WHYEDUCATIONINNOVATIONMATTERS
22
Source:UNESCOGMRPolicyPaperNo.14,2014
Numberofout‐of‐schooladolescentsoflowersecondaryschoolagebyregionandsex,2000‐2012
Numberofout‐of‐schoolchildrenofprimaryschoolagebyregionandsex,2000‐2012
• OutofSchoolChildren
7.WHYEDUCATIONINNOVATIONMATTERS
23Source:GMR2013
• LearningCrisis
7.WHYEDUCATIONINNOVATIONMATTERS
24Source:GMR2013
• Effectiveness
7.WHYEDUCATIONINNOVATIONMATTERS
25
Programs,PoliciesandPractices
thatincreaseaccesstoorimprove
qualityorreducecostsof
educationforthepoor
• WhatareEducationInnovations?
7.WHYEDUCATIONINNOVATIONMATTERS
26
1Progressineducationdoesnotgrowandspreadfastenough
2Solutionsdoexist– promisinginnovations ineducationareconstantlyemerging
3…BUTtheiruptakeanddiffusionisslow!
• EducationChallenge:TheReality&TheOpportunity
7.WHYEDUCATIONINNOVATIONMATTERS
27
Blueprint Validate Prepare Scale
Developideaandcompellingvalueproposition
Testandrefinetheinnovation.Mayinvolvesomegrowth.
Enhancetheconditionsrequiredforscaling(i.e.,customerawareness,
effectivesupplychains)
Rolloutmodeltoreachalargenumberofcustomers
Takes5‐10yearstogothroughfirstthreestages
Source:BlueprinttoScale,producedbyMonitorGroupincollaborationwithAcumenFund
• UnderstandingtheEvolutionofanInnovation:FourStages
GlobalPublicGoodPlatformforDataandConnectionseducationinnovations.org
7.WHYEDUCATIONINNOVATIONMATTERS
28
1
23
Largestglobalpublicdatabaseofinnovations
Newanalysisabout‘whatworks’,howandwhy
Connectionsleadingtoimprovement,scale‐upandadaptation
• TheR4DCenterforEducationInnovations
8.HOWTOASSESSINNOVATION?
29
Interpretations Requirements Expand
Samecontext
But5to10verydifferentfrom50to10,000
Replicate
Differentcontext
Adapt
Samecontext,or
Differentcontext
Analyticalrequirements
Clearidentificationandanalytic
separationof“activeingredients”and
ofcontextualfactors.
Assessmentoftheeffectivenessofthe
activeingredients
Howtodothis?Evolvingtechniques….
• ScalingUp‐ Looseterm……..needtobeprecisewhenusing
8.HOWTOASSESSINNOVATION?
30
Theactivitiesbelowrepresentkeycheckpointsintheprocessofidentifyingpromisingglobalmodelspotentiallyadaptabletoothercontexts.Theorderofthestagesmayvarybyuser,desiredoutput,andinformation/resourcesavailable.Asaniterativeprocess,usersmayrevisitcertainstepstofurtherrefineanalysis.
Searchforprogrammodels
Identifytheactiveingredient(s)
Determineeffectiveness
Assessscalability
Basedontheproblemyouaretryingtosolveforandspecifiedparameters,whatmodelscurrentlyexist?
Whatarethecorecomponentsorattributesoftheprogramthatmakeitsuccessful?Whatarethecontextualfactorsfortheactiveingredient?
Whatarethebarriersandleverstoscalingtheactiveingredient(s)inaspecificcontext?
Whatevidenceexistsontheeffectivenessofidentifiedactiveingredient(s)?
OUTPUTSShortlistofprogrammodels Listofisolatedactiveingredients Narrowedlistofactiveingredients Listofbarriersandleversto
adaptation
• ProposedScalingFramework
8.HOWTOASSESSINNOVATION
• R4DCenterforEducationInnovations(CEI):http://www.educationinnovations.org/• Focusonthepoorinlowandmiddleincomecountries• Over500documentedprograms,mainlynon‐state
• WorldInnovationSummitonEducationHub:http://www.wise‐qatar.org/edhub• Over100describedinnovationsinrichandpoorcountries
• OECDCenterforEducationResearchandInnovation:http://www.oecd.org/edu/ceri/• FocusonOECDcountries
• STIR:http://www.stireducation.org/• Focusonteacherinnovations,IndiaandnowexpandingtoUganda
31
• SourcesofInformationaboutInnovation
9.IMPLICATIONSFORPOLICYMAKERS
32
1. Makingpolicywithinsufficientinformation
• Donotassumeyouknow,e.g.“”wehavealwaysdoneitthisway”
• Donotassumewhatspecialinterestssay,e.g.teacherunionorbankslendingfor
highereducation
• Constantlyscanforrelevantinnovations
• Applylogframe ortestabletheoryofchange
• Seewhichelementsmayhaveevidence
• Forthosewhichdonothaveevidence,makereasonableassumptionsbutinstall
M&Etoverifyornottheseassumptions
• Anotherapproach:rewardresultsbutdon’ttellimplementershowtoachievethem• Results‐basedfinancing• Output‐basedaid• Payforperformanceetc.
9.IMPLICATIONSFORPOLICYMAKERS
33
2. Buildingevidence‐gatheringintopolicy
• Allocatefundsforresearchandmanageresearchbudgetsactively
• Insistexternalfinanceforresearchflowthroughtheministry
• Setuppolicyadvisorycommitteesthatincluderesearchers
• Keepafocusonoutcomesand,toalesserextent,outputs
• Learnfromothercountries’experience
• AlwaysputM&Einplacewhenscalinginnovationsandnewpolicies