Organizing and Organizing and Managing Program Managing Program

EvaluationEvaluation

Presentation OutlinePresentation Outline

1.1. Setting the Stage: U.S. Setting the Stage: U.S. Housing MinistryHousing Ministry

2.2. Broad Organization of the Broad Organization of the Evaluation FunctionEvaluation Function

3.3. Defining Evaluation TopicsDefining Evaluation Topics4.4. Managing Evaluation ProjectsManaging Evaluation Projects5.5. Concrete ExamplesConcrete Examples

Principal tasksPrincipal tasks• Insuring mortgages – FHAInsuring mortgages – FHA• Mortgage refinance – GNMAMortgage refinance – GNMA• Direct loans for elderly and Direct loans for elderly and

handicapped housinghandicapped housing• Housing subsidies to assist low Housing subsidies to assist low

income familiesincome families• Grants to localities for community Grants to localities for community

developmentdevelopment• Promote “fair housing”Promote “fair housing”

Budget and Program Overview:Fiscal Year 2000

Program Program BlockBlock

BudgetBudget($ billion)($ billion)

No. of Major No. of Major ProgramsPrograms

Percent of Percent of budget totalbudget total

Community Community developmentdevelopment

7.87.8 1212 2424

Housing Housing AssistanceAssistance

23.323.3 1111 7272

Housing Housing financefinance

0.80.8 25-3025-30 2.52.5

Fair housingFair housing 0.40.4 22 1.41.4

totaltotal32.232.2 60-6560-65 100100

OrganizationOrganization• Minister Minister • Deputy MinisterDeputy Minister

– Program Assistant Ministers (A/M)Program Assistant Ministers (A/M)– A/M for Administration (budget, A/M for Administration (budget,

personnel)personnel)– CFO (financial management; efficiency)CFO (financial management; efficiency)– Inspector GeneralInspector General– A/M for Policy Development & Evaluation A/M for Policy Development & Evaluation

(PDE)(PDE)

Office of Policy Development Office of Policy Development and Evaluationand Evaluation

• Provide advice and information to Provide advice and information to minister for decision makingminister for decision making

• Monitor and evaluate ministry’s Monitor and evaluate ministry’s programsprograms

• Conduct research on priority housing Conduct research on priority housing and community development issuesand community development issues

• Key participant in the budget Key participant in the budget processprocess

• Feeds evaluation results into policyFeeds evaluation results into policy

2. Organization of the Evaluation Function

Division of tasks among Division of tasks among officesoffices• Program monitoring, including Program monitoring, including

performance measurementperformance measurement– Program A/MsProgram A/Ms– CFOCFO– PDEPDE

• Program evaluation – PDEProgram evaluation – PDE– Within PDE, Office of Research & Within PDE, Office of Research &

EvaluationEvaluation

Office of Research & Office of Research & EvaluationEvaluation

• Defines evaluation agendaDefines evaluation agenda• Working with program offices, Working with program offices,

defines the objectives of the defines the objectives of the evaluations and methodsevaluations and methods

• Conducts competitions to select Conducts competitions to select firms to do workfirms to do work

• Oversees works of contractorsOversees works of contractors

3. Defining Evaluation Topics

Who nominates programs for Who nominates programs for evaluation?evaluation?

• Congressional mandatesCongressional mandates• Office of Management & Budget Office of Management & Budget

(Office of the President)(Office of the President)• Program A/MsProgram A/Ms• Office of Policy Development Office of Policy Development

and Evaluationand Evaluation

Federal Government Budget CycleFederal Government Budget Cycle--Illustrated for the FY2001 Budget----Illustrated for the FY2001 Budget--

Dat eDat e ActionAction

August August 19981998 OMB sends “management letter to OMB sends “management letter to agencyagency

January January 19991999 Minister issues instructions for the Minister issues instructions for the budget process, including guidance budget process, including guidance on prioritieson priorities

February-MarchFebruary-March Offices work on submissionOffices work on submission

April – JulyApril – July Internal review processInternal review process

July-AugustJuly-August FinalizationFinalization

SeptemberSeptember Submission to OMBSubmission to OMB

October-NovemberOctober-November Formal hearings with OMBFormal hearings with OMB

mid-Novembermid-November OMB “pass-back”OMB “pass-back”

mid-Nov to mid-Decembermid-Nov to mid-December Appeals; final decisionsAppeals; final decisions

mid-January mid-January 20002000 Submission to CongressSubmission to Congress

May-August / October 1May-August / October 1 Congress actsCongress acts; new fiscal year; new fiscal year

4. Managing evaluation projects

Basic steps in the Basic steps in the processprocess

1.1. Defining the questions to be Defining the questions to be addressedaddressed

2.2. Determining the methodolgyDetermining the methodolgy3.3. Estimating the budgetEstimating the budget4.4. Preparing the TOR or RFPPreparing the TOR or RFP5.5. Competition and awardCompetition and award6.6. Overseeing the contractorOverseeing the contractor7.7. Getting policy impactsGetting policy impacts

1. Defining the questions to 1. Defining the questions to be addressedbe addressed

• Understand the program before Understand the program before meeting with program officials about meeting with program officials about the evaluationthe evaluation

• ““No client, no impact.”No client, no impact.”– Make the program office your partnerMake the program office your partner– Understand who else may be a “primary Understand who else may be a “primary

intended user”intended user”– Consult them at the beginning and keep Consult them at the beginning and keep

them fully informedthem fully informed

Reality checkReality check

• For each question askFor each question ask– Can the question be answered with Can the question be answered with

program evaluation tools?program evaluation tools?– What is the relevant indicator?What is the relevant indicator?– What is the source of the data? What What is the source of the data? What

does it cost?does it cost?• Is the overall cost affordable?Is the overall cost affordable?• If the answers are positive proceed. If the answers are positive proceed.

If not, redesign.If not, redesign.

2. Determine the methodology2. Determine the methodology

• Depends on the type of evaluation—Depends on the type of evaluation—process, impact or benefit-costprocess, impact or benefit-cost

• Prepare clear statement of method Prepare clear statement of method that can be critically reviewedthat can be critically reviewed

• Where data are to be collected, Where data are to be collected, estimate sample sizesestimate sample sizes

• If you do not have the capacity on If you do not have the capacity on staff, hire an expert to helpstaff, hire an expert to help

3. Estimate the budget3. Estimate the budget

• Labor for structuring the Labor for structuring the project, analysis, reporting project, analysis, reporting (build up task-by-task)(build up task-by-task)

• TravelTravel• Data collectionData collection

– Cost per household or program manager Cost per household or program manager interviewed; office file checkedinterviewed; office file checked

• OverheadsOverheads– Office space, communications, support Office space, communications, support

staff, graphics, computers…staff, graphics, computers…

Staff loading chart for analysis and results write-up Staff loading chart for analysis and results write-up for impact analysis using econometric methods (days)for impact analysis using econometric methods (days)

Task Senior staffSenior staff Mid-levelMid-level Junior staffJunior staff

Data analysisData analysis

--specify models--specify models 1212 88 ----

--estimation--estimation 33 55 1515

--sensitivity tests--sensitivity tests 44 33 99

Results write-upResults write-up

--outline --outline developmentdevelopment

33 11 ----

--writing--writing 1010 1515 22

--table --table preparationpreparation

---- ---- 88

TOTALTOTAL 3232 3232 3434

4. Preparing the Analysis Plan 4. Preparing the Analysis Plan or RFPor RFP

• Share the statement with the client Share the statement with the client even if they are not experts in even if they are not experts in evaluation or social sciencesevaluation or social sciences

• Even if internal staff will do the Even if internal staff will do the analysis, the discipline of the analysis, the discipline of the Analysis Plan is very usefulAnalysis Plan is very useful

• The RFP should be read by the The RFP should be read by the review panel to be sure they review panel to be sure they understand it and agree with it.understand it and agree with it.

The RFPThe RFP

• Be clear aboutBe clear about– Objectives; questions to be answeredObjectives; questions to be answered– Expected products, scheduleExpected products, schedule– Sample sizes, illustrative list of locations, etc. to Sample sizes, illustrative list of locations, etc. to

facilitate analysis of budgetsfacilitate analysis of budgetsDo not give too much guidance on Do not give too much guidance on

methodologymethodologyProvide an indication of level-of-effortProvide an indication of level-of-effortResults dissemination: buy some timeResults dissemination: buy some timeInclude the evaluation criteria and weight Include the evaluation criteria and weight

assigned to eachassigned to each

5. Competition and award5. Competition and award

• Review panel is formed before the Review panel is formed before the RFP is issuedRFP is issued

• Includes people with the necessary Includes people with the necessary competence (use consultant if competence (use consultant if necessary)necessary)

• Have someone from the client’s staff Have someone from the client’s staff participateparticipate

• Formal criteria and scoring sheetsFormal criteria and scoring sheets• ““Orals” and “Best and Finals”Orals” and “Best and Finals”• Absolute confidentialityAbsolute confidentiality

Standard factors evaluatedStandard factors evaluated

• Contractor’s understanding of Contractor’s understanding of the issuesthe issues

• Quality of the methodological Quality of the methodological designdesign

• Quality of the management planQuality of the management plan• Quality of the proposed staffQuality of the proposed staff• Past experience of organizationPast experience of organization• PricePrice

6. Overseeing the contractor6. Overseeing the contractor

• Standard toolsStandard tools– Detailed review ofDetailed review of

• Work planWork plan• methodologymethodology

– Monthly progress reportsMonthly progress reports– Briefings at key pointsBriefings at key points– ““Milestone” reportsMilestone” reports– Draft final reportDraft final report

What to do if the contractor What to do if the contractor underperforms?underperforms?

• More frequent, intense reviews; be More frequent, intense reviews; be clear on areas of weaknessclear on areas of weakness

• Informal contact with senior Informal contact with senior management at the firmmanagement at the firm

• Official letter to firm stating Official letter to firm stating problemsproblems

• Legal action to deny payment (very Legal action to deny payment (very unusual)unusual)

• Important that end-of-project records Important that end-of-project records of performance be maintainedof performance be maintained

7. Getting policy impacts7. Getting policy impacts

• Work with the clientWork with the client– Don’t overlook the positive pointsDon’t overlook the positive points– Make sure they understand the basis for Make sure they understand the basis for

criticismscriticisms– Have specific recommendations for Have specific recommendations for

improvementsimprovements• Report the results to other interested Report the results to other interested

parties—but do it in a constructive parties—but do it in a constructive wayway

Sharing resultsSharing results

• Never “slant” the results or your Never “slant” the results or your credibility will be severely credibility will be severely underminedundermined

• Make sure you are informed by Make sure you are informed by the contractor about their plans the contractor about their plans for release of the resultsfor release of the results

1. Specific examples of program evaluations

A. Demonstration project—“Moving to Opportunity”

SubjectSubject

-- Using housing vouchers to help poor families leave inner-city neighborhoods for middle classareas

-- Evidence from non rigorous studies that suchmoves had significant positive impacts on the families who moved

Context & ClientContext & Client• Use of housing vouchers as a Use of housing vouchers as a

successful anti- poverty instrument successful anti- poverty instrument was very excitingwas very exciting

• Congress could mandate changes in Congress could mandate changes in the use of vouchers, increasing use the use of vouchers, increasing use for this purposefor this purpose

• Congress mandated a rigorous Congress mandated a rigorous evaluation to determine if there is a evaluation to determine if there is a sound basis for changesound basis for change

• Funds included in HUD budgetFunds included in HUD budget

Specific objectivesSpecific objectives• Determine the impacts on Determine the impacts on

participating families of moving to participating families of moving to middle-class neighborhoodsmiddle-class neighborhoods

• Impacts on both Impacts on both – adults (employment, welfare adults (employment, welfare

dependence) dependence) – children (school achievement, crime, children (school achievement, crime,

drugs)drugs)

Participants in determining Participants in determining questions & designquestions & design

• HUD-evaluation officeHUD-evaluation office• Congressional staffCongressional staff• Advisory panel, including Advisory panel, including

foundations (who contributed foundations (who contributed funds for the evaluation)funds for the evaluation)

Evaluation designEvaluation design• 10-year impact evaluation; 5-cities, 10-year impact evaluation; 5-cities,

almost 5,000 householdsalmost 5,000 households• Random assignmentRandom assignment• Three groups selected from residents Three groups selected from residents

of public or assisted housingof public or assisted housing– Families given vouchers to rent housing Families given vouchers to rent housing

in middle class areasin middle class areas– Families give vouchers to use anywhereFamilies give vouchers to use anywhere– Control group, starting in public housingControl group, starting in public housing

Measurement Example: Measurement Example: education achievementeducation achievement

• Achievement measured byAchievement measured by– Scores on series of standardized Scores on series of standardized

teststests– School performance—gradesSchool performance—grades– Advanced courseworkAdvanced coursework– Application of seniors for collegeApplication of seniors for college

Procurement—interim Procurement—interim evaluationevaluation

• Contract awarded through a Contract awarded through a competitioncompetition

• 5 firms competed5 firms competed• Evaluation board--only PDE staffEvaluation board--only PDE staff• Size of contract: $8.2 millionSize of contract: $8.2 million• 3 year contract3 year contract

Monitoring contractor Monitoring contractor performance (only interim report)performance (only interim report)

• Workplan required, including Workplan required, including detailed analysis plandetailed analysis plan

• Monthly progress reportsMonthly progress reports• Several briefings to HUD, Hill Several briefings to HUD, Hill

staff, and foundationsstaff, and foundations• HUD staff time: 1 staff yearHUD staff time: 1 staff year

Conclusions for education Conclusions for education achievementachievement

• Results are for MTO-vouchers in Results are for MTO-vouchers in comparison with the both other comparison with the both other groupsgroups

• Findings for 1994 thru 1997Findings for 1994 thru 1997• No significant effect on No significant effect on

achievement by all measuresachievement by all measures• No significant effect on the No significant effect on the

quality of schools attendedquality of schools attended

Policy impactPolicy impact• Results are very freshResults are very fresh• Interim findings; 5 years to goInterim findings; 5 years to go• Results contrary to Results contrary to

expectationsexpectations• Sufficient to prevent major Sufficient to prevent major

change in use of housing change in use of housing vouchersvouchers

• Congress will probably wait for Congress will probably wait for final resultsfinal results

A. Lending for Economic Developmentin Low-Income Neighborhoods

Subject-1Subject-1

-- HUD has several programs that lend money toprivate businesses, guarantee loans made by banksor provide credit enhancements for guaranteed loans. All involve subsidies CDBG – loans of grants funds by city through banks S.108 – loan guarantees Credit enhancements on S.108 loans

Subject-2Subject-2

-- These loans are to businesses in CDBG target neighborhoods or primarily serving poor households

--Evaluation Issue: how successful are these programs in generating economic development?

Context & ClientContext & Client

• Programs not previously Programs not previously carefully evaluatedcarefully evaluated

• Client: Office of Community Client: Office of Community Planning and Development—Planning and Development—request of the A/Mrequest of the A/M

• Request to evaluation officeRequest to evaluation office

Specific questions Specific questions (selected)(selected)• Impact of programs on business Impact of programs on business

development and job creationdevelopment and job creation• How do these loans perform?How do these loans perform?• What is feasibility of creating a What is feasibility of creating a

secondary market for these loans?secondary market for these loans?• So combination impact and process So combination impact and process

evaluationevaluation

Participants in determining Participants in determining questions & designquestions & design

• HUD-evaluation officeHUD-evaluation office• HUD program officeHUD program office

Evaluation design: business Evaluation design: business development & job creationdevelopment & job creation

MeasurementsMeasurements-- No. of jobs created vs. no. planned-- No. of jobs created vs. no. planned-- Cost per job created vs. cost per -- Cost per job created vs. cost per

job of other federal econ-develop job of other federal econ-develop programsprograms

--survival rate of businesses --survival rate of businesses compared with all small compared with all small businessesbusinesses

Program experience analyzedProgram experience analyzed

• Loans originatedLoans originated– CDBG: 1996 – 1999CDBG: 1996 – 1999– S.108: 1994 - 1999S.108: 1994 - 1999

Evaluation design: Evaluation design: loan performanceloan performance

Measurement: default rates Measurement: default rates compared with those of banks to compared with those of banks to similar businesssimilar business

-- closed loans, 1996-99-- closed loans, 1996-99-- loans still open at the time of the -- loans still open at the time of the

studystudyLoss ratesLoss rates

ProcurementProcurement

• Contract through competitionContract through competition• 3 firms competed3 firms competed• Evaluation board--only PDE staffEvaluation board--only PDE staff• Size of contract: $2.1millionSize of contract: $2.1million

– Huge data collection effortHuge data collection effort• 2.5 year contract2.5 year contract

Monitoring contractor Monitoring contractor performanceperformance

• Workplan, analysis design report, and Workplan, analysis design report, and data collection plan requireddata collection plan required

• Monthly progress reports—narrative, Monthly progress reports—narrative, issues, financial reportissues, financial report

• Two reports: Two reports: – Interim on data assembledInterim on data assembled– finalfinal

• Two briefingsTwo briefings• HUD staff time: 4 staff monthsHUD staff time: 4 staff months

Conclusions for business and job Conclusions for business and job developmentdevelopment

• Jobs created equaled 93% of those Jobs created equaled 93% of those indicated in loan applicationsindicated in loan applications

• Average cost to create one jobAverage cost to create one job– CDBG = $2,675CDBG = $2,675– S.108 (loan guarantee) = $7,865S.108 (loan guarantee) = $7,865– Compared with: Other gov’t econ Compared with: Other gov’t econ

develop: $936 - $6,250develop: $936 - $6,250• Overall positive impact on business Overall positive impact on business

survival ratessurvival rates

Conclusions for loan performanceConclusions for loan performance

• Default rates on closed loansDefault rates on closed loans– CDBG = 26%CDBG = 26%– S.108 = 42%S.108 = 42%

• Unsubsidized loans have lower Unsubsidized loans have lower default rates but are made to default rates but are made to the most creditworthy the most creditworthy borrowersborrowers

Policy impactPolicy impact• Results are only a few months oldResults are only a few months old• Likely that the program office will Likely that the program office will

use results to provide better use results to provide better guidance to cities on use of loan guidance to cities on use of loan programs, particularly the programs, particularly the characteristics of excessively risky characteristics of excessively risky loansloans

• Explicit guidance on underwriting Explicit guidance on underwriting standard may resultstandard may result

Key Steps in Managing a Key Steps in Managing a Program EvaluationProgram Evaluation

• Have a strongly interested clientHave a strongly interested client• Rigorously define the questions to be Rigorously define the questions to be

addressedaddressed• Have a general idea of the Have a general idea of the

methodologymethodology• Open competitionOpen competition• Invest in monitoring contractor Invest in monitoring contractor

performanceperformance• Work with the program office to get Work with the program office to get

the results usedthe results used