Focusing an EvaluationBen Silliman, Youth Development Specialist

NC 4-H Youth Development

Keys to Focusing an Evaluation

Discerning readiness Determining purpose Deciding on level/type Distilling questions Developing plan Directing implementation

Discerning ReadinessEvaluate no program before its time

Internal ChemistryClear objectivesCapable, trained staffRecruitment of target audienceTime to implement and refine programTime and money to conduct evaluationExpertise to plan, gather, analyze data

Discerning ReadinessEvaluate no program before its time

External ChemistryClear objectives—understood by stakeholdersConstructive relationships, staff and stakeholdersReasonable timeline for program developmentResources and assistance to evaluateSupport for gathering, analyzing, using data

Determining PurposeSweeten to taste

Stakeholder PerspectivesCitizens: Does it make a difference?Funding organization: Is it cost-effective?Parents: Is my child safe? learning?Staff: Are we running a quality program?Youth: Are we having a good time?

Determining PurposeSweeten to taste

Evaluation Standards that shape purposeUtility, serving the needs of intended usersFeasibility, examining if the evaluation is realistic,

prudent, diplomatic, practical Propriety, assessing legal, ethical, in the best

interests of all stakeholders Accuracy, technically and broadly conveying the

worth of the program

Deciding on Level/TypeChecking the vintage: Taste test

Formative EvaluationOngoing feedback about performance needs assessment to determine target audience, needs,

best strategies evaluability assessment to determines feasiblliiy, usefulness structured conceptualization to define the program or

technology, the target population, and the possible outcomes implementation evaluation monitors the fidelity of the

program or technology delivery process evaluation investigates the process of delivering the

program or technology, including alternative delivery procedures

Source: Research Methods Knowledge Base

Deciding on Level/TypeChecking the vintage: Taste test

Summative EvaluationLong-term effects or outcomes of a program outcome evaluations explore whether and how much a program

produces targeted results impact evaluation examines the broader effects of the program or

technology as a whole cost-effectiveness and cost-benefit analysis address questions

of efficiency by standardizing outcomes in dollar costs and values secondary analysis reexamines existing data to address new

questions or use methods not previously employed meta-analysis integrates the outcome estimates from multiple

studies to arrive at an overall or summary judgement on an evaluation question procedures

Source: Research Methods Knowledge Base

Distilling QuestionsMade to Order

Program Quality and ProcessBest practices/Youth experienceProgram improvement

Distilling QuestionsMade to Order

Youth OutcomesShort-term: Knowledge-Attitude-Skill-AspirationsMid-term: Behavior Change-ApplicationLong-term: Social-Economic-Environment-Cultural

Change

Developing a PlanTapping the Casks

Selecting Indicators (examples)Health: healthy weight; appropriate exerciseScience: skill in soldering, selecting seeds, calf careCitizenship: knowledge/skill in parliamentary

procedure

Developing a PlanTapping the Casks

Selecting IndicatorsHealth

diet/calorie journalmeal plan for retreat

Scienceaspirations for learningself-efficacy

Citizenship: communityservice plantime priorities

          

Developing a PlanTapping the Casks

Data SourcesChildYouth leaderParentsOthers impacted by child’s work (team members or

helpees)Generic data (School records, community

Developing a PlanTapping the Casks

Data Collection OptionsTest-Survey-InterviewPortfolio-DemonstrationObservation

Developing a PlanTapping the Casks

Data Collection ProtocolEthicalEducationalEffective

               

Developing a PlanTapping the Casks

Analyzing dataQualitative: Trends, anecdotes, commentsQuantitative: Individual growth, group patterns;

correlated elements

Directing ImplementationManaging the Vineyard

Step-by-Step Procedures Budgeting Timeline Monitoring roles Critique

You can’t find out what you want to know until you know what you want to find.