2014 AmeriCorps External Reviewer Training

Assessing Need, Theory of Change, and Logic Model

Topics

¨ Purpose of Logic Model¨ Assessing Need, Theory of Change and Logic

Model

2

3

Preview of Assessment Questions

(True/False)¨ The Logic Model is a visual representation of the

applicant’s Theory of Change.¨ External Reviewers should assess whether the

outcomes in the Logic Model are measurable.

(Multiple Choice)¨ Data about the number of economically

disadvantaged beneficiaries in a community would most likely describe:

(Practice Your Understanding)¨ Complete portion of Review Assessment Form

What is a Logic Model?

¨ A visual summary or snapshot of your program that communicates how your program works, the resources you have to operate your program, the activities you plan, and the outcomes you hope to achieve

¨ A well-specified conceptual framework that identifies key components of the proposed process, product, strategy, or practice and describes the relationships among the key components and outcomes, theoretically and operationally (Notice of Funding Opportunity (NOFO) p. 18)

¨ The purpose of a Logic Model is to describe how a program will create change.

Logic Model

4

How a Logic Model WorksLogic Model

5

Resources/Inputs Activities Outputs Outcomes/

ImpactsLong-Term

Goal

Certain resources are

needed to operate your

program

If you have access to

them, then you can use

them to accomplish

your planned activities

If you accomplish

your planned activities,

then you will hopefully

deliver the amount of

product and/or service that

you intended

If you accomplish

your planned activities to the extent

you intended, then your

participants will benefit in certain ways

If these benefits to participants

are achieved, then certain changes in

organizations, communities,

or systems might be

expected to occur

Figure 1: How to Read a Logic Model

j k l m nYour Planned Work Your Intended Results

Applicant Logic Models

¨ Inputs – Resources, including AmeriCorps members

¨ Activities (Interventions)¨ Outputs (Beneficiaries, other products of program

services)¨ Outcomes

Short Term (Knowledge, Skills, Attitudes, Opinions) Medium Term (Behavior) Long-Term (Condition)

Logic Model

6

Key Points

¨ The Logic Model and narrative work together to depict the program’s Theory of Change

¨ All the information in one doesn’t have to be in the other, as long as the two are aligned

¨ The Logic Model is simply a visual representation of the Theory of Change and program design

¨ Applicants do not measure all outcomes in Theory of Change; they continue to measure outcomes that make sense for their program; External reviewers do not assess applicants’ performance measures and so should not be concerned with whether outcomes are measurable

¨ CNCS provided a template for applicants’ Logic Models

Logic Model

7

Theory of Change ElementsAssessment: Theory of

Change

8

Community Problem/Need

Specific Intervention

IntendedOutcome

Evidence• Guides choice of intervention

• Supports cause-effect relationship

Statistics documenting

the need

How Do The NOFO Pieces Fit?

Applicant’s Overall Theory of Change¨ Need narrative¨ Theory of change narrative¨ Logic model

Assessment

9

The Selection Criteria & Standards

Four Key Themes¨ Completeness (Logic Model)¨ Alignment/Logic¨ Quality of Theory of Change¨ Level/Quality of Evidence

Assessment

10

Review Rubric

¨ Most items scored on a five-point scale¨ Some items scored on a two-point scale

(Satisfactory/Unsatisfactory)¨ Refer to the Review Rubric in the handbook to

read the descriptions for each Rating

Assessment

11

Completeness (Logic Model)

¨ Number of locations and sites where members are providing services

¨ Number of members that will be delivering the intervention

¨ Core activities that define the intervention or program model

¨ Duration of the intervention¨ Dosage of the intervention¨ Target population for the intervention

Assessment

12

How Assessed

¨ Series of check boxes on the review form¨ Reviewers check the box if the item is present in

the Logic Model

Assessment

13

Alignment with Logic Model

¨ The Theory of Change is described consistently in the Logic Model and application narrative

¨ The inputs, activities and outputs are logically aligned

¨ The Theory of Change and Logic Model cover comprehensively the applicant’s entire program

Assessment: Theory of Change

14

Quality of Theory of Change

Problem¨ The community problem/need is prevalent and

severe in the communities where members serve¨ The applicant cites specific, relevant data to

document the needIntervention¨ The intervention is an effective way to address

the problem/need identified by the applicantOutcomes¨ The outcomes identified in the Logic Model are

logically aligned with the problem/need and intervention

Assessment: Theory of Change

15

Assessing Alignment and Quality

Alignment and Quality are assessed using a five-point scale:¨ Greatly exceeds the standard¨ Exceeds the standard¨ Meets the standard¨ Partially meets the standard¨ Does not meet the standard at allRefer to the Review Rubric to read the descriptions for each Rating.

Assessment: Theory of Change

16

Need vs. Evidence

Applicants are asked to provide data to demonstrate:¨ Prevalence and severity of need versus the level

and quality of evidence¨ Specific, relevant data to document the need¨ Evidence to support the choice of intervention

Assessment: Need

17

Data documenting problem/need should demonstrate the prevalence and severity of the problem:

Who and how many are directly affected?

How severe is the problem?

18

Community Problem/NeedAssessment: Need

Examples

¨ Unemployment rate¨ Poverty rate¨ Percent of students eligible for free/reduced lunch¨ Population or size of area affected by natural

disaster¨ Percent of adult population with diabetes

Assessment: Need

19

Data Should Be

¨ From a reputable source such as a government agency, foundation, university or research organization

¨ Recent ¨ Local

Assessment: Need

20

Common Issues

¨ State or national data is provided instead of local data describing the need in the specific communities where the service will occur

¨ Statistics are unrelated to the problem/need¨ Statistics cited are not recent¨ Data sources are not identified or are not

reputable

Assessment: Need

21

Next Steps

To check for understanding and verify that you have completed this orientation session, please reference the remaining slides for the two-part Assessment of this Orientation.

22

Practice Your Understanding

¨ Answer the Assessment Questions at the following link: https://www.surveymonkey.com/s/logicmodels_assessment

¨ Read and assess the norming/sample application and complete the Rating sections for the Problem/Need, Theory of Change, and Logic Model sections of the review form. Do not include the significant strengths, weaknesses, or other narrative sections.

¨ You will receive a review form completed by a CNCS staff member. Compare your assessments to those in the key.

¨ This will complete your verification requirement for this training module.

23

Assessment: Practice

Practice

SELF-ASSESSMENTWhen you receive the key for the practice activity from CNCS, compare your Ratings to those of the CNCS standard example.

If your Rating was more than one Rating higher or lower than the example provided by CNCS, your assessment may not be normed closely enough to CNCS’ expectations for applicants

Assessment: Practice

24