1

Evaluating Partnerships:Why, When and How

February 17, 2015

Presented in Collaboration with the Centers for Disease Control and Prevention

B. J. Tatro, PhDASTDD, bj@bjtatro.com

2

General Reminders

• This webinar will be recorded and archived on the ASTDD website (including speaker notes)

• Questions will be addressed after the speakers are finished. Please type your question into the “chatbox” that will appear at the end of the webinar and then click on the bubble to the right of where you type your question to send it to the moderator

• Please respond to the polling questions at the conclusion of the webinar

3

• This presentation was supported by Cooperative Agreement 5U58DP004919-02 from CDC, Division of Oral Health. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC

4

Desired Results

• State Oral Health Programs (SOHPs) have increased awareness of the importance of evaluating their partnerships and of engaging their partners in the evaluation process

• SOHPs have increased awareness of the importance of matching their partnership evaluation questions and tools to their partnership’s stage of development

• SOHPs are familiar with resources available to them to inform their planning, implementation, and use of partnership evaluation

5

Some Preliminary Thoughtson Partnerships

• What are they?– Nature of the partnership– Purpose of the partnership

6

Some Preliminary Thoughtson Partnerships

• Why do we need them?• What does it take to develop, maintain, and

grow a successful partnership?

7

Some Preliminary Thoughts on Evaluating Partnerships

• Why evaluate partnerships?• How do we know if we are ready to evaluate a

partnership?• How do we approach the design and

implementation of a partnership evaluation?

8

Step 1: Engage Stakeholders

• Identify who should be involved in the evaluation

• Specify the roles they will play• Think about how to engage these stakeholders

throughout the evaluation process

9

Step 2: Describe the Partnership

• Develop your logic model

10

State of HawaiiPurpose = Develop & coordinate partnerships to improve oral health

Inputs Activities Outputs Outcomes

Hawaii Oral Health Team (staff) Internal PartnersFamily Health Services Division Programs: Office of Primary Care & Rural

Health CSHCN MCH (including MIECHV, Parenting

Support, Adolescent & Youth, etc.) WICOther DOH: Chronic Disease Management &

Control Branch (Asthma, Cancer, Diabetes, Heart/Stroke, Obesity, Tobacco)

Office of Planning, Policy & Program Development (Preventive Health & Health Services Block Grant)

Developmental Disabilities

External PartnersState Medicaid OfficeHawaii Federally Qualified Health Centers (FQHC) & Rural Health Clinics (RHC)Hawaii Chapter of the American Academy of PediatricsInsurance providersLegislatorsCommunity representatives CDC Funding & TA ASTDD TA

1. Identify priorities & establish, enhance, & maintain a collaborative relationship with internal DOH partners, with priority focus on CDC-funded chronic disease programs, CSHCN, & MCH 2. Identify priorities & establish, enhance, & maintain a collaborative relationship with external partners, with priority focus on FQHCs & RHCs For both internal & external partners:

a. Identify contactsb. Conduct 1:1 outreachc. Become familiar with each other’s mission & activitiesd. Identify shared interests & common goalse. Identify potential benefits of working togetherf. Identify options for collaborative effortsg. Make decision about whether & how to collaborateh. Establish partnership objectives & develop work plansi. Implement work plansj. Evaluate collaborative strategies

a. Contacts identified, documented b. Communication established, documented c. Information shared, documented d. Shared interests & common goals identified, documented e. Benefits identified, documented f. Options identified, documented g. Decision made, documented h. Objectives established & work plans developed i. Work plans implemented, documented j. Evaluation conducted, documented

Short TermOngoing communication & collaborative action Increased awareness of the relevance of oral health to partners’ target populations Increased awareness of the benefits of prevention, early identification, & treatment to overall health

Intermediate TermOral health addressed in partners’ burden documents if applicable, activities, messaging, plans, policies, &/or practices Resources devoted to oral health expanded/ leveraged

Longer TermIncreased number & commitment of oral health champions Increased attention to prevention, early identification, & treatment Increased number of children have dental sealants

ImpactReduced dental caries, oral cancer, periodontal disease Improved health equity Improved overall health

11

Step 2: Describe the Partnership (continued)

• Identify what you need and what you have

12

Partnership Membership Assessment Tool from Fundamentals of Evaluation Partnerships/Evaluation Guide (National Health Disease & Stroke Prevention Program)

Want Have Roles Want Have Skills/Expertise Want Have Representation

Partnership roles Data analysis, worksites State Emergency Services

Leader Data analysis, healthcare State Obesity Program

Committee leader Reviewer, medical content State Diabetes Program

Task leader Writer State Tobacco Program

Meeting planner Advocate for stroke State Epidemiology

Meeting facilitator Advocate for heart disease State Office of Minority Health

Strategic planner Legislative advocate Hospital Association

Communications Medical expert Primary Care Association

Training Cardiologist State Legislature/Policy Makers

Financial support Neurologist Schools (as worksites)

Content reviewer Healthcare quality improvement Community health clinics

Budget management Nursing Private insurers

Spokesperson Pharmacy Medicaid/Medicare

Funder Media communications Prevention Research Center

Champion, healthcare Workplace wellness Chambers of Commerce

Champion, public health State policy change Unions

Champion, worksites Community policy change Business coalition on health

Strategic implementer Training for healthcare State American Heart Association

Resource linker – connection to groups with influence or resources

Evaluation Disparate groups (race/ethnicity, geographic, gender. SES. Etc.)

Marketing

13

Step 2: Describe the Partnership (continued)

• Identify the stage of development of your partnership– Formation– Building– Maintenance

• Engage stakeholders

14

Step 3: Focus the Evaluation Design

• Develop and prioritize evaluation questions that are matched with your partnership’s stage of development

• Engage stakeholders

15

Step 4: Gather Credible Evidence

• Complete your evaluation plan:– Indicators– Data Sources (tools)– Methods of data collection– Frequency of data collection– Study design– Implementation roles, responsibilities, and

timelines• Engage stakeholders

16

State of Minnesota

Evaluation QuestionIndicator/Performance Measure Method Data Sources Frequency Responsibility

How were the opportunities identified and leveraged to integrate oral health efforts with other programs that address related chronic disease/risk factors?

# of meetings, campaigns, summits organized

Meeting notes, summit reports review

Meeting notes and summit reports Annually

Program coordinator/ evaluator

How many partnerships were developed with a focus on prevention interventions in a year?

# of partnerships that focus on prevention intervention and that provided a letter of support or MOA

Program reports review

Letters of support

Program reports

Partnership database

AnnuallyProgram coordinator/ evaluator

How was the integration of oral health into overall health encouraged/institutionalized?

# of meetings with other chronic disease partners

Meeting notes and reports review

Meeting notes and reports Biannually

Program coordinator/ evaluator

How was the documentation provided for collaboration with partners outside state health department?

# of meetings, meeting notes, emails, letters indicating agreement to collaborate

Meeting notes and reports review

Meeting notes and reports Biannually

Program coordinator/ evaluator

What are types of activities are partners involved in?

# of specific tasks accomplished

Meeting notes and reports review

Meeting notes and reports Biannually

Program coordinator/ evaluator

What are the reasons for long term sustained partnerships?

List of reasons for long term sustained partnerships Partners’ survey Survey questionnaire

Once in a grant period

Program evaluator

What are the challenges partners feel that hinder intended progress?

List of reasons for long term sustained partnerships Partners’ survey Survey questionnaire

Once in a grant period

Program evaluator

17

Evaluation QuestionIndicator/Performance Measure Method Data Sources Frequency Responsibility

How does the coalition focus on the following?• Oral health infrastructure

development and sustainability• State oral health plan

revision and implementation

• School-based/-linked sealant programs

• Community water fluoridation efforts

Coalition initiatives in the areas of • Oral health

infrastructure development and sustainability

• State oral health plan revision

• School-based/-linked sealant programs

• Community water fluoridation

Meeting notes and reports review

Meeting notes and reports Biannually

Program coordinator/ evaluator

What sectors of CDC coalition diversity framework do coalition members represent?

# of members from each sector of CDC coalition framework

# of members from each sector attending coalition meetings

Coalition member roster

Member group is identified on the membership application

Coalition member roster

Stakeholder database

On-goingProgram Coordinator

Data Coordinator

What were the steps taken to facilitate institutionalization and sustainability of coalition?

501c3 status, written vision/mission statement, priorities /plans/strategies Consistency of membership

Review bylaws and coalition work plan By-laws and work plan Annually

Coalition members/ program coordinator

Which programs and policy changes have resulted from the coalition efforts?

# of policy changes as a result of coalition efforts

List of program initiated

Review policy documents

Review reports

Policy document

Coalition reports

Annually

Annually

Coalition members/ program coordinator

18

Evaluation QuestionIndicator/Performance Measure Method Data Sources Frequency Responsibility

What types of communication are being used by partners?

Types of communications by partners

# of member that feel the communication is adequate

Coalition website updated quarterly

Review reports

Member survey and review of agenda, minutes and reports

Use of website for communications

Progress reports

Member survey and Coalition agendas, minutes and reports

Coalition website

On going

On going

On going

Coalition secretary/ program coordinator

How do collaborative partners focus on oral health issues and prevention strategies as outlined in the state oral health plan?

# of partner involved in initiating new activities outlined in state oral health plan

# of partner participating in activities outlined in state oral health plan

Review program progress reports

Program progress reports Quarterly

Program coordinator/ evaluator

What are the forms of documentation of commitment or support available from stakeholders e.g. delineating accountability and clear lines of responsibility for activities?

Letters of support from stakeholders

Description of roles and responsibilities of partners for activities

Review coalition reports

Review of coalition manual

Coalition reports

Coalition manual

AnnuallyProgram coordinator

How did partnerships influence oral health policies, practices and systems?

List of policies, practices and systems influence

Review report and survey results

Annual reports and partners’ survey

Once in the grant period

Program evaluator

19

Step 5: Justify Conclusions

• Ensure accuracy of data and data entry• Analyze the data• Make sense of what you have learned• Develop recommendations• Engage stakeholders

20

Step 6: Ensure Use and Share Lessons Learned

• Develop a communication plan• Communicate results• Incorporate into ongoing planning• Engage stakeholders

21

Resources• ASTDD website (slides/speaker notes from this webinar will also be posted on the

ASTDD website) and B. J. Tatro (ASTDD evaluation consultant) bj@bjtatro.com There are also a variety of resources related to oral health collaborations on the site.

• Collaboration: What Makes It Work (2nded), P. Mattessich, M. Murray-Close, & B. Monsey (Fieldstone Alliance, 2001) Includes The Wilder Collaboration Factors Inventory. This publication includes a review of the literature on factors influencing successful collaboration and The Wilder Collaboration Factors Inventory TOOL.

• Collaborative Effectiveness Assessment Activity (Prevention Institute, 2004) This TOOL is designed to stimulate thinking about some elements of effective collaboration.

• Evaluating Partnerships to Prevent and Manage Chronic Disease, F. Butterfoss, Preventing Chronic Disease, April 2009, Vol. 6, No. 2 This article describes how to develop a comprehensive evaluation strategy based on partnership theory, select outcome indicators, choose methods and tools, and use results for accountability and improving effectiveness.

22

Resources• Evaluation Guide: Fundamentals of Evaluating Partnerships, National Heart

Disease & Stroke Prevention Program (Centers for Disease Control and Prevention, 2008) This is one of a series of evaluation technical assistance tools. The guides supplement the CDC program evaluation framework. This particular guide provides information on how to implement the framework when evaluating partnerships at various levels (annual assessment, basic evaluation, and enhanced evaluation). Appendices include examples of evaluation questions, a variety of TOOLS, and other helpful information.

• Framework for Evaluation (CDC, 2012) CDC provides a six-step framework for conducting program evaluation. CDC also offers guides for developing logic models and developing evaluation plans; see Developing and Using a Logic Model and Developing an Effective Evaluation Plan

• Guide to Evaluating Collective Impact, H. Preskill, M. Parkhurst, & J. Splansky Juster (FSG, Collective Impact Forum, 2014) In addition to three guides, this site links the user to a knowledge exchange related to collective impact.

23

Resources• Handbook on Planning, Evaluating and Improving Interagency Collaboration in St

ate Oral Health Programs (ASTDD, 2012) The handbook and companion workbook provide a framework for evaluating collaboration between state oral health programs and other entities. A variety of TOOLS, as well as oral health examples, are provided.

• Measuring Collaboration Among Grant Partners, B. Frey, J. Lohmeier, S. Lee, & N. Tollefson, American Journal of Evaluation, 2006, 27, 383 This TOOL can be used to assess levels of collaboration—ranging from networking to collaboration for each partner.

• PARTNER This is an online social network analysis tool designed to collect, analyze, and interpret data to improve collaboration within community networks.

24

Resources• Partnership Evaluation: Guidebook and Resources, National Center for Chronic

Disease Prevention & Health Promotion, Division of Nutrition, Physical Activity & Obesity (Centers for Disease Control and Prevention, 2011) This is one of a series of evaluation technical assistance tools. The guides supplement the CDC program evaluation framework. This particular guide provides information on how to implement the framework when evaluating partnerships. Appendices include a variety of TOOLS and other helpful information.

• Partnership Self-Assessment Tool (Center for the Advancement of Collaborative Strategies in Health, 2002) This TOOL can be used to assess how well a collaborative process is working and to identify specific areas for improvement. It is designed for partnerships of five or more that have been in existence for at least six months.

• Tools for Partnership Evaluation (CDC, 2012) This “coffee break” webinar was presented by S. Ladd, CDC Division for Heart Disease and Stroke Prevention Applied Research and Evaluation Branch. It covers three types of TOOLS—evaluability assessment, membership and roles, and process and functioning.

25

Thank You!

Thank you to all who provided information for and input into this webinar, including:– Harder+Company, for sharing a matrix of

partnership/collaboration evaluation tools that they researched and prepared

– John Shoemaker, for sharing a variety of partnership/collaboration tools and other resources

– The many CDC-funded states that provided copies of their partnership/collaboration evaluation logic models and evaluation plans, especially those who allowed us to draw examples from their work

And thanks to all of you who participated today!

26

Polling Questions• Are you planning to do a partnership

evaluation within the next few years?– Yes, No, Not Sure

• Have you developed your partnership evaluation plan yet?– Yes, No, Not Applicable (no study planned)

• What could ASTDD do to help you plan/conduct your partnership evaluation?