EVALUATION PLANNINGLegal Services Corporation

May 19, 2004

Please standby while others connect

Slide 2Innovation Network, Inc.

Goals for the Session

Review the LSC evaluation plan template Learn how to formulate strong evaluation questions Introduce LSC’s evaluation questions Give you the skills to understand which data collection

techniques are best suited to answering what kinds of evaluation questions

Gain a better understanding of how program assessment can help legal services organizations improve services

Slide 3Innovation Network, Inc.

Objectives

At the end of the training grantees will be able to: Complete the LSC evaluation template Integrate LSC’s required evaluation questions with questions you

develop on your own Deploy a data gathering strategy that helps you answer the

questions you created

Slide 4Innovation Network, Inc.

Ground Rules

Using Isoph Blue’s features We ask that you not use a speaker phone unless you have a

mute button to block external sounds. Please do not use the “Hold” function on your phone, as it can

induce a loud beeping noise for the rest of us on the call.

Slide 5Innovation Network, Inc.

Questions During the Session

If you can, please hold your questions till we get to a question break

If you can’t wait, please click the question mark and type your question into the window, we’ll answer as soon as we can.

If we do not get a chance to answer your question, please email it to Ehren Reed at ereed@innonet.org.

Slide 6Innovation Network, Inc.

What is Evaluation?

The systematic collection of information about a program in order to better understand it, improve its effectiveness, replicate it and/or make decisions about future programming.

Slide 7Innovation Network, Inc.

Benefits of Evaluation

Effectively communicates successes Builds common understanding about the program Helps you improve what you do Turns shortcomings into opportunities Helps staff overcome obstacles Positions your organization as a learning organization

Slide 8Innovation Network, Inc.

10 Steps to Evaluation

1. Develop program goals2. Identify specific goal targets 3. Outline strategies and activities4. Develop evaluation questions5. Create data collection instruments6. Collect data7. Analyze and interpret information8. Present findings9. Reflect on findings10. Revise program & institute changes

Slide 9Innovation Network, Inc.

Evaluation Plan

Project Goal(s) Strategies/ Activities to

Achieve Goals

Evaluation Questions

Evaluation Data

The overall aim(s) or intended impact(s) of your project

The actions that the project takes to achieve desired outcomes

The questions you want to answer about your project

The evidence or information that tells you whether or not your project is achieving its outcomes

Slide 10Innovation Network, Inc.

Project Goals

Clear statements identifying the ultimate results you want to achieve

Slide 11Innovation Network, Inc.

Goals & Objectives

Develop objectives or more specific targets for your goals. Consider the major steps will you need to accomplish in order

to achieve each of them. Make your objectives specific and achievable. Some goal words:

Enhance Establish Enable

Tip: Review your grant proposal to see what you said you would accomplish

Slide 12Innovation Network, Inc.

Some Intake Examples

Intake Goal:

Enable Asian-American clients in northern Virginia to seamlessly access legal services through a single, integrated intake system

Objectives: Design integrated system that allows partner organizations to

access critical client information

Slide 13Innovation Network, Inc.

Some Infrastructure Examples

Infrastructure Goal:

Enhance the delivery of legal services to low-income families in rural Mississippi via the website through high quality web applications

Objectives: Develop partnerships with providers and case managers to identify critical

needs Increase access to services by integrating communication systems

Slide 14Innovation Network, Inc.

Some Pro Se Examples

Pro Se Goal:

Enhance the ability of poor people in Ohio to access quality legal services to assert their rights and interests through online and in-person training and technical assistance

Objectives: Identify patterns and causes of fundamental legal programs affecting our

client community Develop and test solutions to recurring legal problems faced by our client

community Ensure appropriate and sustainable web-based access to legal services

Slide 15Innovation Network, Inc.

Strategies & Activities

The major actions a program takes to achieve desired outcomes Use action words like:

Conduct Train Mentor Provide Support

Slide 16Innovation Network, Inc.

Some Intake Examples

Intake Develop database program that gathers case information

and assesses eligibility for legal aid Develop process for integrating systems and databases in

various offices that conduct intake, provide legal advice, and represent clients

Train case workers to use the system Identify interpreters for mono-lingual clients

Slide 17Innovation Network, Inc.

Some Infrastructure Examples

Infrastructure Identify potential partners Hold regular meetings with partners and document community

needs Develop web-based services Train providers in how to use web-based services

Slide 18Innovation Network, Inc.

Some Pro Se Examples

Pro Se Prepare training materials for template developers Participate in national template developers training Prepare logistics for Ohio template developers training Conduct Ohio template developers training Organize multi-state template developer work group

Slide 19Innovation Network, Inc.

Questions

Slide 20Innovation Network, Inc.

Work on Strategies and Activities

Take a look at your grant proposal or work from memory and complete the first two columns of the LSC Evaluation Plan grid.

List your activities and organized by category or strategy. Remember to use appropriate action words.

When you’re done, click in the check box next to you name

Slide 21Innovation Network, Inc.

Check in / Debrief / Discussion

Slide 22Innovation Network, Inc.

Evaluation Questions

Identify broad questions that will guide your evaluation report. Process questions: What did you do and what were the

tangible results of your work? Outcome questions: What change occurred among your

constituents as a result of your activities? Try to keep the questions open-ended

Slide 23Innovation Network, Inc.

Some Intake Examples

Intake How has the intake system assisted clients in accessing

seamless services? To what extent has the new system increased efficiency among

intake staff?

Slide 24Innovation Network, Inc.

Some Infrastructure Examples

Infrastructure To what extent has the website helped strengthen client

education? How have the partnerships supported the development of quality

content for the website? How has the website increased access to legal services for rural

clients? In what ways has this project been supported by LSC and the

organizations partners?

Slide 25Innovation Network, Inc.

Some Pro Se Examples

Pro Se To what extent did the training provide necessary skills and

knowledge to achieve intended outcomes? Was the training curriculum and delivery offered of high quality? How well have the online services met the needs of the

community and addressed the current trends in necessary legal assistance?

Has our online process and materials adequately prepared clients to accurately complete their legal forms?

Slide 26Innovation Network, Inc.

What are Your Questions?

What information do you want to know about your program? Think about the assumptions that you have made about how

your program will be planned and implemented. Draft questions that you want to answer about your program.

Tip: Avoid Yes/No questions. Instead, ask when, how, how much or to what extent

Slide 27Innovation Network, Inc.

Check in / Debrief / Discussion

Slide 28Innovation Network, Inc.

Evaluation Data

Talk to people Interview, focus group, facilitated discussion

Gather written responses Surveys, questionnaires, paper tests

Observe Individuals, communities, group activities

Review documents Attendance records, progress reports, journals

Slide 29Innovation Network, Inc.

….and the source

Where will you find the information? What target group or documents will you need to gather the

necessary information?

Slide 30Innovation Network, Inc.

Some Intake Examples

Intake Survey of intake staff Focus Group with mono-lingual clients Exit survey of mono-lingual clients

Slide 31Innovation Network, Inc.

Some Infrastructure Examples

Infrastructure Monthly update form completed by project staff Survey of rural clients/users Survey of case workers Content review of website

Slide 32Innovation Network, Inc.

Some Pro Se Examples

Pro Se Survey clients Document review/Content review of online process

Slide 33Innovation Network, Inc.

Finish Your LSC Evaluation Plan

List the data collection methods and source of information for each one of your evaluation questions

Remember to keep in mind the following questions: Who or what has the answer to each evaluation question (partners,

clients, records, etc What is the best way to access it (survey, focus group, document

review, etc) Tip: Think about what information you are already gathering and

try to integrate those tools into this data collection effort.

Slide 34Innovation Network, Inc.

Evaluation Plan

Project Goal(s)

Strategies/

Activities

Evaluation Questions

Evaluation Data

Slide 35Innovation Network, Inc.

Check in / Debrief / Discussion

Slide 36Innovation Network, Inc.

Questions

Slide 37Innovation Network, Inc.

Next Steps

Let us know where you are in the process Have you completed your plan? Do you want us to review it? Are you ready to create data collection instruments?

Slide 38Innovation Network, Inc.

We can provide more assistance

Asynchronous training Individual technical assistance Synchronous training

Potential topics include: Survey Document review

Slide 39Innovation Network, Inc.

For More Information

Innovation Network, Inc.1001 Connecticut Ave., NW, Suite 900

Washington, DC 20036(202) 728-0727

www.innonet.org