jsgs 824: health program planning & · pdf file jsgs 824: health program planning &...
TRANSCRIPT
w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a
JSGS 824: HEALTH PROGRAM PLANNING &
EVALUATION -WINTER 2018
UNIVERSITY OF REGINA CAMPUS UNIVERSITY OF SASKATCHEWAN CAMPUS
INSTRUCTOR: Jasmin Bhawra, MSc
PHONE: Skype: jasmin.bhawra
E-MAIL: [email protected]
OFFICE HOURS: Web Delivered
OFFICE LOCATION: WEB delivered course
TERM: Winter 2018
ROOM:
DATE AND TIME: January 8 - April 13, 2018
CALENDAR DESCRIPTION
This course is designed to provide the key principles of health program planning and evaluation to
health professionals who aspire to inform program development in their respective fields of expertise.
The areas of focus for this course include program planning models; design of evaluation plans; program
theory; mixed-methods data collection; indicators and measures of program outcomes; special topics
including Indigenous evaluation and policy evaluation, as well as evaluation reporting. By the end of the
course, students will be familiar with a variety of tools used in program planning and evaluation, and will
have gained experience in developing an evaluation plan.
LEARNING OBJECTIVES
Understand the relationship between program planning, research, and evaluation Develop a knowledge base of evaluation that could be used within multiple contexts of health
planning and programming Explore pros/cons and application of multiple evaluation approaches and types Demonstrate knowledge of mixed methods in conducting program evaluation Design an evaluation plan for a public health program, including the development of a logic model
(Insert learning objectives)
ATTRIBUTES OF JSGS GRADUATES
w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a
1. Management, Governance, and Leadership: Ability to inspire support for a vision or course of
action and successfully direct the teams, processes, and changes required to accomplish it.
2. Communication and Social Skills: Ability to communicate effectively and build enduring, trust-
based interpersonal, professional relationships.
3. Systems Thinking and Creative Analysis: Ability to identify key issues and problems, analyze
them systematically, and reach sound, innovative conclusions.
4. Public Policy and Community Engagement: Ability to understand how organizational and public
policies are formulated, their impact on public policy and management and how to influence
their development.
5. Continuous Evaluation and Improvement: Commitment to on-going evaluation for continuous
organizational and personal improvement.
6. Policy Knowledge: Ability to analyze and contribute content to at least one applied policy field.
COURSE CONTENT AND APPROACH
All course material will be posted on the course website: https://urcourses.uregina.ca/. PowerPoint
presentations will be posted on Wednesday every week (except for topics spanning two weeks).
Students are encouraged to use the discussion board to ask questions and engage in conversations
with their peers about the weekly modules (Insert course content and approach)
COURSE OUTLINE AND ASSIGNMENTS
MODULE 1- Introduction to Program Planning and Evaluation Week of January 8
Topics:
Characteristics of program planning and evaluation
Key definitions (i.e., program, stakeholder, target group, etc.)
Rationale for program planning/evaluation
Approaches to program planning
Social, political, and economic context considerations
Theories, frameworks and models
Program planning cycle
Difference between research and evaluation
Readings:
1. Fraser Health. 2009. A Guide to Planning and Conducting Program Evaluation. Section 2.0 – Background on Program Evaluation p. 10-15.
2. The Health Communication Unit - Centre for Health Promotion University of Toronto. 2001. Introduction to Health Promotion Program Planning. p.1-7.
w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a
3. Kegler MC, Rigler J, Honeycutt S. 2011. The role of community context in planning and implementing community-based health promotion projects. Evaluation and Program Planning, 34:246-253.
4. Levin-Rozalis M. 2003. Evaluation and research: Differences and similarities. Canadian Journal of Program Evaluation, 18(2):1-31.
5. Crosby R, Noar SM. 2011. What is a planning model? An introduction to PRECEDE-PROCEED. Journal of Public Health Dentistry, 71:S7-S15.
6. Tramm R, McCarthy A, Yates P. 2011. Using the Precede-Proceed model of health program planning in breast cancer nursing research. Journal of Advanced Nursing, 68(8):1870-1880.
MODULE 2- Planning a Program and an Evaluation Week of January 15
Topics:
Question(s)/problems, target populations, program goals and objectives,
Stakeholder relationships
Evaluability assessment
Internal vs. external evaluation
Introduction to logic models
Readings:
1. Dunet DO & Reyes M. 2006. Stakeholder-focused evaluation of an online course for health care providers. Journal of Continuing Education in the Health Professions, 26(4):257-67.
2. March of Dimes (2004). “SMART” objectives. 3. Fraser Health. 2009. A Guide to Planning and Conducting Program Evaluation: Section 3.0 –
Preparing for Evaluation and Section 4.0 – Develop an Evaluation Plan. p. 16-35. 4. Craig P, Campbell M. 2015. Evaluability assessment: A systematic approach to deciding whether
and how to evaluate programs and policies. What Works: Scotland. 5. Additional reading (in-depth description of the steps from the slides): Public Health Ontario.
2015. Planning health promotion programs: Introductory workbook. (4th Edition). Toronto, ON: Queens Printer for Ontario.
MODULE 3- Program Theory and Logic Models Week of January 22
Topics:
What is program theory
How evaluation questions flow from program theory
Program theory assessment
Development of logic models
Examples of theories relevant to health practice
w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a
Readings:
1. Tudor-Locke C, Myers AM, Rodger NW. 2001. Development of a theory-based physical activity intervention for individuals with Type 2 Diabetes. The Diabetes Educator, 27(1):85-93.
2. Astbury B & Leeuw FL. 2010. Unpacking black boxes: mechanisms and theory building in evaluation. American Journal of Evaluation, 31(3):363-381.
3. Chen H. 2003. Theory-driven approach for facilitation of planning health promotion or other programs. The Canadian Journal of Program Evaluation, 18(2):91-113.
4. Chapter 1: Introduction to Logic Models. 5. W.K. Kellogg Foundation. 2004. Logic Model Development Guide.
MODULE 4- Evaluation Approaches Week of January 29
Topics:
Overview of evaluation approaches
Use and purpose of approaches in guiding program evaluation
Examples of evaluation approaches:
Utilization-focused evaluation
Participatory evaluation
Appreciative inquiry
Developmental evaluation
Readings:
1. Patton MQ in Evaluation Models. (2000). Chapter 23: Utilization-focused Evaluation. 2. Patton MQ. (2013). Utilization-focused Evaluation Checklist. 3. Sette C. Participatory Evaluation. Better Evaluation website. Available at:
http://www.betterevaluation.org/en/plan/approach/participatory_evaluation 4. Minian N et al. Ontario Tobacco Health Unit: Using the principles of participatory evaluation to
evaluate tobacco smoking in Toronto’s Church and Wellesley neighbourhood. PowerPoint Presentation.
Additional readings:
5. Ramirez R & Brodhead D. (2013). Utilization-focused evaluation: A primer for evaluators. 6. MacLellan-Wright MF, Patten S, M. dela Cruz A, Flaherty A. (2007). A participatory approach to
the development of an evaluation framework: process, pitfalls, and payoffs. CJPE, 22(1):88-124. 7. Coghlan AT, Preskill H, Tzavaras Catsambas T. (2003). An overview of appreciative inquiry in
evaluation. New Directions for Evaluation, 100:5-20. 8. Reed J, Pearson P, Douglas B, Swinburne S, Wilding H. (2002). Going home from hospital – an
appreciative inquiry study. Health and Social Care in the Community,10(1):36-45. 9. Krasner MS et al. (2009). Association of an educational program in mindful communication with
burnout, empathy, and attitudes among primary care physicians. JAMA, 302(12): 1284-93. 10. Dozois E, Langlois M, Blanchet-Cohen N. (2010). A practitioner’s guide to developmental
evaluation. JW McConnell Family Foundation; International Institute for Child Rights and Development.
w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a
MODULE 5- Types of Evaluation Week of February 5 and February 12
(spans 2 weeks)
Topics:
Formative evaluation
Needs assessments
Process (implementation) evaluation
Outcome (effect) evaluation
Cost evaluations and economic analyses
Examples from the field of public health
Readings:
1. Langevin CM. 2001. An evaluation framework for the maison decision house substance abuse treatment program. Canadian Journal of Program Evaluation, 16(1):99-129.
2. Mercer NJ. 2009. Cost analysis of public health influenza vaccine clinics in Ontario. Canadian Public Health Association, 100(5):340-343.
3. Saunders RP, Evans MH, Praphul J. 2005. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promotion Practice, 6:134-147.
4. Lim S.S. Et al. 2010. India’s Janani Suraksha Yojana, A conditional cash transfer programme to increase births in health facilities: an impact evaluation. The Lancet, 375:2009-23.
5. Andresen MA, Boyd, N. 2009. A cost-benefit and cost effectiveness analysis of Vancouver’s supervised injection facility. International Journal of Drug Policy, doi:10.1016/j.drugpo.2009.03.004.
6. Boyd NR, Windsor RA. 2003. A formative evaluation in maternal and child health practice: The partners for life nutrition education program for pregnant women. Maternal and Child Health Journal, 7(2):137-143.
7. Types of Evaluation. 2015. CDC: National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention.
8. Gobin M, Verlander N, Maurici C, Bone A, Nardone A. 2013. Do sexual health campaigns work? An outcome evaluation of a media campaign to increase chlamydia testing among young people aged 15-24 in England. BMC Public Health, 13:484.
9. Quality Improvement and Innovation Partnership. 2009. Needs Assessment Resource Guide.
OPTIONAL ASSIGNMENT DEADLINE (EVALUATION PLAN PART 1): FEBRUARY 19
READING WEEK: FEBRUARY 20-24
MODULE 6 - Indicators and Measuring Program Outcomes Week of February 26
w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a
Topics:
Measurement and monitoring of program outcomes
Identification of relevant indicators
Assessment of program impact
Detection, interpretation, and analysis of program effects
Readings:
1. Adamchak et al. 2000. A guide to monitoring and evaluating adolescent reproductive health programs. Tool Series 5. Chapter 4: Indicators.
2. Rogers T, Chappelle EF, Wall HK et al. 2011. Using DHDSP outcome indicators for Policy and Systems Change for Program Planning and Evaluation. Atlanta, GA: Centers for Disease Control and Prevention.
3. National Resource Center. 2010. Measuring Outcomes guidebook. Available at: http://strengtheningnonprofits.org/resources/guidebooks/MeasuringOutcomes.pdf
4. University of Kansas. 2016. The Community Tool Box. Section 9: Developing Baseline Measures of Behaviour. Available at http://ctb.ku.edu/en/tablecontents/chapter_1003.htm
MODULE 7- Evaluation Design, Methods and Data Collection Tools Week of March 5 & March 12
(spans 2 weeks)
Topics:
Evaluation design Quantitative methods (i.e., surveys, google analytics) Qualitative methods (i.e., focus groups, content analysis, goal attainment scaling, Photovoice) Sources of data for health programs
Readings:
1. University of Kansas. 2015. Community Tool Box. Section 4, Chapter 37 – Selecting an appropriate design for the evaluation. Available at: http://ctb.ku.edu/en/table-of-contents/evaluate/evaluate-community-interventions/experimental-design/main
2. Measurement, Learning and Evaluation Project. 2013. Types of Evaluation Designs: Available at: https://www.urbanreproductivehealth.org/toolkits/measuring-success/types-evaluation-designs
3. Fraser Health. 2009. A Guide to Planning and Conducting Program Evaluation: Section 4.5, p. 35-40.
4. University of Kansas. 2015. The Community Tool Box. Available at http://ctb.ku.edu/en/tablecontents/chapter_1003.htm
Section 6: Conducting Focus Groups
Section 7: Needs Assessment Survey
Section 12: Conducting Interviews
w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a
Section 13: Conducting Surveys
Section 15: Qualitative Methods to Assess Community Issues
Section 24: Arranging Assessments that Span Jurisdictions 5. Kramer L, Schwartz P, Cheadle A, Rauzon S. 2013. Using photovoice as a participatory evaluation
tool in Kaiser Permanente’s Community Health Initiative. Health Promotion Practice, 14(5):686-94.
6. Rao V, Woolcock M. 2003. Integrating qualitative and quantitative approaches in program evaluation. Available at: http://siteresources.worldbank.org/SOCIALANALYSIS/1104890-1120158274352/20566665/Integratingqualitativeandquantapproachesraoandwoolcock.pdf
7. Mailloux Z, May-Benson TA, Summers CA, et al. 2007. Goal attainment scaling as a measure of meaningful outcomes for children with sensory integration disorders. American Journal of Occupational Therapy, 61(2):254-59.
8. Office of Quality Improvement, University of Wisconsin-Madison. 2010. Survey Fundamentals: A guide to designing and implementing surveys. Available at : http://oqi.wisc.edu/resourcelibrary/uploads/resources/Survey_Guide.pdf
9. Optional: Katapally Longo webinar on innovative methods
OPTIONAL ASSIGNMENT DEADLINE (EVALUATION PLAN PART 2): MARCH 12
MODULE 8- Special Topic: Conducting Evaluation in Indigenous Communities
Week of March 19
Topics:
Considerations for conducting evaluation in Indigenous communities/of Indigenous-focused programs
Appropriate evaluation approaches
Case example
Readings:
1. Jamieson et al. (2012). Ten principles relevant to health research among Indigenous Australian populations. MJA, 197(1): 16-18.
2. LaFrance J, Nichols R. (2010). Reframing evaluation: defining an Indigenous evaluation framework. CJPE, 23(2):13-31.
3. Chouinard JA, Cousins JB. (2007). Culturally competent evaluation for Aboriginal communities: a review of the literature. JMDE, 4(8):40-57.
Additional readings:
4. Katz I, Newton BJ, Bates S, Raven M. (2016). Evaluation theories and approaches; relevance for Aboriginal contexts.
5. Inuit Tapiriit Kanatami & Nunavut Research Institute. (2007). Negotiating research relationships with Inuit communities: a guide for researchers.
w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a
6. LaFrance, J., Nichols, R., & Kirkhart, K. E. (2012). Culture writes the script: On the centrality of context in indigenous evaluation. In D. J. Rog, J. L. Fitzpatrick, & R. F. Conner (Eds.), Context: A framework for its influence on evaluation practice. New Directions for Evaluation, 135, 59–74.
7. Public Health Agency of Canada. (2015). Ways tried and true: Aboriginal methodological framework for the Canadian Best Practices Initiative.
MODULE 9- Challenges and Ethical Considerations Week of March 26
Topics:
Professional standards and ethical conduct
Management of multiple stakeholders
Measurement bias and validity issues
Readings:
1. Canadian Evaluation Society Code of Ethics: Available at: https://evaluationcanada.ca/sites/default/files/ces_ethics_ethique_sce.pdf
2. American Evaluation Association Guiding Principles for Evaluators. Available at: http://www.eval.org/p/cm/ld/fid=51
3. Mohan R, Sullivan K. 2006. Managing the politics of evaluation to achieve impact. New Directions for Evaluation, 112:7-23
4. http://www.evaluatod.org/assets/resources/evaluation-guides/evaluationethics-2-09.pdf 5. Ray M. 2006. Choosing a truly external evaluator. American Journal of Evaluation, 27(3):372-77.
MODULE 10 – Policy Evaluation Week of April 2
Topics:
The role of evaluation in evidence-informed decision and policymaking
Approaches in conducting policy evaluation (key questions, data sources, methods)
Example of a policy evaluation
Readings:
1. National Center for Injury Prevention and Control (CDC). Brief overview of policy evaluation. Available at: http://www.cdc.gov/injury/pdfs/policy/Brief%201-a.pdf
2. National Collaborating Centre for Healthy Public Policy. 2012. A framework for analyzing public policies: Practical Guide. Available at: http://www.ncchpp.ca/docs/Guide_framework_analyzing_policies_En.pdf
3. World Health Organization. 2007. Monitoring and evaluation of mental health policies and plans. Mental Health Policy and Service Guidance Package. Available at: http://www.who.int/mental_health/policy/services/14-monitoring%20evaluation_HKprinter.pdf –
Section 2: Framework for Setting up and Conducting Evaluation of the Policy and Plan
w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a
Section 3: Case Study: An Evaluation of a National Mental Health Policy and Plan in a Hypothetical Country
MODULE 11 – Evaluation Reporting and Wrap Up Week of April 9
Readings:
1. Mueller NB, Burke RC, Douglas AL, et al. 2008. Getting the word out: multiple methods for disseminating evaluation findings. Journal of Public Health Management and Practice. 14(2):170-176.
2. Fraser Health. 2009. A Guide to Planning and Conducting Program Evaluation: Section 5.0 Conduct Evaluation and Disseminate Findings. p. 45 – 54.
3. Centers for Disease Control and Prevention. Evaluation Reporting: A Guide to Help Ensure Use of Evaluation Findings. Atlanta, GA: US Dept of Health and Human Services; 2013. Available at: http://www.cdc.gov/dhdsp/docs/Evaluation_Reporting_Guide.pdf
4. Health Canada. 2015. Evaluation of the nutrition policy and promotion program 2009-2010 to 2014-2015. Executive Summary. Full text available at: http://www.hc-sc.gc.ca/ahc-asc/performance/eval/nppp-evaluation-pppn-eng.php
5. Reference/supplementary reading only: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health, Division of Nutrition, Physical Activity and Obesity. (2013). Developing an effective evaluation report: Setting the course for effective program evaluation. Available here: http://www.cdc.gov/eval/materials/Developing-An-Effective-Evaluation-Report_TAG508.pdf
FINAL EVALUATION PLAN ASSIGNMENT DEADLINE: APRIL 13
RECOMMENDED READINGS
All of the reading material for this course will be provided through the course website. This course has
no required textbooks, however optional reference books are listed below:
Issel LM. Health program planning and evaluation. 2nd or 3rd Edition. Toronto: Jones and Bartlett, 2008.
Green LW and Kreuter MW. Health Program Planning: An Educational and Ecological Approach. 4th
edition. New York: McGraw-Hill, 2005.
Myers A. Program evaluation for exercise leaders. 1st edition. Human Kinetics, 1999.
(Insert required readings)
w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a
EVALUATION
Assessment Weight Deadline
Participation in online discussion board 20% Ongoing
Evaluation Plan
- Part 1: Program description, evaluation purpose, and stakeholder
summary (20%)
- Part 2: Program logic model (30%)
- Part 3: Mixed methods evaluation design and draft survey or
interview guide (30%)
Students will be given the option to either:
1) Submit this assignment in its entirety on April 13 or;
2) Submit each section on its respective due date and receive
feedback prior to continuing with the rest of the assignment.
80%
Part 1 – February 19
Part 2 – March 12
Part 3 – April 13
*All written assignments will be assessed for overall grammar, clarity, presentation/formatting, and writing
style. A total of 5% of your assignment grade will be for writing style. Please consult University of Regina
Writing Services if you require help with writing or proofreading your assignments.
Participation
The discussion board will be used as a venue to post questions and engage in discussions with your
classmates. For each module, the course instructor will post a question or discussion topic for students
to discuss on the board for participation marks. Students are encouraged to share their experiences and
post any questions they may have about the material, however the grading will only be based on
responses to the question/discussion topic posted.
Discussion forum posts do not need to include references. There will be a word limit of 200 words to
help you keep responses succinct and incorporate readings/content from the module where
appropriate.
Live webinars will be held once per month to provide students with an opportunity to interact and ask
questions. Webinars will be scheduled using a poll after the term has commenced. Attendance for
webinars are optional and will not be graded for participation.
Evaluation Plan Assignment
This assignment will test students’ ability to apply the course concepts and design an evaluation for a
health program. Students will be given the option to choose between three different programs. The
assignment can be submitted in three parts, or in its entirety at the end of the course. More detailed
instructions and templates are available on the course website.
Policy on Late Assignments: It is essential that assignments are submitted by the time and date
specified. Late assignments will be penalized by deduction of 5% per day. In extenuating circumstances,
please contact the course instructor.
w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a
STUDENTS WITH SPECIAL NEEDS
University of Regina (U of R): Students in this course who, because of a disability, may have a need
for accommodations are encouraged to discuss this need with the instructor and to contact the
Coordinator of Special Needs Services at (306) 585-4631.
U OF S: Students in this course who, because of a disability, may have a need for accommodations
are encouraged to discuss this need with the instructor and to contact Disability Services for
Students (DSS) at 966-7273.
Students Experiencing Stress
University of Regina (U of R): Students in this course who are experiencing stress can seek assistance
from the University of Regina Counselling Services. For more information, please see the attached
document, visit this website: http://www.uregina.ca/student/counselling/contact.html, or call (306)
585-4491 between 8:30 a.m. to 4:30 p.m. Saskatchewan time Monday to Friday.
ACADEMIC INTEGRITY AND CONDUCT
U of R: Ensuring that you understand and follow the principles of academic integrity and conduct as
laid out by the University of Regina (available at http://www.uregina.ca/gradstudies/grad-
calendar/policy-univ.html) is vital to your success in graduate school. Ensuring that your work is your
own and reflects both your own ideas and those of others incorporated in your work is important:
ensuring that you acknowledge the ideas, words, and phrases of others that you use is a vital part of
the scholarly endeavour. If you have any questions at all about academic integrity in general or
about specific issues, contact your course instructor to discuss your questions.