outcomes-based, evidence-based program review
DESCRIPTION
Cyd Jenefsky, Ph.D Prof & Assoc VP Academic Affairs JFK University [email protected] WASC Assessment Leadership Academy August 5, 2011. outcomes-based, evidence-Based Program Review. Today’s Session:. Overview of OB-EB program review: Conceptual Framework - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/1.jpg)
OUTCOMES-BASED, EVIDENCE-BASED PROGRAM REVIEW
Cyd Jenefsky, Ph.D Prof & Assoc VP Academic Affairs
WASC Assessment Leadership AcademyAugust 5, 2011
![Page 2: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/2.jpg)
Today’s Session:
Overview of OB-EB program review: Conceptual Framework Program Assessment : Program Review Major Components Key Qualities Use of Results
Designing for Purpose
2
![Page 3: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/3.jpg)
Basics: What is It? Cyclical process for evaluating programs to make
decisions about improvements
Comprehensive audit, inquiry, reflection & analysis quality, effectiveness, currency, assets, risks,
sustainability/viability , mission alignment
Can be standardized, theme-based, customized or combination
Uses evidence from multiple sources as basis of inquiry, evaluation, and follow-up planning and budgeting
3
![Page 4: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/4.jpg)
Conceptual Frame-Shift From audit to inquiry From traditional input-based, process-focused
model to outcomes-based model From description & advocacy to evidence-based
analyses and planning Heightened attention to improving the quality of
student learning From focus on conducting effective program
review to using the results effectively
4
![Page 5: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/5.jpg)
Comprehensive
5
Learning Ecosystem
Processes
InputsResources
OutputsOutcomes
![Page 6: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/6.jpg)
6
Assessment of Student Learning
(multiple years)
Program Review
![Page 7: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/7.jpg)
Program Review
7
Assessment of Student Learning
Retention & Graduation
Rates
Curriculum Design & Currency
Admission Requirements
Employer Satisfaction
Advising
Enrollment Trends
Licensure RatesFaculty
Program Reputation
Student Satisfaction
![Page 8: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/8.jpg)
Main components of PR process:
• Self-Study• External Review• Internal Review
• Recommendations for Improvement
• MOU = Final Results• Action Plan – Implementation
8
![Page 9: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/9.jpg)
• Self-Study• External Review• Internal Review
Analysis of Evidence of Program Quality and Viability
9
![Page 10: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/10.jpg)
Clear identification of evidence you want programs to analyze
Use guiding questions for analysis of evidence to structure self-study inquiry and report to structure feedback reports from
internal and external reviewers10
![Page 11: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/11.jpg)
What data/evidence do you need to see to
understand program quality?
11
![Page 12: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/12.jpg)
Evidence of Quality and Effectiveness: (Institution-Specific)
• Curriculum (content, sequencing, currency, depth, rigor, breadth, scheduling, syllabi, delivery modalities, etc.)
• Student learning and success• Co-curricular integration• Assessment practices• Learning environment & support• Student profile• Faculty• Research, recognition, awards
12
![Page 13: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/13.jpg)
Evidence of Viability/Sustainability (inc. administrative cost effectiveness)
Demand for program - enrollment Student support services Faculty Staff Information & technology resources Facilities Financial resources
13
![Page 14: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/14.jpg)
Sample results to analyze
Student learning results (academic, co-curricular) Retention/graduation rates Licensure, placement rates Enrollment trends Student satisfaction with support services Alumni achievements Faculty grants, research, scholarship, awards Feedback from external constituencies
(e.g., employers, peer reviewers, community agencies, specialized accreditors, professional organizations)
14
![Page 15: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/15.jpg)
What do you want to know about multiple years’ worth
of learning results?
Identify Key Questions
15
![Page 16: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/16.jpg)
Main components of PR process:
• Self-Study• External Review• Internal Review
• Recommendations for Improvement• MOU
• Action Plan16
![Page 17: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/17.jpg)
Recommendations for Improvement
• Significant actions• Evidence-based• Feasible• Aligned with priorities• Include improvements in student
learning17
![Page 18: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/18.jpg)
18
Exercise and Sports Science (2005)
Better lab and office space. More grants Move offices closer to Sports Management. Retreat to discuss goals. Hire sports psychologist. Increase gender and racial/ethnic diversity. Prioritize health promotion. Reduce number of upper division electives. Encourage community outreach. Increase collaboration w/other departments
and programs.
![Page 19: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/19.jpg)
19
Theology & Religious Studies (2004)
Merge separate degree programs into one. Work to bridge the divide between T&RS. Revise and shorten student learning outcomes. Start long term planning around curriculum. Introduce more dynamic courses. Review the graduate program. Greater mentorship of junior faculty. Co-chairs and department sub-committees.
![Page 20: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/20.jpg)
MOU Agreement between program/division and
institution (or dept & division) Clear expectations Time line Resources committed
Used for planning & budgeting at multiple levels in institution
Accountability for implementation
20
![Page 21: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/21.jpg)
Sample MOUs from SFSU
21
![Page 22: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/22.jpg)
Action Plan - Implementation
Concrete actions Persons responsible Timeline Resources procurement Tracking progress and impact
22
![Page 23: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/23.jpg)
Action Plan Template and
Sample Action Plan from CSU Fresno
23
![Page 24: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/24.jpg)
MOU + Action Plan= PR RESULTS
24
![Page 25: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/25.jpg)
Clearly Identify All Players’ Roles
IUPUI: http://planning.iupui.edu/39.html
SFSU Guide: http://air.sfsu.edu/prog_review/sixth.html
CSUN Program Review Distribution of Tasks: www.csun.edu/assessment/docs/ProgramReview
USD Guidelines for PR
JFKU PR Guide (on your Blackboard site)
![Page 26: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/26.jpg)
Coordinate with specialized accreditation
cycles
26
![Page 27: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/27.jpg)
Summary of 7 Key Qualities of
Outcomes-BasedEvidence-Based
Program Review 27
![Page 28: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/28.jpg)
1. Meaningful collective inquiry
• Research questions that matter• Appropriate methodologies• Thematic, customized, uniform• Open to surprise• Multiple stakeholders involved
Including students!
28
![Page 29: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/29.jpg)
2. Stakeholder engagement
• Faculty, support staff, students, peer reviewers, employers, community, external experts
• Inquiry, evaluation and decisions for improvement
29
![Page 30: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/30.jpg)
3. Outcomes-based
• Focused on results
• Quality, effectiveness determined by evidence of outcomes achieved
30
![Page 31: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/31.jpg)
4. Evidence-based
• All evaluative claims about strengths, weaknesses, proposed improvements supported by relevant qualitative and/or quantitative evidence
• Evidence cited throughout
• PR results used as evidence for follow-up planning, budgeting
31
![Page 32: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/32.jpg)
5. Meaningful analysis
• Reasoned analysis of evidence (v. advocacy)
• Significance for future program planning, inc. improving student learning
• Responsive to changes in external environment
32
![Page 33: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/33.jpg)
6. Usable results
• Yields info relevant and pertinent to program improvement, inc. student learning
• Avoid busy-work unlikely to be used• Informs evidence-based action plan (with
timeline) for improving program/student performance
33
![Page 34: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/34.jpg)
7. Results Used
• Planning and budgeting at various levels throughout institution
• Program• Department• Division/College• University-wide
• Intentional capacity-building for constituencies to use results effectively
34
![Page 35: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/35.jpg)
Program-Level USE of Results - examples
Refine learning or service outcomes Refocus curricula to reflect changes in
discipline or profession Cut redundant classes & redesign other
courses to align better with program learning outcomes
Re-sequence courses for better scaffolding Coordinate course assignments in the major Develop undergraduate research initiative
35
![Page 36: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/36.jpg)
Program-Level USE of Results (cont’d)
Strengthen or streamline assessment practices Enable a digital learning community for
students Reassign faculty/staff or request new lines Design professional development program to
improve online teaching, learning, assessment Develop applied learning initiative with
employers, community
36
![Page 37: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/37.jpg)
USE of Results Beyond the Program - examples
Identify and respond to trends across programs Coordinate developmental courses Writing across the curriculum Clear flow-charts & paths to completion
Enhance intra-institutional synergies to promote student success: E.g., collaborate with student services &
general education to improve student retention and performance
Create college-wide first-year program Reallocate resources to academic support
services37
![Page 38: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/38.jpg)
USE of Results Beyond the Program (cont’d)
Create professional development programs across units
Better align department, college, institutional goals Develop cross-disciplinary service-learning
collaborations Grant resources university-wide to integrate
adjunct faculty into program assessment Reallocate resources to growing programs Allocate resources to undergrad research initiative Improve e-Portfolio software Revise program review process to produce more
usable results38
![Page 39: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/39.jpg)
Design with purpose
How do you want to useprogram review
in your program, departmentinstitution?
Purposes you want it to serve.39
![Page 40: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/40.jpg)
PURPOSES:
improve student learning improve curriculum, pedagogy, learning environment,
program administration, advising, etc. identify trends across program/dept units set program/division/institutional goals inform program/division/institutional planning inform decision-making, resource re/allocation at multiple
levels re/align programs with strategic priorities and plans respond to environmental changes report program performance to stakeholders
40
![Page 41: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/41.jpg)
General PURPOSE:
To generate useful, usable results to inform planning and budgeting at
various levels of the institution in order to improve
quality, effectiveness and sustainability of programs.
41
![Page 42: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/42.jpg)
Evaluation of Program Review Process• Is it effective?
• Achieving intended purposes?• Criteria for evaluation
• Your intended purposes • WASC Sr. and ACCJC rubrics• Criteria for evaluating PR design
• Feedback from multiple users/stakeholders
• Plan for improvement• Accountability for implementation
42
![Page 43: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/43.jpg)
How well does your current program review process
serve your purposes?
43
![Page 44: outcomes-based, evidence-Based Program Review](https://reader035.vdocuments.mx/reader035/viewer/2022062408/5681330d550346895d99cab2/html5/thumbnails/44.jpg)
Reflection Questions Are purposes of your PR process clear? Articulated
in the guidelines? What are the gaps or adjustments needed in your
current design in order to achieve purposes, including using results more effectively?
In what parts of your program review process could you integrate a focus on student learning and assessment? What kinds of professional development are needed to
support this focus? What kind of professional development is needed
to improve evidence-based decision-making and use of results to inform planning? 44