patrice lancey, ph.d divya bhati,ph.d university of [email protected] operational excellence...
TRANSCRIPT
![Page 1: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/1.jpg)
Patrice Lancey, Ph.DDivya Bhati,Ph.D
University of Central Florida
![Page 2: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/2.jpg)
UCF institutional effectiveness assessment process
Evolution of the assessment process
Rubric development and implementation
Benefits
2
![Page 3: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/3.jpg)
Basic Carnegie classification: research universities (high research activity)
UCF also achieved Carnegie Community Engagement Classification
2nd largest university in U.S. with 56,337 students 1,415 acres on Orlando campus 10 regional campuses and numerous other
instructional sites extensive distance learning offerings 12 colleges, including a medical college 216 degree programs (91 bachelor’s, 92 master’s, 3
specialist, 29 doctoral, 1 professional)
![Page 4: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/4.jpg)
4
Coordinators
DRC Members
UAC
ProvostVPs and Deans
President
![Page 5: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/5.jpg)
Change
5
Plan
Measure
Analyze
Continuous Improvement
![Page 6: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/6.jpg)
355 programs and units report• Year round process• Centralized online reporting system • Ongoing reviews and feedback by DRC
September – Coordinators submit final results and plans
October – DRCs review results and plans
November to December – UAC final review
6
![Page 7: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/7.jpg)
7
IE Assessment
Program or Unit
Reviews
Strategic Planning
![Page 8: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/8.jpg)
Loads of paper documents
Manual submission of assessment plans
No common assessment plan template
No structured review of plans
Little faculty and staff involvement
Difficult to manage or use
8
![Page 9: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/9.jpg)
Establishment of University Assessment Committee
Creation of an assessment support office
Formation of a common assessment template in Microsoft Word
Knowledge management –manually driven• Communication by email• Electronic submission of assessment plans
by email
9
![Page 10: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/10.jpg)
Includes more players in the process Increases communication
• promotes best practice• institutional memory
Reduces work load for faculty and staff• doers• support staff
Promotes collaboration and mentoring Centralized capture of knowledge Extract and report information
• improve process and support• meta analysis
10
![Page 11: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/11.jpg)
IE Assessment Web Application
www.assessment.ucf.edu
![Page 12: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/12.jpg)
12
![Page 13: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/13.jpg)
13
![Page 14: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/14.jpg)
Prior Rating Scales for Results
14
![Page 15: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/15.jpg)
Achieve clear and consistent rating system
Deepen collaborative model for reviewers and coordinators
Enhance the usefulness of the assessment process and deepen quality
Tie IE assessment with strategic planning
15
![Page 16: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/16.jpg)
Sub committee of the University Assessment Committee (UAC) was established to develop rubrics
Drafts circulated to UAC Revisions incorporated Pilot tested with coordinators and Divisional
Review Committee (DRC) members Designed feedback survey Analyzed feedback survey to improve the
content and language of rubrics
16
![Page 17: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/17.jpg)
17
UniversityofCentralFloridaInstitutionalEffectivenessAssessmentPlanRubricBeginning(1) Emerging(2) MeetsExpectations(3) Accomplished(4) Exemplary(5)
Threeormore oftheMeetsExpectationsindicatorsarenotmet.
Uptotwo oftheMeetsExpectationsindicatorsarenotmet.
Allofthefollowingindicatorsaremet. AlloftheMeetsExpectationsindicatorsaremetandatleastone oftheadditionalindicatorsismet.
AlloftheMeetsExpectationsindicatorsaremetandalloftheadditionalindicatorsaremet.
1. Missionstatementincludesthefollowing:nameofprogram/unit,purpose,primaryfunctionsandactivities,andstakeholders
AdditionalIndicators AdditionalIndicators
2. Assessmentprocessisprovidedanddescribesassessmentstrategies,andhowtheprogramorunitmembersareinvolved
7. Specificassessmentinstrumentsaremadeavailable(e.g.,viaURL,asattachments,etc.),ifnotproprietary
7. Specificassessmentinstrumentsaremadeavailable(e.g.,viaURL,asattachments,etc.),ifnotproprietary
3.Numberofoutcomes:•administrativeunits:minimumofthreeoutcomes•graduateacademicprograms:minimumofthreestudentlearningoutcomes•undergraduateacademicprograms:minimumofeightstudentlearningoutcomesthatincorporatesacademiclearningcompacts
8. Outcomesintheplanincludestretchtargetsorincludemeasurementofacademicoroperationalinitiativesthatresultedfrompreviousassessment
8. Outcomesintheplanincludestretchtargetsorincludemeasurementofacademicoroperationalinitiativesthatresultedfrompreviousassessment
4. Minimumoftwoappropriatemeasuresforeachoutcome;atleastoneisadirectmeasure
9. DescribestherelationshipbetweentheInstitutionalEffectivenessAssessmentplanandtheUniversity’sStrategicPlan
9. DescribestherelationshipbetweentheInstitutionalEffectivenessAssessmentplanandtheUniversity’sStrategicPlan
5. Measuresestablishspecific,quantifiableperformancetargets
6. Measuresandtargetsaredesignedtopromoteimprovement
*Ifprogramsorunitsfailtoprovideanyinput,theirplanwillbeevaluatedwith“Noeffort(0).”
Copyright©2010byUniversityofCentralFlorida
![Page 18: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/18.jpg)
18
UniversityofCentralFloridaInstitutionalEffectivenessAssessmentResultsRubricBeginning(1) Emerging(2) MeetsExpectations(3) Accomplished(4) Exemplary(5)
Threeormore oftheMeetsExpectationsindicatorsarenotmet.
Uptotwo oftheMeetsExpectations indicatorsarenotmet.
Allofthefollowingindicatorsaremet. AlloftheMeetsExpectationsindicatorsaremetandatleastone oftheadditionalindicatorsismet.
AlloftheMeetsExpectationsindicatorsaremetandalloftheadditionalindicatorsaremet.
1.CompleteandrelevantdataareprovidedforallmeasuresORifdataareincompleteormissing,anexplanationisprovided
AdditionalIndicators AdditionalIndicators
2.Datareportingisthorough(seebelow)
7.Includesdescriptionofhowtheassessmentprocesshasbeenusefultoyourprogramorunit
7.Includesdescriptionofhowtheassessmentprocesshasbeenusefultoyourprogramorunit
3. Resultsforeachmeasureindicatewhetherthetargetforthatmeasurehasbeenmet
8. IncludesdescriptionofhowIEAssessmenthasresultedinqualityimprovementinitiatives
8. IncludesdescriptionofhowIEAssessmenthasresultedinqualityimprovementinitiatives
4.Reflectivestatementsareprovidedeitherforeachoutcomeoraggregatedformultipleoutcomes
9.Datacollectionandanalysisareusedtoassesstheimpactofimplementedchanges,demonstratingafully“closedloop”process
9. Datacollectionandanalysisareusedtoassesstheimpactofimplementedchanges,demonstratingafully“closedloop”process
5.Implementedandplannedchangesareincludedandarelinkedtoassessmentdata,orifnochangesarereported,anexplanationisprovided
6.Assessmentinstrumentsareattachedorlinkedtoifnotproprietary
2.i.e.,populationsaredefined;samplingmethodsandresponseratesareprovidedwithsurveydata,etc.
*Ifprogramsorunitsfailtoprovideanyinput,theirplanwillbeevaluatedwith“Noeffort(0).”
Copyright©2010byUniversityofCentralFlorida
![Page 19: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/19.jpg)
Programmed into an existing assessment web application
Replaced existing reviewer rating scales in the Institutional Effectiveness Assessment plans and results templates
Made link in templates to provide easy access to PDF of rubrics for coordinators, DRC members and DRC Chairs (UAC)
19
![Page 20: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/20.jpg)
![Page 21: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/21.jpg)
Conducted a series of workshops and training sessions to clarify levels and indicators
Applied rubrics to actual plans reports and results reports
Worked toward establishing inter-rater reliability
Developed rubric reports for university stakeholders to show how programs or support services areas are meeting the expected standards
21
![Page 22: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/22.jpg)
22
Program or Unit StatusPlans not approved by the DRC Chair(s) 0Plan not created 0Plan in progress 0Plans submitted to DRC 0Plan review process has begun, but the review has not been approved 0
Total number of program and unit plans approved by the DRC Chair 355
Total number of programs and units doing assessment 355
Program or Unit Rubric Level Number of Programs or Units meeting the level
Percentage of Programs or Units meeting the level
Exemplary 53 15Accomplished 71 20Meets Expectations 178 50Emerging 35 10Beginning 18 5
* Note: Numbers and percents are fictitious and are given as an example.
University of Central Florida*
University of Central Florida*
![Page 23: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/23.jpg)
Communication tool• Sets clear expectations• Uses common terminology• Offers concise, focused and timely feedback
Guides self-evaluation• Programmed into web application
Improves accuracy and consistency throughout the assessment process
Generates meaningful discussion – more involved faculty and staff members
23
![Page 24: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/24.jpg)
Tool to deepen the collaborative model
Increases channels of communication
Results in more “off-line” consultations
Higher attendance of assessment workshops
University strategic plan linkages integrated into plan rubric
24
![Page 25: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/25.jpg)
Continue training and one to one consultations and collaborative work
Develop library of examples aligned to the rubric levels
Share reports with trends over time
Clarify distinction between indicator 7 and 8 on IE assessment results rubric
25
![Page 26: Patrice Lancey, Ph.D Divya Bhati,Ph.D University of ......divya.bhati@ucf.edu Operational Excellence and Assessment Support 26. Title: Microsoft PowerPoint - UCF_Implementing an Institutional](https://reader033.vdocuments.mx/reader033/viewer/2022061002/60b13c45f1cc2d05c516538d/html5/thumbnails/26.jpg)
Continue the conversation:Dr. Patrice [email protected]. Divya BhatiAssociate [email protected] Excellence and Assessment
Support www.oeas.ucf.edu
26