caate ppt glata 2014 workshop final rob only › wp-content › uploads › 2014 › 06 ›...
TRANSCRIPT
3/17/14
1
Using Outcomes to Drive Change in your Program!
March| 2014!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Outcomes….!
• Student Learning Outcomes!
• Program Outcomes!
• Clinical Outcomes!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Student Learning Outcome One row for each SLO
Objectives (Expected Learning Outcome)
List all Objectives of the given SLO in same row (1.1,1.2,1.3).
Assessment Strategy and Benchmark Note: The same instrument/tool (e.g., an external exam, a capstone project, or other cumula>ve experience) may be used to assess mul>ple objec>ves.
Assessment Results Summarize the results found for each objec>ve, related to the benchmark.
Use of Results Summarize the planned program changes intended to improve student learning.
3/17/14
2
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Outcomes: Develop a Plan (4): !
• There must be a comprehensive assessment plan to evaluate all aspects of the educational program. !
• Assessments used for this purpose must include those defined in Standards 6 and 7. Additional assessments may include, but are not limited to, clinical site evaluations, clinical instructor evaluations, completed clinical proficiency evaluations, academic course performance, retention and graduation rates, graduating student exit evaluations, and alumni placement rates one year post graduation.!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Annual Report Section V: Outcomes!
34. Provide one outcome for your program:!35. Provide the objectives that you measured for this
outcome: (2-7)!36. What assessment strategies did you use to
measure each objective: !37. Provide the benchmark(s) you set to determine if
your students are competent based on your assessment strategy: !
38. Which benchmarks were met/not met: !39. What changes did you make in the program as a
result of the outcomes measurement: !
3/17/14
3
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Section V: Outcomes – WHAT NOT TO DO!!
34. Outcome: Pass the BOC …!35. Objectives: Learn all competencies and proficiencies!36. Assessment strategies: Class Grades! !37. Provide the benchmark(s): Each Student will … !38. Which benchmarks were met/not met: All. !39. What changes did you make in the program as a
result of the outcomes measurement: None!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Outcomes: Develop a Plan (4): !
• There must be a comprehensive assessment plan to evaluate all aspects of the educational program. !
• Assessments used for this purpose must include those defined in Standards 6 and 7. Additional assessments may include, but are not limited to, clinical site evaluations, clinical instructor evaluations, completed clinical proficiency evaluations, academic course performance, retention and graduation rates, graduating student exit evaluations, and alumni placement rates one year post graduation.!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Developing an assessment plan:!
The Basics!1. Institutional Mission Statement!2. Department / Program Mission Statement!3. Institutional Goals and Outcomes!4. Department / Program Outcomes!5. Measureable Objectives!6. Assessment Strategies / Benchmarks!7. Administrative Assessment of Results!
3/17/14
4
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Mission Statement !
• This is the broad statement of purpose and aspiration of the department or program. It should be closely aligned to the University’s mission statement and should be semi-permanent. !
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Mission Examples!
• The University of Wisconsin Oshkosh provides a wide array of quality educational opportunities to the people of northeastern Wisconsin and beyond through the discovery, synthesis, preservation and dissemination of knowledge. The interaction of our dedicated faculty, staff and students fosters an inclusive learning environment that prepares our graduates to meet the challenges of an increasingly global society.!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Mission Examples!
• The University of Wisconsin Oshkosh Athletic Training Program exists to provide an effective and interactive learning environment for individuals pursuing a career in athletic training. The program is committed to preparing high caliber graduates equipped to provide quality health care to physically active populations in a changing world.!
3/17/14
5
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Developing an assessment plan:!
The Basics!1. Institutional Mission Statement!2. Department / Program Mission Statement!3. Institutional Goals and Outcomes!4. Department / Program Outcomes!5. Measureable Objectives!6. Assessment Strategies / Benchmarks!7. Administrative Assessment of Results!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Institutional Outcomes!
• How many institutional Outcomes/Goals does your program meet?!
• Your Assessment Committee’s want to know!!
• Your institution’s accreditors will look for alignment!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Eastern U. Institutional Learning Outcomes !1. Students develop intellectual curiosity, passion, and agility, valuing the life of the
mind and life-long learning.!2. Students develop their critical thinking, reflection, analysis and communication
skills.!3. Students develop knowledge and competencies in the arts, sciences and
professions.!4. Students develop and expand their Christian worldview, grounded in the
Scriptures.!5. Students discern the ethical consequences of decisions and actions.!6. Students are motivated to assume responsibility for justice and to show a
transformative influence—especially regarding social, political and economic justice.!
7. Students are prepared to live in an interdependent world, aware of societal and global problems and committed to engage in solving them.!
8. Students increase in self-awareness and in their sensitivity towards others and others’ needs and situations.!
9. Students contribute to fostering an environment where diversity is appreciated and reconciliation is practiced.!
3/17/14
6
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Eastern U. Institutional Learning Outcomes !1. Students develop intellectual curiosity, passion, and agility, valuing the life of the
mind and life-long learning.!2. Students develop their critical thinking, reflection, analysis and communication
skills.!3. Students develop knowledge and competencies in the arts, sciences and
professions.!4. Students develop and expand their Christian worldview, grounded in the
Scriptures.!5. Students discern the ethical consequences of decisions and actions.!6. Students are motivated to assume responsibility for justice and to show a
transformative influence—especially regarding social, political and economic justice.!
7. Students are prepared to live in an interdependent world, aware of societal and global problems and committed to engage in solving them.!
8. Students increase in self-awareness and in their sensitivity towards others and others’ needs and situations.!
9. Students contribute to fostering an environment where diversity is appreciated and reconciliation is practiced.!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Developing an assessment plan:!
The Basics!1. Institutional Mission Statement!2. Department / Program Mission Statement!3. Institutional Goals and Outcomes!4. Department / Program Outcomes!5. Measureable Objectives!6. Assessment Strategies / Benchmarks!7. Administrative Assessment of Results!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Characteristics of outcomes!The outcome is our destination!
!
• Primary purpose is student learning!• Should reflect the culture, mission and needs of
an individual institution/program!• Why we do what we do!• What does a graduate from your program
“look like”? !
3/17/14
7
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
What are we telling them?!• Do our students meet our standards?!• Do we comply with external standards?!• How do we compare with peer standards?!• Value-added – are our students improving?!• Historical trends – is teaching and curricula
improving?!
= What we expect our graduate’s will look like!– Knowledge, skills, values and habits!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Not an outcome…
!Entry-level professional knowledge, comprehension, and proficient skills of an orthopedic clinical examination and diagnosis of the knee!
3/17/14
8
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Not an outcome !Prepare graduates to apply a wide variety of specific health care skills and knowledge!!within each of the following 8 content areas (NATA, 2011): evidence based practice, prevention and health promotion, clinical examination and diagnosis, acute care of injury and illness, therapeutic interventions, psychosocial strategies and referral, healthcare administration, professional development and responsibility. !
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
ATS winning National Championship during clinical rotation with Villanova University!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
UW Oshkosh Students Winners of the 2013 National Quiz Bowl!
3/17/14
9
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Example Outcome#1:!!Students will use knowledge of basic science and research methodology to interpret evidence-based research related to athletic training to answer clinical questions and to guide clinical practice.!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Eastern U. Institutional Learning Outcomes !1. Students develop intellectual curiosity, passion, and agility, valuing the life of the
mind and life-long learning.!2. Students develop their critical thinking, reflection, analysis and communication
skills.!3. Students develop knowledge and competencies in the arts, sciences and
professions.!4. Students develop and expand their Christian worldview, grounded in the
Scriptures.!5. Students discern the ethical consequences of decisions and actions.!6. Students are motivated to assume responsibility for justice and to show a
transformative influence—especially regarding social, political and economic justice.!
7. Students are prepared to live in an interdependent world, aware of societal and global problems and committed to engage in solving them.!
8. Students increase in self-awareness and in their sensitivity towards others and others’ needs and situations.!
9. Students contribute to fostering an environment where diversity is appreciated and reconciliation is practiced.!
3/17/14
10
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Developing an assessment plan:!
The Basics!1. Institutional Mission Statement!2. Department / Program Mission Statement!3. Institutional Goals and Outcomes!4. Department / Program Outcomes!5. Measureable Objectives!6. Assessment Strategies / Benchmarks!7. Administrative Assessment of Results!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Writing Objectives!
• Measurable, demonstrable components of a Student Learning Outcome!– Specific knowledge, skills, and behaviors!– Clinical and/or Didactic!
• Taught developmentally in the curriculum!– introduced, reinforced/practiced and ultimately applied/
integrated into the practice of the discipline in higher level courses!
• Foundation for assessments at the course and department/program level.!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Writing Objectives!
• Bloom’s Taxonomy!– Need varied levels!– Many sources of “Verbs” at each level!
• SMART format!– Specific!– Measurable!– Attainable!– Realistic!– Time-Oriented!
3/17/14
11
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Good Example!Outcome Example: Students will use knowledge of basic science and
research methodology to interpret evidence-based research related to athletic training to answer clinical questions and to guide clinical practice.!
!Objectives of Outcome Example:!1. !Successfully access and accurately interpret a variety of healthcare
related research data. [Classroom]!2. !Demonstrate contemporary skill in comprehensive care of patients with
injuries. [Clinical Education]!3. !Demonstrate contemporary skill in comprehensive care of patients with
illnesses. [Clinical Education] !4. !Develop patient-centered treatment plans that consider the impact of the
condition on the patient’s life (eg, socioeconomic, religion). [Clinical Education]!
5. !Incorporate patient-centered outcome measures to evaluate the quality of care provided. [Clinical Education]!
6. !Develop a comprehensive plan for continuing education. [Classroom]!!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Poor Example!
Outcome Example: Entry-level professional knowledge, comprehension, and proficient skills of an orthopedic clinical examination and diagnosis of the knee!
Objectives of Outcome Example:!!Taking a patient's history; inspection of knee along with observation of gait; palpation of bony and soft-tissues; joint motion assessment using a goniometer; muscle function assessment using manual muscle tests; circulatory assessment; neurological function assessment including sensory, motor, and reflex testing; selective tissue testing utilizing appropriate special tests; clinical diagnosis of common injury, condition, illness, and/or disease based on clinical examination; professional behaviors exhibited during the clinical examination.!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Laurent & Weidner (2002)
3/17/14
12
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Good Example!Outcome Example: Students will use knowledge of basic science and
research methodology to interpret evidence-based research related to athletic training to answer clinical questions and to guide clinical practice.!
!Objectives of Outcome Example:!1. !Successfully access and accurately interpret a variety of healthcare
related research data. [Classroom]!2. !Demonstrate contemporary skill in comprehensive care of patients with
injuries. [Clinical Education]!3. !Demonstrate contemporary skill in comprehensive care of patients with
illnesses. [Clinical Education] !4. !Develop patient-centered treatment plans that consider the impact of the
condition on the patient’s life (eg, socioeconomic, religion). [Clinical Education]!
5. !Incorporate patient-centered outcome measures to evaluate the quality of care provided. [Clinical Education]!
6. !Develop a comprehensive plan for continuing education. [Classroom]!!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Developing an assessment plan:!
The Basics!1. Institutional Mission Statement!2. Department / Program Mission Statement!3. Institutional Goals and Outcomes!4. Department / Program Outcomes!5. Measureable Objectives!6. Assessment Strategies / Benchmarks!7. Administrative Assessment of Results!
3/17/14
13
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Assessment Strategies / Benchmarks!
• Strategies may be used to cover multiple objectives!
• Rubrics must be used to gauge learning!• Assignments may be used but grades don’t tell
the story!– Multiple assignments !– Projects / Presentations!– Practicum evaluations!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Outcome: Students will demonstrate post-graduation preparedness !
Objectives:!1. Students will demonstrate their ability to
integrate themselves into a post graduate professional setting.!
2. Students will be able to practice in a competent and safe manner.!
3. Students will indicate they are prepared to work as a professional!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
What assessment strategies did you use to measure each objective:!q Alumni Survey!q Case Study!q Class Presentation!q Employer Survey!q Exam!q Exit Interview!q IPE!q Objective Structured Clinical Exam (OSCE)!q Practical Examination!q Preceptor Evaluation!q BOC Exam!
3/17/14
14
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Assessment Strategies / Benchmarks!• Determine % of students that earn at least a
minimally acceptable score!• Compare against national averages or compare
to peer institutions!• Base some off of CAATE’s standards!
– BOC Pass rate!– Job placement!– Alumni Surveys!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Benchmarks!• 80% of students will score above a 70 on their senior
case study according to the faculty rubric.!
• Which benchmarks were met / not met?!
• “We met our benchmark of 80% - no further action needed at this time”!
• Problems??
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
3/17/14
15
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Benchmarking 2.0!Outcome Excep,onal Very Good Adequate Inadequate
Communica>on 5% 85% 10% 0
Cri>cal Thinking 7% 70% 20% 3%
Wri>ng 4% 60% 30% 6%
Professionalism 15% 75% 10% 0
Evidenced-‐based Clinical Prac>ce 6% 40% 50% 4%
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Developing an assessment plan:!
The Basics!1. Institutional Mission Statement!2. Department / Program Mission Statement!3. Institutional Goals and Outcomes!4. Department / Program Outcomes!5. Measureable Objectives!6. Assessment Strategies / Benchmarks!7. Administrative Assessment of Results!
3/17/14
16
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Utilizing the results for change!• Improve curriculum!• Maximize learning!• Tutoring!• Library Services!• Academic advisement and counseling!• Technology !• Co-curricular opportunities!• Budgeting!• Academic review!• Improve relationships / learning at clinical sites!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
2. OBJECTIVES What are the
measurable pieces that make up our
outcomes
3. ASSESSMENT (10) Have we met our benchmarks and what do our
students look like
4. ACTION PLAN (12) -‐ Curricular Changes -‐ Preceptor Educa>on -‐ Modify Tools -‐ Increase Benchmarks
1. OUTCOMES What we want our graduates to look
like
3/17/14
17
CLOSING REMARKS!
March| 2014!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Annual Report Section V: Outcomes!
34. Provide one outcome for your program:!35. Provide the objectives that you measured for this
outcome: (2-7)!36. What assessment strategies did you use to
measure each objective: !37. Provide the benchmark(s) you set to determine if
your students are competent based on your assessment strategy: !
38. Which benchmarks were met/not met: !39. What changes did you make in the program as a
result of the outcomes measurement: !
3/17/14
18
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Student Learning Outcome One row for each SLO
Objectives (Expected Learning Outcome)
List all Objectives of the given SLO in same row (1.1,1.2,1.3).
Assessment Strategy and Benchmark Note: The same instrument/tool (e.g., an external exam, a capstone project, or other cumula>ve experience) may be used to assess mul>ple objec>ves.
Assessment Results Summarize the results found for each objec>ve, related to the benchmark.
Use of Results Summarize the planned program changes intended to improve student learning.
T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!
Assessment Strategy & Indicators Measured → Details about Assessment Strategy Methodology ↓
Assessment Strategy #1: Objectives Measured:
Assessment Strategy #2: Objectives Measured:
Assessment Strategy #3: Objectives Measured:
Assessment Strategy #4: Objectives Measured:
Description of Strategy (e.g., test, rating scale, culminating assignment)
Assessment Result yielded (e.g. rubric score, test score means)
Benchmark (Criteria for Success)
Sample Size and Source
Administrator
Time of Administration of Assessment Strategy
Results maintained/archived where and by whom
Time of Analysis of Results
Analyzed by
Feedback to Faculty/ Discussion